A PRACTICAL APPROACH

By W H Inmon

It is no surprise to anyone who has been alive and awake the past few years that the world of information has undergone a tremendous change. Into this new world of information processing has arrived AI – artificial intelligence. Every technician, every business person and every consumer in the world is profoundly affected by the arrival of AI.

THE DATA ADMINISTRATOR/DATA GOVERNANCE PROFESSIONAL

For years now – since the advent of technology – the world of business has relied on data administrators to build, maintain and operate the IT environment. The data administrators have developed a whole skill set based on the demands of keeping information systems up and running.

It is a surprise(and a shock) to the data administrators and the people doing data administration and data governance in the earlier world of IT that as AI arrives the skills learned over decades of managing data simply no longer apply.

A RUDE AWAKENING

It is a rude awakening to the data administrator and the data governance professional to find out that his/her skill sets do not work and are not applicable to the world of AI.

This discovery is very unsettling to nearly all of these early data administrators. People with an established and well earned career wake up to find that there is a need to learn an entirely new set of skills.

People well established in a career find that having to earn a new and unfamiliar skill set to be very unsettling.

REALITY

However unsettling this may be, it is simply reality. The transition from classical structured, transaction processing systems to the world of AI is a permanent transition.

The production of buggy whips is simply no longer relevant to the world of AI. The world of data administration and data governance that was originally created knows well how to produce those buggy whips

.THE OBJECTIVES OF DATA MANAGEMENT IN AI

The education process as to what is needed for the world of AI for the purpose of managing data begins with an explanation of the objectives of data management in the world of AI data administration. The objectives of data management in the AI environment are very different from the objectives of data management in the classical structured world.

There are two basic objectives of data management in the world of AI –

1) Minimize the size of the LLM

2) Keep the LLM focused on business value and the business itself.

There are some very good reasons for the focus that has been described. By keeping the size of the LLM as small as possible, the data administrator has minimized the cost of performing AI. At some point the cost of doing AI starts to exceed its value. At this point AI becomes self defeating. The data administrator wants to avoid this circumstance at all costs.

The less expensive AI is, the more cost effective it becomes.

The second objective of data administration for AI is to focus the information found inside the LLM. The focusing of the LLM greatly minimizes the opportunity for the creation of hallucinations. In addition, the focus on the business of the organization enhances the opportunity of using AI to achieve business value.

Therefore, the minimization of the size of the LLM and the focusing of the LLM on business becomes the primary objective of the data administrator/data governance professional in the AI environment.

SOME DIFFERENCES

Data administration in the AI environment is fundamentally different from the administration of data in the structured, transaction processing environment.

In the classical structured, transaction processing environment data administration is performed in a hands on basis. Data bases are designed. Data models are created. Indexes are created. Data bases are monitored, and so forth. The data administrator in the structured environment is a very proactive, hands on person. If something needs to be fixed, the data administrator goes and fixes it.

Data administration in the AI environment is very different. The data administrator in the AI environment is similar to a puppeteer. The data administrator in the AI environment does not directly affect much of the processing of data. Instead, the data administrator in the AI environment indirectly affects the data. This is done by carefully considering and treating the text that enters the LLM.

There is then a fundamentally different role played by the data administrator in the AI environment and the structured environment.

SO WHAT DO YOU NEED TO DO?

So the question remains – how does an individual go about actually doing AI data administration/data governance?

In order to understand the role of the data administrator in the AI environment, consider how the AI environment has typically been envisioned.

The classical way that AI has been envisioned is in a state where raw text is simply loaded into an LLM.

Simply trying to cram a lot of text into an LLM in order to train the LLM is absolutely the wrong thing to do. It defeats – in the long run – even having an AI facility

.The proper approach to take is to load data into the LLM with great care. By loading the right data and by extension, not loading the wrong data) into the LLM, the data administrator greatly influences the success of AI in the corporate environment.

A TOOL

To the end of managing the loading of text properly, there is a tool -Textual ETL – or TETL. TETL is dedicated to the needs of data management and data governance.

SEPARATING BUSINESS RELEVANT TEXT FROM EXTRANEOUS TEXT

The first task of the data administrator is to keep the size of the LLM as small as possible. The primary way this is done is by not placing extraneous, non business relevant text into the LLM.

Consider the following document. In the document part of the document is business relevant and part of the document is not. The data administrator needs to make sure that the relevant portion of the document enters the LLM and the extraneous portion of the document does not get to be presented to the LLM.

In doing so the data administrator keeps the LLM small and efficient and keeps the LLM focused.

The following diagram depicts this basic need.

There is a good reason for removing the extraneous portion of a document. If extraneous text enters an LLM for the purposes of training, the extraneous text bloats the LLM and raises the probability of hallucinations – exactly what is not needed or desired in the building of the LLM

.BOILERPLATE

The second technique for managing the size of an LLM is to ensure that repetitive text does not enter the LLM. Repetitive text is common in business. It is often call called “boilerplate”.

Boilerplate is found in legal documents, financial documents, tax documents and elsewhere. It is a common and widely used technique of running a business.

Boilerplate text is identical text that found in many places. Consider the corporate 10k which is filed with the SEC each quarter if the corporation is publicly traded. The corporate 10k is filed every quarter in accordance with the SEC rules. Approximately 90% of the corporate 10k is boilerplate. If the corporation submits the quarterly 10k to the LLM each quarter, a very large amount of text is ingested by the LLM for which there will be no return on investment.

(But there certainly will be an expense!)

But corporate 10k reports are hardly the only source of boilerplate. The corporation has many other instances of repetitive text being created. In order to keep the LLM as small and as focused as possible, the boilerplate must never be presented to the LLM.

The boilerplate needs to be recognized and removed from the stream of text entering the LLM

.STRUCTURED TRANSACTION DATA

The structured, transaction processing systems contain much useful and interesting information which needs to be placed in the LLM. Typical structured data bases include SQL Server, Oracle, DB2, Teradata, and others. But structured data is not in a format that is suitable for the inclusion into the LLM.

The LLM ingests input in the form of chunks.

There is a need then to be able to search structured data bases and retrieve appropriate data and reformat the data into an acceptable format for inclusion into the LLM.

The following figure shows the selection of a record from the structured environment and the reformatting of the record into a textual based chunk

.THE NEED FOR SECURITY FOR STRUCTURED DATA

There is an important consideration in the selection of structured text for the inclusion into an LLM. That consideration is that much data found in the structured environment is private and/or confidential data. It would cause many unwanted problems to the corporation if structured data that should be protected should find its way into the LLM.

Therefore, a serious screening and security process is needed for the vetting and selection of structured data to be introduced into the LLM.

The consequences of not paying attention to the concerns of privacy and confidentiality can be very severe. Furthermore, the consequences can be easily prevented

.MANUAL CONTROL BEFORE ENTRY

There is a need for a manual override of the chunks of text that will be presented to the LLM. Instead of blindly throwing text into the LLM, there needs to be a point of manual intervention before the text is presented to the LLM.

Human intervention for the selection and control of text before the text ever arrives at the LLM is an absolute necessity for managing the text that is presented to the LLM.

INTERNET SCRAPING

Another need for the loading of text into the LLM is the ability to select text for processing from the Internet.

There is a need for a custom scraper for selection and collection of text from the Internet

.The judicious selection of the sources of text from the Internet is an important starting factor in focusing the contents of the LLM.

DATA ADMINISTRATION/DATA GOVERNANCE IN THE AI ENVIRONMENT

All of these factors become the job of the data administrator.

To state things differently, when an organizations goes about building its own LLM, what does the organization want –

An LLM that is bloated, expensive, and unfocussed, or

An LLM that is small, efficient, inexpensive and focused, yielding few hallucinations?

If an organization wants the efficient and focused LLM, data administration and data governance is the way to achieve that environment.

TO TOOL OR NOT TO TOOL?

As mentioned earlier, there is technology – TETL – that is build for the administration of the LLM environment.

Is such technology a necessity? The answer is no. An organization does not have to have TETL. But a tool GREATLY accelerates and eases the challenges of doing data administration in the AI environment.

To use an analogy, if it were your job to cut down a tree, would you rather have a pocket knife or a chain saw?

With a pocket knife you may never cut the tree down.

But with a chain saw, the tree comes down quickly and safely

.WHEN YOU DECIDE TO GET SERIOUS ABOUT MANAGING YOUR AI ENVIRONMENT

Textual ETL – TETL – is designed specifically for the AI LLM management environment. You will want TETL when you decide to get serious about managing your corporate LLM environment.

You can find out a lot more about TETL at https://LLMMGMT.com

.

By W H Inmon

Jamie Knowles, ERStudio, @semanticsarethesolution

There is much confusion in the world of corporate AI. The world has seen the public version of AI and is entranced by it. There is great demand to take the public version of AI into the walls of the corporation and allow the corporation to enjoy the same powers and privileges conferred by AI on the general public.

Bringing AI into the corporation is a different kettle of fish than using the public version of AI. In the public version of AI there is little or no maintenance. The public AI vendors do much (or all) of the behind the scenes activities that need to be done.

CORPORATE AI PROCESSING

But bringing the corporate version of AI into the corporation is a different story. There is the issue of confidentiality of corporate data. Whereas public data is not confidential, the corporation hardly wants its confidential data available to the public on the public LLM. And there is the issue of building and managing the private corporate LLM. The corporation is responsible for that activity, not the public vendor.

The challenge is that the corporation has never before been responsible for building and maintaining its own LLM. The corporation certainly has witnessed the building and management of structured, transaction processing systems. But the corporate LLM that is needed for AI in the corporation is very different from the structured, transaction processing systems that the corporation is used to handle. The activities of building and managing the corporate LLM are ENTIRELY different from the activities of data management and data governance that precede the building of the corporate LLM.

The most foolish thing a corporation can do is to try to employ the tools and techniques of data management and data governance that were learned in the structured world in the new world of the corporate LLM.

That approach is blatantly a failure.

DATA MANAGEMENT/DATA GOVERNANCE OBJECTIVES

So what are the objectives of data governance and data management of the corporate LLM environment?

There are several objectives –

Affordability. The cost of AI query processing has got to be as low as possible in order to keep corporate LLM processing affordable. AI in the corporation becomes untenable if the costs of processing queries rises to a significant level

Access to all kinds of data. Access to both text and classical structured data bases are a necessity. The corporate LLM needs to be able to process not just text, but the core data wrapped up in the structured, transaction based corporate data bases. The processing of corporate structured data bases carries with it an issue. That is the issue of security. SOME, but not all corporate structured data belongs on the public domain of the corporation. But a lot of data found in the corporate, structured data bases does not belong anywhere other than the data base is which it was recorded. So security of data and privacy of data becomes an issue in the bringing of corporate, structured data to the corporate LLM.

Semantic unification of the meaning of data across the organization. The semantics of the corporate AI world need to be aligned with the semantics of the structured world. Without a semantic alignment of data across the corporation the corporation turns into the Tower of Babel. It is noted however that not all data needs to fit beneath the corporate semantic umbrella. Only the core data of the corporation belongs there. The modern world of core corporate data is centralized at the semantic level, not the physical level, as it once was. Without semantic unification of core corporate data, AI coupled with structured systems becomes a Tower of Babel.

The ability to search such sources as the Internet for large documents that may hold relevance to the corporate LLM.

The need to create focus the data found in the corporate LLM. The corporate LLM does not need to become a polyglot of every kind of information in the world. Instead, the corporate LLM needs to be focused on its intent and on its usage. The corporate LLM needs to be dedicated to the business needs of the corporation.

It is hard to say which of these objectives of data management and data governance is more important. The truth is that they are ALL important. They are important in different ways. But they nevertheless are all important.

That then is the objective of data management and data governance in the corporate LLM environment

.PROACTIVE MANAGEMENT

In order to achieve these objectives, it is necessary to have proactive management of the building and maintenance of the corporate LLM environment. The corporate LLM serves as the brain center of the AI processing that takes place in the corporation. Merely finding some raw text and feeding the raw text to the RAG of the corporate LLM is not adequate for the management of the corporate LLM environment. There needs to be thought and purpose to the text fed to the corporate LLM.

There are many ill advised consequences of just feeding unvetted raw text to the private, corporate LLM. It is the job of the data management and data governance organization to protect and focus the corporate LLM appropriately.

CONTRASTING DATA MANAGEMENT/DATA GOVERNANCE IN THE DIFFERENT ENVIRONMENTS

The management and governance of data in the corporate LLM environment is philosophically different from the tasks of classical data management and data governance in the structured environment.

Data in the structured, transaction oriented environment is managed directly. Data models are built. Data bases are designed. DDL is created and so forth.

Management activities are done directly on data in the structured data management/data governance world.

But in the corporate LLM data governance/data management world, activities are done indirectly. This is because AI has taken over many of the tasks that were traditionally done by the data architect by hand. In the world of the corporate LLM, data management/data governance is done indirectly, not directly

.MANAGING THE COST OF CORPORATE AI

The first and most obvious objective of the data architect in the corporate LLM world is that of the need for the management of the costs of doing corporate AI processing. If the costs of processing rise too high, then the cost of analysis starts to become more expensive than the business payback. At this point the corporate LLM and AI infrastructure becomes a liability, not an asset.

The center of the universe of the corporate AI world is the corporate LLM. The corporate LLM takes raw text and converts that text into tokens. The corporate LLM achieves this transition by passing the raw text to a RAG. The tokens produced by the RAG are then used in the analytics of the queries that are sent to the corporate LLM.

The corporate LLM is constantly having more raw text fed to it. As such the corporate LLM is eternally growing. And as the corporate LLM grows, so grow the costs of doing analytical processing against the corporate LLM.

And as the corporate LLM grows, so grows the cost of processing. Each new addition of raw text to the corporate LLM increases the cost of processing the query, or prompt.

The result is – if the data architect wants to keep the costs of processing queries to a reasonable level, the data architect must keep the corporate LLM to as small a size as is workable

.SEPARATING BUSINESS RELATED TEXT FROM BUSINESS IRRELEVANT TEXT

The best way to keep the corporate LLM at a minimal size is to ensure that no extraneous, non business related, irrelevant text is ever submitted to the corporate LLM. In doing so the corporate LLM is kept to the minimum effective size and operates as efficiently as is possible. In addition, the corporate LLM should not be fed repetitive text. Repetitive text only bloats the corporate LLM and makes processing more inefficient.

Fortunately, there is an easy way to accomplish this separation of raw textual data before it enters the corporate LLM. The raw text that has been gathered is passed through textual ETL. Textual ETL analyzes the text. Text that is business related is passed on to the corporate LLM. Text that is not related or relevant to business is ignored and is not passed on to the corporate LLM. In doing so the size of the corporate LLM is kept to a minimum.

In addition textual ETL edits the input to the corporate LLM and ensures that no duplicate data is being fed to the corporate LLM.

The type and format of the document is irrelevant. Textual ETL handles all text types and all formats.

SEPARATING BUSINESS RELATED TEXT FROM BUSINESS IRRELEVANT TEXT

The best way to keep the corporate LLM at a minimal size is to ensure that no extraneous, non business related, irrelevant text is ever submitted to the corporate LLM. In doing so the corporate LLM is kept to the minimum effective size and operates as efficiently as is possible. In addition, the corporate LLM should not be fed repetitive text. Repetitive text only bloats the corporate LLM and makes processing more inefficient.

Fortunately, there is an easy way to accomplish this separation of raw textual data before it enters the corporate LLM. The raw text that has been gathered is passed through textual ETL. Textual ETL analyzes the text. Text that is business related is passed on to the corporate LLM. Text that is not related or relevant to business is ignored and is not passed on to the corporate LLM. In doing so the size of the corporate LLM is kept to a minimum.

In addition textual ETL edits the input to the corporate LLM and ensures that no duplicate data is being fed to the corporate LLM.

The type and format of the document is irrelevant. Textual ETL handles all text types and all formats.

The net effect of passing raw text through textual ETL in order to separate out business relevant text from business irrelevant text first is to greatly reduce the cost of processing. Textual ETL acts as a preprocessor to raw text. Textual ETL filters the raw text and makes the corporate LLM much more efficient and much more effective. In a word, textual ETL focusses the data found in the corporate LLM environment.

FLAB

Another way of understanding the value of removing non business related text from the corporate LLM is to consider that extraneous, non business related text is flab – non productive fat that has to be processed every time a query is issued.

If the cost of processing flab had to be born only a few times when a query is processed, then the cost might not be considered to be high. But the truth of the matter is that flab has to be processed EVERY TIME a query is processed.

As such even a small amount of flab amounts to increased processing costs over time, especially as the volume of flab grows in size

.REDUCING HALLUCINATIONS

But there is another benefit to the removal of flab from the corporate LLM. That benefit is the reduction of hallucinations that might occur in the answers provided by the corporate LLM. By removing text that never belonged in the corporate LLM in the first place, textual ETL reduces the probability of hallucinations that might occur in processing the corporate LLM.

THE CONTEXT CATALOG

One of the more useful reports that is generated by TETL is that of the context catalog. The context catalog contains the words that have been selected as words important or relevant to business from the source documents that have been read. The source of the words are captured at the same time as the word is captured. In addition, the low level context of the word (or the immediate context of the word) is ascertained. Then the high level context is identified.

As an example of the different levels of contextualization, suppose the word “aspirin” is captured in a medical document. The low level context of aspirin might be analgesic. And the high level classification might be medication.

The words and their descriptive assessment of their context are then placed in the context catalog.

The context catalog is useful to data administrator because it gives the data administrator the opportunity to examine and correct the text that is going to be fed to the corporate LLM. This puts the data administrator in a proactive position of managing the corporate LLM. If the word has been incorrectly contextualized, then the data administrator can change the interpretation of the contextualization before it ever enters the corporate LLM.

As an example of what might be found in the context catalog, consider the three entries found in the following figure.

The same word – “fire” - has been found in three different documents. In one document the word fire refers to a conflagration. In another document, the word fire refers to the termination of employment of an individual. In yet another document the word fire refers to the shooting of a gun.

If the proper context of “fire” is conflagration but the system has interpreted it as the termination of an employee, the data administrator has the opportunity to make the correction before the raw text is entered into the corporate LLM by examining the context catalog.

FROM GOVERNING INFORMATION TO GOVERNING MEANING

The objective of Textual ETL and the Context Catalog is to ensure that enterprise AI is provided with trusted, relevant and authoritative information. This represents a significant advancement over traditional approaches in which AI systems consume large collections of unfiltered documents.

However, governing information is only the first step.

Enterprise AI must also understand what that information means.

A corporate AI system does not simply retrieve facts. It reasons about customers, products, suppliers, contracts, employees, financial transactions and countless other business concepts. If those concepts are interpreted inconsistently, AI responses will also become inconsistent.

Traditional data governance has long recognized the importance of establishing common business definitions across the enterprise. Corporate AI extends this requirement. Enterprise meaning must now be governed with the same discipline as enterprise data.

This requires more than managing documents. It requires governing the business concepts, relationships, policies and metadata that define how the enterprise conducts its business.

The Enterprise Semantic Backbone provides the semantic framework through which this semantic unification is achieved. Rather than allowing AI to infer enterprise meaning from generalized public knowledge, the Enterprise Semantic Backbone enables the enterprise to explicitly define how information should be interpreted.

In this way, governance evolves beyond controlling what information enters the AI environment. Governance also determines how that information is understood.

The result is enterprise AI that reasons according to the language, policies and business semantics of the organization rather than relying solely upon generalized knowledge.

THE ENTERPRISE SEMANTIC BACKBONE

One of the greatest challenges facing enterprise AI is semantic consistency.

A Large Language Model possesses an extraordinary understanding of general knowledge. However, it does not understand how an individual enterprise defines its business. Every organization develops its own terminology, business rules and operational relationships.

For example, one organization may define a Customer as an individual, while another defines a Customer as a legal entity. Some organizations distinguish between Customer, Client and Account. Others treat these terms as equivalent. Employees understand these distinctions because they have learned the language of the enterprise. AI has not.

Without explicit guidance, AI is forced to infer enterprise meaning from its general understanding of the world. Governance should never depend upon statistical inference where authoritative business knowledge already exists.

The enterprise must teach the AI.

An Enterprise Semantic Backbone provides the governed semantic framework through which the semantic unification described earlier can be achieved. In many organizations, the Enterprise Logical Data Model provides the foundation for this Semantic Backbone, organizing the business concepts that define how the enterprise operates.

Rather than documenting isolated definitions, the Enterprise Semantic Backbone provides a connected understanding of the enterprise. A business concept such as Customer is understood not only by its definition, but also by its relationships with Orders, Contracts, Products, Suppliers and every other significant concept within the business.

This governed semantic framework provides the semantic unification required for enterprise AI to reason consistently across structured data, documents and enterprise knowledge.

In many organizations, this Enterprise Semantic Backbone is authored and governed using an enterprise data modeling platform such as ER/Studio.

METADATA AS THE FOUNDATION OF AI GOVERNANCE

Metadata has always been fundamental to data governance. Within enterprise AI, metadata becomes even more important because it provides the context required for trustworthy reasoning.

When AI retrieves information, it should not simply determine whether a document contains relevant text. It should also understand the characteristics of that information.

Governance metadata may identify:

· ownership

· stewardship

· security classifications

· regulatory requirements

· confidentiality

· business rules

· lineage

· quality

· lifecycle status

· confidence

This metadata allows AI to distinguish between authoritative information and information that should be treated with caution.

Governance is established against the Enterprise Semantic Backbone, where the meaning of each business concept is precisely defined and unambiguous. Classifications, security policies, regulatory requirements and stewardship responsibilities are associated with these governed business concepts rather than with individual physical data assets. As databases, documents and AI knowledge sources are mapped to the corresponding business concepts, they inherit a consistent set of governance policies across the enterprise.

In this way, metadata becomes the operational mechanism through which governance is consistently applied across enterprise information.

READING THE CORPORATE DATA BASE

Another feature of textual ETL is the ability to go and read data out of a corporate structured transaction oriented data base and prepare it for consumption in the corporate RAG module.

The corporate structured data bases that can be read and transported to the corporate LLM include such data bases as SQL Server, Oracle, Teradata, DB2, Postgres and others.

Data can be read from those data bases and presented to the corporate LLM.

The standard records from the structured, transaction oriented data bases can be read and presented to the corporate LLM in two manners –

1) A profile of the data base in its entirety. In this case the corporate LLM system is presented with such things as total number of records, minimum/maximum values, and so forth

2) The ability to go and access individual records in these structured, transaction data bases and the presentation of these individual records to the corporate LLM.

One of the more important issues in the movement of data from a corporate data base to the corporate LLM is that of privacy and security. For many very good reasons, some data from a corporate structured data base should never be placed in the corporate LLM. Consider a person’s balance of money in a bank account. That sort of information should never be placed in a corporate LLM.

But there are a lot of other kinds of data that are very appropriate for entry into the corporate LLM.

The security for structured, transaction oriented data being passed to the corporate LLM exists at two levels – the system level and the administrative level. If the system protects the data base, then textual ETL does not access it. But even if the data base can be accessed, the administrator can determine which attributes and keys can be accessed by textual ETL and which cannot be accessed

.The ability to read a classical structured, corporate, transaction based data base adds great range to the analytical capabilities of the corporate LLM. Without this capability the range of queries that the corporate LLM environment can process is limited to information that is found in text. Being able to include structured information in the corporate LLM opens up a world of new possibilities for analysis.

There are many important reasons why being able to include corporate, structured, transaction data in the corporate LLM is vital. But the main reason has to be that there is MUCH – if not the majority – of corporate, structured, transaction data that contains business value. While there indeed is data that contains business value in text, the details of the day to day running the organization and the financial activities of the corporation are found in the corporate, structured, data bases. For this reason alone, it is vital that the data be included in the corporate LLM.

Stated differently, if the corporate LLM does not include appropriate data from the structured data base environment, then the corporate LLM is only capable of providing a limited amount of business value

.As an example of what a transition from a standard structured data base to the corporate LLM might look like, suppose the analyst has done a search for the banking record for Bill Inmon.

The banking record is in an Oracle data base.

The record is found. The keys and the attributes of the record are accessed. The system determines which attributes can be accessed and which cannot be accessed. The values that can be accessed are then turned into text, which is then fed to the corporate LLM. The administrator protects the attributes that need protection. Only the attributes that can be placed in the corporate LLM are allowed in.

DOCUMENT SEARCHES

Yet another capability of textual ETL is that of being able to search the Internet and find documents that are candidates for providing text to the corporate LLM.

In this case the analyst would provide the name that is of interest. For example, the analyst might choose Ford Motor Company. The Internet gathering program would then go look for as many occurrences of Ford that can be found on the Internet. The search of the Internet is done not just on the name “Ford”, but on the many permutations of Ford as are known.

When found, the documents on the Internet are then secured and are made available to textual ETL.

Textual ETL then uses the document as input and separates business related data from non business related data, and passes the business related data off to the corporate LLM

.It is noted that in the case of Ford Motor Company that permutations of the name Ford Motor Company – Ford, FMC, etc. are searched

.By passing the documents that have been found through textual ETL, the effect is a funnel effect. First the entire document is accessed, then the business related text is squeezed from the document

.

These then are a few of the tools needed for management and governance of the corporate LLM environment that are provided by LLM MGMT and ERStudio

.The tools for doing data administration and data governance are shown

in the diagram.

THE FOUR PILLARS OF CORPORATE AI GOVERNANCE

Traditional data governance focuses primarily on structured data. Enterprise AI significantly expands the scope of governance. Information used by AI now includes structured databases, unstructured documents, external information sources, enterprise policies and business knowledge. As a result, corporate AI governance must address four related domains.

Enterprise Data

Data quality, ownership, lineage, security and regulatory compliance continue to provide the operational foundation for enterprise AI.

Enterprise Knowledge

AI consumes far more than structured data. Policies, procedures, contracts, reports and technical documentation all become enterprise knowledge that must be governed before AI can consume it.

Enterprise Meaning

Enterprise AI must understand the meaning of enterprise information. The Enterprise Semantic Backbone provides the semantic unification that allows business concepts to be interpreted consistently across structured data, documents and AI knowledge sources.

AI Interaction

Governance extends beyond information to the operation of AI itself. Organizations must govern prompts, retrieval strategies, approved knowledge sources, response validation and continuous improvement.

Corporate AI governance therefore governs not only enterprise information, but the complete environment in which enterprise AI learns, reasons and responds.

GOVERNANCE ROLES AND RESPONSIBILITIES

Corporate AI governance introduces responsibilities beyond traditional data stewardship.

Organizations must determine which documents are authoritative, which external information sources are trusted, who approves enterprise business definitions, who validates AI-generated responses and how semantic consistency is maintained across structured and unstructured information.

These responsibilities require close collaboration between business experts, data architects, governance professionals and AI practitioners.

Corporate AI governance therefore becomes a multidisciplinary responsibility rather than solely a data management function.

GOVERNING THE AI KNOWLEDGE LIFECYCLE

Corporate AI governance is not a one-time implementation effort. Enterprise knowledge continually evolves as new information is created, business policies change, regulations are updated and products evolve. Governance must therefore be viewed as a continuous lifecycle.

Knowledge is acquired from internal and external sources and evaluated through the Textual ETL process. Relevant information is incorporated into the Context Catalog, while business experts validate its meaning and context. Information is then mapped to the Enterprise Semantic Backbone, where governance policies, classifications and stewardship responsibilities are consistently applied.

Governance continues after deployment. Enterprise knowledge changes over time, requiring new information to be incorporated, obsolete information to be retired and AI responses to be continually monitored to ensure they remain accurate and aligned with current business practices.

Corporate AI governance is therefore a continuous process of acquiring, governing and maintaining enterprise knowledge so that AI remains trustworthy as the business evolves.

MEASURING THE SUCCESS OF AI GOVERNANCE

Like all governance programs, corporate AI governance must demonstrate measurable business value.

The effectiveness of corporate AI governance should be measured not only by the quality of AI responses, but also by the maturity, adoption and business value of the Enterprise Semantic Backbone that supports enterprise AI. As the Enterprise Semantic Backbone becomes a trusted enterprise asset, it enables consistent governance, the uniform application of security and regulatory policies, improved interoperability and more reliable AI outcomes across the organization.

Examples include:

· Reduction in AI hallucinations.

· Reduction in token consumption through Textual ETL.

· Increased use of authoritative enterprise knowledge.

· Improved consistency of AI responses.

· Increased coverage of enterprise information mapped to the Enterprise Semantic Backbone.

· Increased reuse of governed business concepts across applications, analytics and AI initiatives.

· Increased confidence in AI-generated recommendations.

· Faster access to trusted enterprise information.

· Reduced risk of exposing sensitive information to unauthorized users or AI applications.

· Improved business adoption of enterprise AI.

Successful governance does not restrict AI. It enables enterprise AI to operate safely, consistently and with confidence across the organization.

IMPLEMENTING CORPORATE AI GOVERNANCE

Textual ETL can read and accept raw text from any source in any language and in any format. The raw text is read and separated. Business irrelevant text never finds its way into the corporate LLM.

The Internet can be scanned and documents can be collected and sent to textual ETL. The document is read and business irrelevant data is removed from the stream of text entering the corporate LLM.

Corporate transaction data bases can be read and data that is appropriate can be read from the data base and passed to the corporate LLM. In doing so the type of data that can be encompassed by the corporate LLM is greatly enhanced.

The data, its context and its discovered business semantics can then be reviewed and incorporated into the Enterprise Logical Data Model maintained within ER/Studio, where they become part of the organization’s governed Enterprise Semantic Backbone.

The corporate context catalog is created. The data administrator then has the chance to correct and edit the text before it is passed to the corporate LLM.

ER/Studio is a mature enterprise data modeling platform that enables organizations to develop and maintain an Enterprise Logical Data Model (ELDM). The ELDM provides a business-oriented representation of the organization’s core concepts, their definitions and their relationships, creating a common understanding of enterprise information.

Traditionally, organizations have used the ELDM as the foundation for designing and documenting databases, ensuring that business definitions and data structures remain consistent across projects and technology platforms. As enterprise AI has emerged, the role of the ELDM has expanded beyond traditional data modeling.

Today, many organizations are using the Enterprise Logical Data Model as the foundation of their Enterprise Semantic Backbone. Business concepts defined within the ELDM become the authoritative semantic framework that supports analytics, data integration, governance and enterprise AI. ER/Studio can then publish these governed business definitions into the semantic layers required by modern data platforms, enabling consistent interpretation across data catalogs, knowledge graphs, AI frameworks and other semantic technologies.

Its business-oriented graphical representations and collaborative capabilities enable business subject matter experts, data architects and governance teams to work together on a single enterprise model. The result is a governed semantic asset that supports semantic unification across the organization while providing the authoritative business context required for trusted enterprise AI.

ER/STUDIO AND TEXTUAL ETL

Textual ETL and ER/Studio perform complementary roles within the enterprise AI architecture.

Textual ETL discovers and contextualizes enterprise knowledge, while ER/Studio governs the enterprise meaning of that knowledge through the Enterprise Semantic Backbone. During this process, the extracted information is associated with the governed business concepts defined within the Enterprise Semantic Backbone. This enables enterprise knowledge to inherit the business meaning, governance policies and security classifications associated with those concepts.

At the same time, the discovery of new terminology, business concepts and relationships during Textual ETL provides valuable input into the ongoing evolution of the Enterprise Semantic Backbone. Business experts can review this newly discovered knowledge, determine whether it represents new enterprise concepts or refinements to existing definitions, and incorporate approved changes into the Enterprise Logical Data Model.

In this way, Textual ETL continuously enriches enterprise knowledge while the Enterprise Semantic Backbone provides the authoritative semantic framework through which that knowledge is interpreted. Together they form a closed governance cycle in which enterprise information improves the semantic model, and the semantic model improves the interpretation of enterprise information.

CONCLUSION

Enterprise AI represents a fundamental change in the responsibilities of data management and data governance. In the structured world, governance focused primarily on managing data. In the AI environment, governance must also manage enterprise knowledge, business meaning and the way AI interprets information.

Simply loading large volumes of enterprise information into an AI environment is not sufficient. The information must first be filtered, contextualized and governed. Textual ETL ensures that enterprise AI is provided with trusted, relevant and authoritative information, while the Context Catalog enables that information to be reviewed and managed before it becomes part of the AI knowledge base.

However, trustworthy enterprise AI requires more than authoritative information. It also requires authoritative meaning. The Enterprise Semantic Backbone provides the governed semantic framework through which business concepts are consistently defined and interpreted across structured data, documents and enterprise knowledge. Together, these capabilities provide the semantic unification required for enterprise AI to reason according to the language and business rules of the organization.

The result is an enterprise AI environment that is more efficient, more secure and more trustworthy. Governance is no longer limited to controlling data. It extends to governing the knowledge and meaning upon which enterprise AI depends. As organizations increasingly rely upon AI to support critical business decisions, this combination of trusted information and governed semantics will become an essential capability for every enterprise.

As enterprise AI continues to evolve, organizations that govern both information and meaning will be best positioned to build AI systems that are trusted, explainable and aligned with the business.

ER/Studio is an Idera company. You can find out more about ER/Studio at www.erstudio.com

LLM MGMT builds and manages textual ETL. You can find out more about textual ETL on the web site www.llmmgmt.com.

By W H Inmon

With contributions from Jamie Knowles, Jamie Knowles @semanticsarethesolution

Data management has travelled a long road from the beginning of the computer industry to the day and age of generative AI. Data management has been elevated from the role of simple technician to the role of great strategic importance in the world of generative AI.

That path has had many steps and missteps along the way. But today data management arrives at the position of being crucial to the success and adoption of generative AI in the corporation. If AI does not operate on believable data, then any analysis done by AI is questionable.

IN THE BEGINNING…

In the very beginning, systems were built by application developers. The developer had many tasks – requirements gathering, methodology conformation, programming, testing and the like. One of the tasks the early application developer had was the task of defining the data on which the system would run.

The task of designing the data and the structure of the data for the application system being built was but one of the many tasks of development.

TRANSACTION PROCESSING APPLICATIONS

The world progressed from early simple systems to transaction processing systems. With transaction processing systems came a whole new world of response time, system availability, transaction execution integrity and so forth. What was needed was a data base designer who could incorporate all of these needs into the data base design.

With transaction applications, the life of the data administrator became quite a bit more complex.

THE FLOOD OF TRANSACTION PROCESSING APPLICATIONS

And soon there was a plethora of transaction processing systems – a flood. It was recognized that there was a need for control and coordination across the many transaction processing data bases that were being built.

What was needed was a data base administrator.

DATA INTEGRITY

And with the growth of the many data bases came the realization that data integrity was an issue. In the spiders web of applications systems that resulted, the same element of data appeared in many places, each with a different value in each place. No longer did the organization have the problem of having data. Now the organization had the problem of believing what data existed. In one place the data showed one value. I another place the data showed another value. And resolving this discrepancy was no small task.

Furthermore, adding more application systems and more technology only exaggerated and accelerated the problem of lack of data integrity

.A technical solution was not needed. What was needed was an architectural solution.

AN ARCHITECTURAL SOLUTION

The architectural solution that resulted was the separation of operational data from analytical data. Thus born was the data warehouse.

THE NEED FOR A DATA MODEL

In order to achieve the architectural solution, the world discovered that it need a data model. The data model was an abstraction of the types of data that existed and the data that needed to exist in the future.

The role of the CIO – chief information officer - appeared.

The world of operational systems and data warehouses was a world that was exclusively a structured world. In a structured world each record format is identical in a data base. Only the contents of the record differ from each other. All applications, data bases, and transaction processing data bases were based on structured data. The structured data was managed by a data base management system – a dbms.

The world of data management changed dramatically with the movement to data warehousing.

And with the tremendous influx of new transaction processing systems came the recognition that data management was an important function. With transaction processing systems came a new role for systems. With transaction processing systems, when the system was malfunctioning, the business and its customers were directly and negatively affected. ATM machines did not work. Airline reservation systems would not function. It became very obvious to the corporation at the business level that reliable, accurate systems were needed, and those systems ran on data.

The enhanced role of the importance of transaction processing systems elevated the value of data administration and management to the organization.

THE ELEMENTS OF DATA MANAGEMENT IN THE STRUCTURED ENVIRONMENT

So what were the elements of data administration in the day of the chief data officer?

One function was that of managing the data model. Not only did the data model need to be built, but the data model needed to be maintained and altered as businesses changed. And every business was subject to constant change. There were changes in economic conditions. In competition. In new technologies. In legislation and market conditions. The business of every corporation changes over time.

And as these changes to business occurred, the data model needed to be kept abreast of those changes.

A related task of the structured data manager is the task of conducting data base design. Data base design – in its infancy - started by conforming to the rules of normalization. However, the performance requirements of transaction processing mandated that some amount of denormalization of data be done in order to achieve the performance objectives of the transaction processing system..

The data manager is responsible for the balancing of the goals and needs for normalization and denormalization of data.

Another task of the data manager in the structured environment is that of constructing the DDL that defines the data base. The DDL is patterned after the data model.

Capacity planning is another task of the data manager. Capacity planning is especially important in transaction processing. The organization had a need to know when the limits of a machine were going to be reached.

Monitoring both data base activity and data base growth is another ongoing activity of data management in the structured environment.

THE ROLE OF THE DATA MODEL

The data model is a necessary component of the transformation of application data to analytical data, or the data warehouse.

DATA MANAGEMENT IN THE STRUCTURED ENVIRONMENT

In many respects data management in the structured environment is very much a hands on experience. When the data manager in the structured environment sees something that needs an amendment, the data manager goes directly to the problem and fixes it.

ENTER UNSTRUCTURED/TEXTUAL DATA

As interesting and as important as structured data is in the corporation, structured data is hardly the only kind of data found in the corporation. Another pervasive data type is unstructured or textual data. It is estimated that as much as 90% of the data in the corporation is textual data. Furthermore, there is much very important data that is wrapped up in the form of text.

Structured data is entirely different from unstructured/textual data.

And textual data – like structured data – needs to be managed by the data management organization

.THE TEXTUAL DATA MODEL

There is no classical data model that fits well with the textual environment. Instead, the equivalent of the classical data model is the ontology/taxonomy. Stated differently, the ontology/taxonomy model fits the textual environment just as the classical data model fits the structured environment.

However analogically similar they are, the classical structured data model is VERY different from the ontology/taxonomy.

A typical mistake made by data management when they first engage with the world of text is to try to apply the skills needed for classical data modelling and data management in the structured world to the world of text.

Such an misapplication of skills is a serious misfit and yields very few actual positive results

.DATA MANAGEMENT IN THE GENERATIVE AI ENVIRONMENT

So what are the tasks for data management in the textual environment? The first and most important task is to ensure that raw text before it enters the LLM is vetted and that extraneous, non business relevant text does not find its way into the LLM. If the data manager does nothing else, the insurance that no extraneous text ever enters the LLM is sufficient.

The removal of extraneous, non business related data from the stream of data entering the LLM is important because of several reasons –

1) Removing extraneous text from the stream of text entering the LLM makes the LLM much more efficient to process against. The savings are in hard core dollars that are saved every time a query is issued.

2) Removing extraneous, non business related text provides focus for the LLM. It is clear what the LLM represents.

THE ELDM – ENTERPRISE LOGICAL DATA MODEL

Another task of the data manager for the generative AI environment is that of marrying the ontology/taxonomy environment with the larger corporate ELDM. The ELDM – enterprise logical data model – is necessary to meld the classical data model with the ontology/taxonomy.

In addition the data manager of the generative AI environment needs to keep the ontology/taxonomy data model up to date, as business needs change.

BROAD SWATHS OF DATA

Yet another task for the data manager for the generative AI environment is that of the selection of broad swaths of raw text from sources such as the Internet. This selection process sets the stage for the secondary process of selecting business relevant data from the text presented to the LLM.

ONGOING MAINTENANCE

The data manager for the generative AI environment has got to do constant surveillance and management of the ontology/taxonomy environment. Business and world conditions are constantly changing. The ontology/taxonomy needs to be kept in synch with these changing conditions.

INDIRECT CONTROL OF THE LLM

Data management in the generative AI environment is much like the manipulation of a puppet. The data manager in the generative AI environment does not have the direct control that the data manager in the structured environment has. Instead, the data manager in the generative AI environment has indirect control. The indirect control is the management of what text is fed the LLM.

The following figure shows the difference between data management in the structured environment and data management in the generative AI environment. One type of control is direct. The other type of control is indirect.

THE DATA MANAGER IN THE ORGANIZATION CHART

In the beginning data management was merely one more activity that the system developer had to tend to. There was no specific corporate function for data management in the beginning.

When transaction processing became a reality, it was recognized that data base design was its own activity (that of course had to integrate with other activities of development.) Thus evolved the role of the data base designer. The data base designer was the first formal recognition of the corporation of the need for data management.

Soon there were LOTS of data bases, as the spiders web environment grew. From the spiders web environment grew the recognition of the need for a data base administrator. There were simply too many data bases and too many elements of data to be managed. A larger corporate role was recognized.

Soon the confusion over the disintegrity of data in the corporation that was found in the spiders web environment led to the need for the chief information officer, or the CIO.

And finally the need to integrate textual data with structured data to have a true corporate understanding of data led to the CDO – chief data officer.

With the advent of the many forms of data, there grew great confusion and great diversity of opportunities and challenges with data. The job of managing grew both in complexity and in importance.

And the corporation recognized that need.

THE CORPORATE PROGRESSION

The progression of the role of the data administration is shown in the following chart.

CENTRALIZATION

One of issues of data management that has radically changed in the transition from early systems to the world of generative AI is the whole idea of centralization of data. In order to achieve a cohesive enterprise understanding of data and its meaning, it is mandatory that there be a centralized understanding of data.

But the whole notion of what is meant by centralization has also changed.

In the very early days of computing centralization referred to the physical centralization of data. Data bases were physically centralized into large mainframe computers.

But as time passed, physical centralization became an impossibility.

Data was on personal computers. Data was on the Internet. Data was on corporate systems. Data was in fact – everywhere.

And with the diaspora of data, came an even greater need for centralization. But the impetus for centralization took place not at the physical level but at the semantic level.

What someone was referring to on the Internet needed to be synchronized with what was being said in the corporation, for example.

The result was that centralization took place at the semantic level, not the physical level.

The vehicle for shepherding the semantics of data became the ELDM – enterprise logical data model.

The ELDM held the enterprise understanding of data centrally, where the data resided in many physical locations.

Bill Inmon lives in Denver with his wife and his two Scotty dogs – Lena and Rollie. Lena got to go to the art fair in Evergreen last weekend. She came home and told Rollie all about it. He was jealous. But he will get to go on the next adventure up into the mountains. Then Lena will be jealous.

LLM MGMT, LLC – a Bill Inmon company - is a company for the preparation of raw text to enter the corporate LLM. LLM MGMT separates business related text from extraneous text before the text is entered into the LLM. LLM MGMT allows text to be gathered automatically from a wide variety of sources in order for the text to be entered into the corporate LLM. LLM MGMT allows structured data bases to be read and the data that is gathered is entered into the corporate LLM.

For more information, see the web site – llmmgmt.com.

By W H Inmon

Marcel Dybalski

What is the value of “100”?

On a spreadsheet, “100” is innocuous. It sits quietly in a cell. But what does it mean? “100” can be the age of a centenarian. “100” can be a perfect score on an exam. “100” can be the speed at which a vehicle becomes dangerous. Or, “100” can be the price of a stock just before it crashes.

Without context, the number “100” is not data; it is merely digital noise. To transform that number into information, and that information into wisdom, you must have the metadata (the context) surrounding it.

For decades, organizations have treated metadata as an afterthought. It was the “documentation” that developers promised to write after the project went live (a promise rarely kept). But the landscape has shifted. We have moved from an era of data scarcity to an era of data deluge. We are no longer just counting rows in a ledger; we are ingesting customer service emails, analyzing contracts, and scraping web logs.

In this environment, metadata matters more than ever because the distance between the data producer and the data consumer has widened. The analyst in New York does not know the engineer in Bangalore who built the table. Without explicit, rigorous metadata, the analyst is guessing. And in high-stakes business, guessing is a liability.

The Cost of Silence

When metadata is ignored, the modern data platform becomes a liability rather than an asset. This is not merely a technical inconvenience; it is a financial drain.

There are hidden taxes paid by organizations that neglect metadata. One tax is speed. How long does it take a data scientist to find the right dataset? In a metadata-poor environment, the vast majority of an analyst’s time is consumed by searching for data and verifying its origin, leaving very little time for actual analysis.

Another tax is trust. If the Finance department defines “churn” as a cancellation within a specific short window, and the Marketing department defines “churn” as a failure to renew over a longer period, the resulting reports will contradict one another. When executives see conflicting indicators, they stop trusting the data entirely. They revert to gut feeling. The investment in the data architecture is effectively wasted.

But perhaps the greatest cost lies in the inability to process text. Text is fraught with ambiguity. Structured data is rigid; textual data is fluid. If you ignore metadata in unstructured text, you are not just missing context: you are missing the sentiment, the negation, and the intent. You are blind to the customer who says, “I am not happy,” because your system only searched for the word “happy.”

Scope of This Monograph

We will show how properly designed metadata stops being a layer of description and becomes the engine that generates schemas, orchestrates pipelines, enforces policies, and powers impact analysis.

The end state is simple and radical: the human no longer maintains the data architecture. The metadata does.

When that inversion happens, the platform finally becomes trustworthy, fast, and adaptable at enterprise scale.

That is the promise we intend to deliver.

Types of Metadata

To build a data architecture driven by metadata, we must first recognize that metadata is not a monolith. It is a spectrum.

Different questions require different answers. A search engine asks, “What is this file about?” A compliance officer asks, “Who is allowed to see this?” A rendering engine asks, “What is the resolution?”

If we attempt to answer all these questions with a single, flat list of tags, the architecture fails. We must instead divide our metadata into five distinct domains, each governing a specific aspect of the data’s lifecycle.

Descriptive Metadata

The Context of Discovery

Descriptive metadata is the label on the jar. It provides the essential attributes, like: title, author, keywords, and summary, that allow an asset to be found within a vast repository. Without descriptive metadata, a file exists, but it is effectively invisible.

Consider a streaming service like Spotify. The platform’s library hosts tens of millions of audio tracks. Descriptive metadata converts a raw audio file into a discoverable piece of art by tagging it with “Composer”, “Duration”, and “Album Art”. This allows a listener to query the interface for “Jazz Instrumentals from 1959”, slicing through the silence to locate a specific sound.

Structural Metadata
The Context of Organization

If descriptive metadata is the label, structural metadata is the skeleton. It defines how data elements are organized, hierarchically arranged, and related to one another.

Data rarely lives in isolation; it lives in compounds. On a website, structural metadata dictates that a specific page is a child of the “Home” page and a sibling of the “About Us” page. It functions like a table of contents in a book, guiding the system (and the user) through the complexity of the whole.

This structure is the domain of Taxonomy. A global retailer cannot function if one region classifies “Mobile Phones” under “Utilities” while another classifies them under “Consumer Electronics.” Structural metadata enforces order upon entropy. It ensures that “T-shirts” always reside beneath “Clothing”, allowing for consistent aggregation across the enterprise. It turns a pile of pages into a coherent narrative.

Administrative Metadata

The Context of Governance

Administrative metadata is the rulebook. It is less concerned with what the data is and more concerned with how the data may be used.

This layer details ownership, access rights, and the legal lifecycle of the asset. It answers critical questions: Who created this record? Who is authorized to modify it? How long must we legally retain it before deletion?

In a regulated environment, administrative metadata is the primary defense against liability. It creates a chain of custody, recording who accessed a file and when. It manages retention schedules, ensuring that sensitive data is purged in compliance with privacy laws (like GDPR or CCPA) or retained for tax purposes. It is the bridge between the data architecture and the legal department.

Technical Metadata

The Context of Mechanics

Technical metadata describes the physics of the digital asset. It captures the file type, encoding standard, storage location, and resolution.

For a data platform to process a file, it must know the technical specifications. An image processing pipeline needs to know if an asset is a JPEG or a PNG, its color profile, and its pixel dimensions to display it correctly. A data engineer needs to know if a dataset is delimited by commas or pipes.

Technical metadata ensures interoperability. It provides the crucial instructions that allow different systems: content management systems, cloud storage buckets, and analytics engines, to handshake and interpret data accurately. Without this layer, a file is not information; it is a meaningless stream of bytes that the machine cannot digest.

Preservation Metadata

The Context of Continuity

Data is fragile. Formats rot; software becomes obsolete. Preservation metadata is the insurance policy against the passage of time.

This type of metadata captures the information required to ensure the long-term usability of an asset. It tracks the history of backups, the checksums for data integrity, and the strategies for migration.

Consider a healthcare organization transitioning from a twenty-year-old legacy system to a modern Electronic Health Record. The clinical data is worthless if it cannot be read by the new system. Preservation metadata guides the porting process, ensuring that patient history survives the technological shift. In industries like legal services and healthcare, where data retention is measured in decades rather than days, preservation metadata is the only thing standing between an archive and a digital graveyard.

Lineage and Dependency GraphsData is rarely static. It flows. It transforms. It mutates.

A report on the CEO’s desk shows a dip in revenue. Why? To answer that, you must trace the river back to the source. This is Data Lineage.

Most organizations operate in a “black box” environment. Raw data enters a pipeline, magic happens, and a dashboard appears. But what happened inside the box?

Did a filter remove specific transactions?

Was a currency conversion applied using yesterday’s rate or today’s?

Did a join fail, resulting in dropped records?

Metadata must capture the movement. It is not enough to know what the data looks like now. You must know what it looked like then, and exactly which logic altered it.

The Dependency Graph is the result of this metadata. It is the visual representation of fragility. If I change the schema of the INVOICE table, what breaks? Which reports fail? Which machine learning models drift? Without dependency metadata, every schema change is a game of Russian Roulette.

Quality Rules and Thresholds

How do you know if your data is lying to you?

You cannot manually check a billion rows. You rely on trust. But trust in data architecture is not a feeling; it is a metadata construct.

Quality is not a binary state of “good” or “bad”. It is a measurement against a standard. Those standards are metadata. Examples:

● Completeness: “This column must be populated 100% of the time.”

● Validity: “This date cannot be in the future.”

● Consistency: “The sum of the line items must equal the order total.”

These rules should not be buried in the code of an ETL script. They must be elevated to first-class citizens in the metadata repository. By treating quality rules as metadata, you decouple the definition of quality from the execution of the check.

When the metadata says, “This field requires a valid email format” and a row arrives with “N/A”, the system raises a flag. This is the difference between a silent failure (where bad data pollutes decision-making) and a managed exception.

In structured systems, data creates value; metadata makes it trustworthy.

Metadata for Structured Data

There is a comforting illusion found in the grid.

When an analyst looks at a spreadsheet, or a developer looks at a database table, they feel a sense of control. There are rows. There are columns. Every cell holds a value. It looks orderly. It looks defined.

But looking is not understanding.

Just as a single word is often meaningless without a sentence, a number in a database column is meaningless without metadata. What is the number 42? Is it an age? A price? A street number? The answer to the ultimate question of life, the universe, and everything?

Without the metadata wrapper, structured data is merely a pile of digital bricks without a blueprint. It has weight, but it has no purpose. To build an architecture that functions, we must peel back the layers of structure and manage the metadata that holds the grid together.

Classical Constructs: Schemas, Keys, and Relationships

The most fundamental form of metadata for structured data is the schema. It is the physics engine of the database world. It dictates what can exist and how it occupies space.

But simple column names are rarely enough.

Consider the data type. The distinction between an integer and a decimal is not just a technicality; it is a business rule encoded in code. If you store a price as an integer, you have implicitly decided that cents do not matter. That is metadata.

Then there is the concept of identity. The Primary Key.

This is the DNA of a record. It asserts that this specific row is unique and distinct from all others. But what defines that uniqueness? Is it a generated system ID? Is it a social security number? The metadata must capture the nature of the key, not just the value.

And what of the Foreign Key?

Data does not live in isolation. A customer exists in relation to an order. An order exists in relation to a product. The Foreign Key is the metadata construct that binds these entities together. It creates the web of logic. If this relationship is not explicitly defined in the metadata repository, the database is nothing more than a collection of disconnected islands. You cannot navigate from the sale to the salesperson without the bridge.

Data Dictionaries and Glossaries

There is a serious gap between what IT builds and what the Business expects. This gap is where projects go to die.

The Data Dictionary is the technical map. It tells the engineer that the column CUST_LTV exists in the T_SALES table, is a FLOAT, and allows NULL values. It is the physical reality.

The Business Glossary is the semantic truth. It defines what “Customer Lifetime Value” actually means. Does it include returned items? Does it calculate based on the last 12 months or the inception of the account?

Here is the obstacle: Homonyms.

Marketing defines a “Customer” as anyone who has visited the website. Finance defines a “Customer” as anyone with a paid invoice. Support defines a “Customer” as anyone with an active ticket.

Three departments. One word. Three different meanings.

If your metadata architecture does not explicitly map the Business Glossary term to the specific Data Dictionary element, you will have executives arguing over reports that show three different totals for the “same” metric. The metadata must serve as the Rosetta Stone, translating the dialect of the database into the language of the boardroom.

Metadata for Textual Data
Like all complex environments, a model greatly helps in the management and understanding of the complex environment. And the world of text is no exception. In many ways the world of language and text is even more complex than the environments that hold and manage data.

The model that is useful for the management of text is the ontology/taxonomy model.

An ontology is a set of taxonomies. The ontology has a general subject area. Typical ontology subject areas might be the airlines industry, the banking industry, the telecommunications industry. Or an ontology might be fore the celebration of holidays, the country of Brazil, or medical research.

There is no limit as to what the theme of an ontology might encompass.

In the case of examining and managing text for business, the ontologies naturally focus on businesses.

The ontology is made up of taxonomies. Each of the taxonomies describes some one aspect of the ontology. For example, the ontology for a bank might contain the taxonomies for such things as –

Credit cards

Loans

Savings

Accounting

Security

And so forth.

Or the ontology for an airline might focus on taxonomies that describe such things as –

Flights

Airports

Gate agents

Reservation systems

Cabin crew

And so forth.

Each taxonomy is focused on its own subject area.

The taxonomy is nothing more than a collection or related words. The taxonomy for credit card might have vocabulary for such things as –

Card holder

PIN number

Credit limit

Credit card number

Cardholder rating

And so forth

The elements of a taxonomy may or may not be connected. For example, there may be a taxonomy called country and another taxonomy called state. The states of Texas, New Mexico and Utah would all relate to the country – USA. But there are no states in Japan, for example.

The world of ontologies and taxonomies is very, very different from the world of data models. Both of the types of models are necessary. But the similarities between the different models ends there.

Central Metadata Model and Repository

If you walk into a library and throw the books on the floor, you still have the content. You have the knowledge. But you do not have a library. You have a pile of paper.

Most data environments today are piles of paper.

The ETL tool knows how data moves. The database knows how data is stored. The BI tool knows how data is visualized. But nothing knows the whole story. To move from a chaotic pile to a functioning architecture, you need a unifying structure. You need a Central Metadata Model.

This section outlines how to organize the invisible layer of your enterprise and the conceptual architecture required to support it.

Core Entities

Eight core entities are sufficient to describe and govern the essential semantics, lineage, and control surface of a modern data platform.

1. Asset - the physical or logical thing (table, file, report, model, API, dashboard, document, topic, email, contract)

2. Business Term - the canonical business definition (concepts like “Gross Revenue”, “Active Customer”, “Churn”)

3. Definition - the technical implementation of a business term (formula, JSON path, or NLP entity)

4. Rule - quality, masking, retention, access, tagging, or classification rule

5. Policy - the enforceable business or legal directive (“PII must be masked for non-production environments”)

6. Flow - pipeline, directed acyclic graph (DAG), notebook, or query model (the executable unit of transformation)

7. Person/Role - owner, steward, custodian, consumer

8. Classification - sensitivity level, domain, criticality, lifecycle stage

Other concepts in the data universe are expressed as a relationship between these eight atoms.

A table is an Asset.

A column that calculates Gross Revenue is a Definition linked to the Business Term “Gross Revenue”.

A rule that says “Gross Revenue must never be negative” is attached to that Definition.

A policy that says “Gross Revenue can only be seen by Finance” is attached to the same Definition.

Relationships Between Metadata Objects
The power is not in the entities.

The power is in the directed, versioned graph that connects them.

Every relationship is stored explicitly as a directed edge, and the graph allows traversal in both directions.

Asset → contains → Definition

Definition → implements → Business Term

Business Term → owned by → Person

Flow → reads → Definition → writes → Definition (Flows transform existing Definitions into new Definitions.)

Rule → attached to → Definition/Asset

Policy → enforced by → Rule (Policies exist independently; Rules operationalize them.)

Asset → classified as → PII → triggers → masking Rule (Classifications are descriptive by default; when linked to Rules, they trigger enforcement actions like masking.)

Because these relationships are stored as first-class objects in the graph, the repository can answer questions that previously required weeks of tribal archaeology:

“Who owns the definition of Churn?”

“Show me every dashboard that would return incorrect results if we change the revenue Definition from gross to net.”

“Which pipelines will be affected if new regulations force us to delete records older than seven years?”

The graph answers almost instantly.

Versioning and History: The Fourth Dimension

History is slippery.

What was the definition of “Sales Territories” in 2019?

It is likely different from the definition of “Sales Territories” today.

If you run a report on 2019 data using today’s definition, you will get the wrong answer. You will judge the past by the rules of the present.

Therefore, the Central Metadata Repository must be versioned.

It must capture the history of the metadata itself. It must be able to say:

“From January 1, 2018 to December 31, 2020, ‘Revenue’ included shipping costs. Starting January 1, 2021, ‘Revenue’ excludes shipping costs”.

Without versioning, the analyst is blind to the changes of the past. They see the data, but they do not possess the context of time.

Integration with Catalogs and Glossaries
The central model does not replace the commercial catalogs, quality monitors, or transformation engines.

It colonizes them.

Every external tool becomes a thin presentation layer sitting on top of the central graph.

They are allowed to exist, but only as read-only mirrors.

When a data steward updates a business term in a catalog interface, the change is immediately rejected unless it is pushed through the central repository’s approval workflow.

Once approved in the center, the new version is pushed out to every tool simultaneously.

This ends the religious wars about which tool is the source of truth.

There is only one source.

Everything else is a consumer.

The same principle applies to transformation frameworks, quality testing suites, and access control platforms.

They all read from and write to the same immutable graph via APIs.

When this repository exists, metadata finally stops being documentation.

It becomes the living, executable nervous system of the enterprise.

And the architect’s job changes forever: from designing pipelines to designing the graph that builds the pipelines.

Driving Architecture from Metadata
In the past, metadata was always the final step. Engineers built the pipelines, created the tables, and wrote the code. Then, if they had time, they wrote down what they did.

This approach is backwards. It means the documentation is always chasing reality.

In a modern platform, we flip the script. Metadata comes first. It does not just describe the architecture; it generates the architecture.

This represents the shift from Passive Metadata (a catalog you look at) to Active Metadata (a configuration the system reads). By treating metadata as the source code of the platform, we eliminate manual work, reduce human error, and ensure everything stays consistent and up to date.

Schema Generation from Metadata Definitions

Writing CREATE TABLE statements by hand is a liability. It invites inconsistency. One engineer defines a timestamp as DATETIME; another defines it as TIMESTAMP_NTZ. One names a customer identifier cust_id; another names it customerId.

When architecture is driven by metadata, the engineer does not write SQL. They write a declarative definition (often in YAML or JSON) that describes the entity.

For example, the metadata definition states:

● Entity: Customer

● Attribute: Email

● Type: String

● Nullable: False

A continuous integration process reads this definition and automatically executes the precise DDL commands required to create or update the table in the target data warehouse. If the metadata definition changes, the system detects the drift and automatically applies the necessary ALTER TABLE command. The database schema becomes a physical manifestation of the metadata contract.

Pipeline Orchestration Based on Dependencies

The most brittle architectures rely on time-based scheduling. Job A runs at 01:00. Job B runs at 02:00. If Job A takes 65 minutes, Job B processes incomplete data.

Metadata-driven orchestration rejects the clock in favor of the graph.

Because metadata captures lineage (the knowledge that Table B depends on Table A) the orchestration engine can dynamically generate the execution path. The system does not run a job because it is 02:00; it runs the job because the metadata signals that the upstream dependencies have successfully refreshed.

This approach allows for dynamic backfilling. If a business rule changes for a metric calculated three steps upstream, the metadata engine identifies every downstream dependency. It can automatically trigger a re-computation of historical data for exactly those tables affected, without human intervention to map out the dependency tree.

Policy Enforcement via Metadata

You cannot trust human memory for security.

“Always hide the Social Security Number”. This is a simple rule. But developers get tired. They forget. Eventually, someone will make a mistake and show the number.

In a metadata-driven system, we do not rely on memory. We use tags.

1. Tagging: A manager puts a tag called Sensitive on the Social Security Number in the system.

2. Enforcement: The system sees the tag.

3. Action: The system automatically locks that column.

Now, when a normal user looks at the data, they see ***-**-****. The developer did not have to write code to hide it. The metadata did it automatically. The rule is in the system, not in the person’s head.

Impact Analysis and Change Management

The most terrifying button in data engineering is “Deploy”.

In a complex environment, changing the data type of a single column can break a dashboard used by the CFO, fail three downstream ETL jobs, and corrupt a machine learning model. In a manual architecture, these dependencies are guessed at.

Metadata provides a “blast radius” calculation. Before a change is applied, the system queries the metadata repository: “If I modify this column, who consumes it?”

The architecture can then block a deployment that violates a contract (e.g., removing a column used by a production API) or automatically notify the owners of the downstream dashboards that a change is incoming.

Closing word

We began this monograph with a single, innocuous number: “100”.

Sitting in a cell without context, it was meaningless. It could have been the age of a grandmother, the speed of a reckless driver, or the price of a stock. We end this journey not just with a definition of that number, but with a blueprint for an architecture that systematically guarantees meaning across the entire enterprise.

The transition to a metadata-driven data architecture is not merely a technical upgrade; it is an evolution of organizational maturity.

Measuring Success

Do not measure success by the number of columns defined or the number of users logged into a catalog. Those are vanity metrics. True success is measured in velocity and trust.

● Velocity: How long does it take a new analyst to encounter the number “100” and know exactly what it measures? In a traditional setup, this takes days of asking around. In a metadata-driven setup, it takes seconds.

● Trust: When the CFO asks if that “100” is accurate, can you provide the lineage map instantly, or do you need a week to audit the SQL scripts?

Future Directions

We are standing at the edge of the Generative AI era. Large Language Models and corporate AI agents are hungry for data. But these models are incredibly sensitive to the context crisis. If you feed an AI the number “100” without metadata, it guesses. It hallucinates.

In the coming years, metadata will be the bridge between your raw data and your AI. It will be the guardrail that ensures your automated agents understand that “100” is a percentage, not a currency, preventing costly errors.

The era of managing data by brute force is over. The era of managing data through metadata has arrived. Provide the context, and the wisdom will follow.

Never again let “100” stand alone.

About the Authors

W.H. Inmon is widely recognized as the father of the data warehouse and was named as one of the ten people who most influenced the first 40 years of the computer industry. Bill has sold over 1,500,000 books in his life.

Marcel Dybalski is a consultant and engineer specializing in AI and data platforms. For almost a decade, Marcel has helped organizations design and implement platforms with a strong emphasis on solid data architecture and management, always keeping the broader enterprise landscape and long-term business goals in mind.

ER/Studio: The Semantic Backbone for Enterprise AI

Generative AI does not eliminate the need for enterprise data architecture. It increases it.

As AI systems reason across applications, data products, documents, analytics, and governance assets, they depend on a consistent understanding of enterprise business meaning. Without that foundation, organizations face semantic drift, conflicting definitions, unreliable analytics, and untrustworthy AI outcomes.

ER/Studio helps organizations build and manage that foundation.

At the center of ER/Studio is the Enterprise Logical Data Model (ELDM), a shared representation of enterprise business concepts, relationships, identifiers, and rules. The ELDM provides a semantic backbone that connects governance, analytics, integration, data products, and AI grounding through a common understanding of business meaning.

Using ER/Studio, organizations can:

• Define and govern shared enterprise business concepts

• Create logical and physical models that reuse and operationalize those concepts

• Support Data Product and Data Mesh initiatives with consistent enterprise semantics

• Use Enterprise Logical Data Models as canonical models for integration and messaging

• Connect data architecture with governance, glossaries, catalogs, and metadata platforms

• Provide trusted semantic context for AI, RAG, and knowledge graph initiatives

As enterprises move from data-centric architectures to AI-centric architectures, the need for governed enterprise semantics becomes increasingly important.

ER/Studio helps organizations turn enterprise meaning into an operational asset.

To learn more at www.erstudio.com

MANAGING THE ENTERPRISE GEN AI LLM ENVIRONMENT:

PRACTICAL RULES OF THUMB

(ROT)

By W H Inmon

Jamie Knowles

Dave Rapien

Generative AI in the corporate environment rests on a foundation of corporate data. The effectiveness, reliability, and trustworthiness of generative AI directly depend on the quality, consistency, and meaning of the data it operates upon. Enterprise generative AI is subject to the most fundamental law of information processing: GIGO. If the underlying data coming from the corporation is inconsistent, ambiguous, fragmented, or poorly governed, the results produced by enterprise generative AI will reflect those same problems.

To successfully use generative AI in the enterprise, organizations must pay careful attention to the many aspects of the corporate data foundation that feeds enterprise generative AI. This foundation of data includes not only structured data, but also documents, policies, reports, metrics, classifications, business rules, metadata, and the semantic relationships that connect them together.

The challenge is that enterprise data exists across many corporate systems, domains, applications, and technologies. Over time, business meaning naturally diverges across such a large and diverse environment. Different systems define the same concepts differently. Business terminology changes. Duplicate entities emerge. Metrics drift. Governance becomes fragmented. As enterprises grow, this semantic entropy accumulates and increasingly affects integration, analytics, governance, and AI reliability.

Enterprise Generative AI amplifies these issues because it operates across large collections of structured and unstructured information simultaneously. AI systems infer meaning probabilistically from the information they are given. Without governed enterprise semantics, AI systems can produce inconsistent, unreliable, and difficult-to-govern answers.

Managing the corporate LLM environment therefore becomes a critical enterprise discipline. The management of tokens within the LLM environment is an important part of this discipline. Large Language Models operate on tokens derived from enterprise content. The volume, quality, structure, and governance of those tokens directly affect processing cost, retrieval quality, semantic consistency, and the reliability of AI-generated results. Minimizing unnecessary or poorly governed content reduces both computational overhead and the propagation of semantic inconsistency within the AI environment. The Large Language Model environment depends not only on prompts and vector databases, but also on the semantic quality, structure, governance, and consistency of the information used to ground AI reasoning.

The Enterprise Logical Data Model – ELDM - plays an important role in this environment. The ELDM provides stable business entities, relationships, identifiers, and definitions that form a semantic backbone for enterprise AI. Business glossaries, ontologies, governance policies, classifications, dimensional models, data products, canonical messages, and AI retrieval systems all become more effective when grounded in shared enterprise business meaning.

The rules of thumb contained in this document are intended as practical guidance for the practitioner responsible for implementing and managing generative AI in the corporation. They are based on decades of experience in data architecture, data warehousing, governance, integration, and enterprise modeling, adapted for the realities of modern AI systems.

These rules are intentionally practical, concise, and experience-based. They are not theoretical academic principles. They are operational observations intended to help organizations reduce semantic inconsistency, improve AI trustworthiness, lower integration and governance cost, and create a more stable foundation for enterprise AI.

Ultimately, trustworthy enterprise generative AI depends on trustworthy enterprise meaning.

The following collection of rules of thumb (ROT) are a compendium of best practices in the management of the LLM environment.

LLM Management Rules of Thumb

This booklet is a guideline for understanding and implementing LLM’s in the corporation.

1. Rule Of thumb: an ontology typically contains less than ten taxonomies.

Explanation: if an ontology contains more than ten taxonomies, the data becomes increasingly more difficult to work with and understand.

2. Rule Of thumb: the taxonomies in an ontology all relate in some form or the other to the subject/focus of the ontology.

Explanation: all taxonomies relate in some way or the other to the subject of the ontology. However, the relationship of the taxonomies to the subject/focus of the ontology can be quite different from each other.

3. Rule Of thumb: There may or may not be a relationship between the elements of the taxonomies in an ontology to each other.

Explanation: some taxonomies have a natural intersection with other taxonomies inside an ontology. Other taxonomies have a limited or no relationship to the elements found in other taxonomies found in an ontology.

4. Rule Of thumb: All of the elements in a taxonomy have the same or very similar relationship to the subject/focus of the taxonomy.

Explanation: If the subject/focus of the taxonomy is type of car, bread, soccer, and world peace do not belong in the taxonomy.

5. Rule Of thumb: if the subject/focus of the ontology domain is restricted, than the elements of the taxonomies that are in the ontology have the same domain restrictions.

Explanation: if the ontology subject/focus was North American landscape then a taxonomy in the ontology would not include the Nile, Amazon or the Mekong Delta. These rivers are not in North America.

6. Rule Of thumb: minimizing the number of taxonomies is a good practice.

Explanation: There are an infinite number of ways to make a classification. The LLM analyst tries to keep the number of taxonomies that are developed to the smallest number possible. This is to simplify and streamline the LLM environment.

7. Rule of thumb. Generic taxonomies can save huge amounts of time.

Explanation: Some taxonomies can be generic, such as accounting and finance taxonomies. These taxonomies can be used generically across lots of industries, saving unneeded time for development and maintenance.

8. Rule of thumb. Ontologies and taxonomies require ongoing maintenance.

Explanation: Most developers forget that ontologies and taxonomies require periodic maintenance. The good news is that after the ontology or taxonomy is created, existing maintenance is incremental or marginal.

9. Rule Of thumb: Do not attempt to build your LLM on a one time, grab everything basis.

Explanation: for many reasons building the tokens inside the LLM is an ongoing proposition, not a one time do it and finish effort. There are many reasons for this –

Business changes

New material keeps appearing

There is a lot of material to be entered in any case

Source material comes from a wide variety of sources

10. Rule Of thumb: Do not enter extraneous text – non business related text - into your RAG to be converted to tokens,

Explanation: when extraneous text – non business related text - is entered into the LLM, the result is that the amount of resources needed to execute queries grows to be quite large. In addition, extraneous text increases the chances of hallucinations. In order to make analytical processing efficient, the only text that should be used as a source for an LLM is business based text.

11. Rule of thumb. A log should be kept for exactly what source data was used to load the LLM.

Explanation: At a later point in time it may be necessary to remove text from an LLM. If a log is not kept of the text that was loaded into an LLM, it is not going to be possible to find the text that needs to be removed.

12. Rule Of thumb: When text is presented to the LLM for entry, the source of the text should always be attached to the text itself.

Explanation: some generative AI can keep track of the source of text. If there is ever any question as to the veracity of the text, it must be captured and presented to generative AI at the moment that text is prepared to enter the LLM.

13. Rule Of thumb: if at all possible the context of the text that is being entered into the LLM needs to be captured.

Explanation: Having the context of the text being entered into the LLM is a first and important step in the removal of hallucinations from the results being presented to the end user.

14. Rule Of thumb: Buffer management of queries can reduce the cost of the query. But buffer management has to be invoked on every query and does not stop the processing of extraneous data.

Explanation: Query buffer management does reduce the cost of query processing in LLM query processing. And query buffer management should be done. But query buffer management does nothing to prevent extraneous text from entering the LLM and only applies when iterative query processing is done.

15. Rule Of thumb: Removing stop words before they arrive in the RAG has only a minimal effect on the reduction of resources consumed during LLM query processing.

Explanation: Stop word editing should be done before raw text is sent to the RAG. But removing stop words has only a very marginal effect on the reduction of the cost of LLM query processing.

16. Rule of thumb. Reducing the text that goes into the LLM by removing extraneous text before entry has the potential of reducing hallucinations by up to 80%.

Explanation: Extraneous text entered into an LLM increases the odds of hallucinations by an order of magnitude.

17. Rule Of thumb: Preparing text for entry into an LLM can be done either on the cloud or on premise.

Explanation: text that contains private data should be processed on premise. But text that is not private is easier and less expensive to be processed on the cloud.

18. Rule Of thumb: It is a good idea to keep a log of the text used to feed the LLM. The log should identify the document, the date and who conducted the loading of text into the LLM.

Explanation: It may be necessary to reprocess the loading of the text. It is useful to know the specifics of the loading of text that has occurred.

19. Rule Of thumb: editing and refining text at the moment of loading into the RAG and removing extraneous text greatly clarifies and focusses the results that are generated.

Data Modeling Rules of Thumb

A collection of practical enterprise data modeling principles focused on logical modeling, semantic consistency, and business understanding for enterprise AI.

1. Rule Of thumb: The semantic backbone is the foundation of trustworthy enterprise AI.

Explanation: Without explicit business context, AI systems infer meaning probabilistically, often producing inconsistent or incorrect answers. An ELDM is an ideal implementation of this.

2. Rule Of thumb: Every data asset should follow a design process of logical model to physical model to code.

Explanation: This process ensures that business meaning, architectural standards, and governance are established before implementation begins. It also creates approval gates that confirm requirements are understood and met before progressing to the next stage.

3. Rule Of thumb: A logical data model should describe the business, not the application.

Explanation: Applications change frequently. Core business concepts change slowly.

4. Rule Of thumb: If the business cannot read the model, the model is too technical.

Explanation: Logical models exist to validate understanding before implementation begins.

5. Rule Of thumb: Every entity should represent a business concept with a clear definition.

Explanation: Tables are technical structures. Entities represent real business meaning.

6. Rule Of thumb: A model with duplicate entities usually reflects duplicate thinking.

Explanation: Multiple versions of the same concept create confusion, inconsistency, and reconciliation problems later.

7. Rule Of thumb: Reusable architectural patterns derived from the Enterprise Logical Data Model create consistency across the enterprise.

Explanation: Using common business structures, naming standards, and integration patterns improves interoperability, governance, maintainability, and shared understanding.

8. Rule Of thumb: The Enterprise Logical Data Model should describe business concepts before analytic structures.

Explanation: Star schemas and data vaults optimize implementation and delivery. The enterprise model defines the underlying business meaning they are built from.

9. Rule Of thumb: Business users should not need a decoding guide to read a model.

Explanation: Clear attribute names improve communication, governance, integration, and long-term maintainability.

10. Rule Of thumb: Attribute names should describe business meaning, not system implementation.

Explanation: Names like “Customer Status Code” are useful. Names like FLG1 or TXT_FIELD_7 are not.

11. Rule Of thumb: The enterprise model should survive technology migrations.

Explanation: A good logical model remains stable even when databases and platforms change.

12. Rule Of thumb: Relationships are often more important than attributes.

Explanation: Businesses operate through interactions between concepts, not isolated pieces of data.

13. Rule Of thumb: If an entity has no business key, the business may not truly understand it.

Explanation: Natural business identifiers reveal real-world meaning and integration points.

14.Rule of thumb: A well-understood business entity usually has more than one business identifier.

Explanation: Customers, products, and accounts are often identified differently across operational systems, requiring alternate business keys for integration.

15. Rule Of thumb: Connecting a business glossary to the Enterprise Logical Data Model creates a governed semantic backbone.

Explanation: The glossary contributes definitions, synonyms, policies, and rules, while the model contributes structure, relationships, and integration context.

16. Rule Of thumb: Normalization clarifies meaning before denormalization optimizes performance.

Explanation: Understanding should come before optimization.

17. Rule Of thumb: A data warehouse should integrate business concepts, not merely copy source systems.

Explanation: The purpose of warehousing is harmonization, not replication.

18. Rule Of thumb: A logical model should resolve business ambiguity before physical design begins.

Explanation: Physical databases cannot fix unclear business definitions.

19. Rule Of thumb: The same business term should not mean different things in different models.

Explanation: Semantic consistency is essential for analytics, governance, and AI.

20. Rule Of thumb: Good modelers ask business questions, not database questions.

Explanation: The goal is understanding the business before choosing technology structures.

21. Rule Of thumb: A many-to-many relationship usually hides an important business process.

Explanation: Intersection entities often reveal transactions, events, or activities the business cares about.

22. Rule Of thumb: History should be modeled deliberately, not accidentally.

Explanation: Temporal requirements become expensive when ignored early.

23. Rule Of thumb: Every model should have a clearly defined scope.

Explanation: Unbounded models quickly become unmanageable and politically contentious.

24. Rule Of thumb: Master data should appear once in the enterprise model.

Explanation: Core business entities should be standardized before downstream systems multiply variations.

25. Rule Of thumb: If two entities differ only by department name, they probably should not be separate entities.

Explanation: Organizations often duplicate structures instead of generalizing concepts properly.

26. Rule Of thumb: Naming standards matter more in enterprise models than in physical databases.

Explanation: Enterprise models are communication tools used across teams and over long periods of time.

27. Rule Of thumb: A logical model should be stable enough to outlive projects.

Explanation: Projects are temporary. Enterprise meaning is long-term.

28. Rule Of thumb: Data models are negotiation tools as much as design tools.

Explanation: Modeling exposes disagreements in business understanding early, when they are cheaper to resolve.

29. Rule Of thumb: The first version of the model is always wrong.

Explanation: Modeling is an iterative process of discovery and clarification.

30. Rule Of thumb: Business requirements should be validated in a logical model, not a physical schema.

Explanation: Physical models contain technical implementation details that distract from the business meaning stakeholders are trying to confirm.

31. Rule Of thumb: A model that mirrors the source system too closely probably is not integrated.

Explanation: Enterprise integration requires abstraction beyond operational structures.

32. Rule Of thumb: Optional relationships deserve scrutiny.

Explanation: Excessive optionality often hides weak business rules or unclear processes.

33. Rule Of thumb: Good data models reduce meetings.

Explanation: Clear shared definitions eliminate repeated debates about meaning and structure.

34. Rule Of thumb: Ontologies work best when grounded in stable enterprise business structures.

Explanation: Enterprise logical models provide the enduring business entities and relationships that ontologies, glossaries, and AI systems build upon.

35. Rule Of thumb: AI should not be grounded on a business glossary alone.

Explanation: Glossaries define terms, but enterprise models provide the structural relationships and context needed for reliable reasoning.

36. Rule Of thumb: Data governance is strongest when aligned to the Enterprise Logical Data Model.

Explanation: Policies, classifications, stewardship, and business rules become more consistent when attached to shared enterprise business structures.

37. Rule Of thumb: Enterprise models provide the structural context that governance tools often lack.

Explanation: Logical models expose relationships, hierarchies, and integration paths needed for trustworthy analytics and AI.

By Mario Meir-Huber, with economic analysis by Karl Ivo Sokolov.

Foreword

By Karl Ivo Sokolov

The frontier AI labs made a deliberate bet years ago to solve coding before any other use case, on the reasoning that solving coding unlocks the rest through self-feeding loops. In the second half of 2025, and unmistakably since the early 2026 model releases, that bet has paid off. Across our recent migration work, including SQL Server to Databricks, Oracle Forms to Angular and Postgres, and SAS to Snowflake, AI has begun to handle complex transformation lineages spanning thousands of tables in ways that simply were not possible eighteen months ago. The economics of building software are no longer what they were when Mario started his story.

The consequence in capital-allocation terms is bifurcated. Horizontal generic SaaS, the seat-priced workflow tools that won the last decade, is commoditizing as the cost of building a credible alternative collapses toward zero. At the same time, vertical bespoke software is in the early stages of an explosion, because problems that were never worth a data scientist’s salary plus a development team plus a year of build can now be solved in weeks by one capable builder running an agentic setup. The capex floor for custom software has collapsed, and the long tail of business problems that were too uneconomical to solve is suddenly addressable.

For the CFO, CIO, or board reader who arrived expecting another piece on AI productivity, the cleanest reframe is to read what follows through the eyes of a founder building a new software business rather than a steward maintaining an existing one. The enterprise version of Mario’s story is about what small enabled agentic-first teams inside the enterprise can now produce: vertically integrated bespoke software that previously required a vendor relationship, a multi-year program, and a P&L line a CFO would not have wanted to sign.

The commentary on this transition tends to stop at the capex floor, but the architectural floor did not collapse with it. Anyone can vibe-code a working prototype, and almost no one can vibe-code maintainable software. Systems-level thinking, the discipline of choosing the right structural and product boundaries before the AI starts generating, has become the disproportionate survival trait of the moment. The builders who win are the ones who direct AI at the system level rather than the typing level, and the gap between average developers using AI and architects using AI is widening rather than narrowing.

The piece that follows is one builder’s record of learning that lesson the hard way and turning it into a working method. I will return at three points along the way to put numbers on what Mario describes and extend the economics. Read his story as the operating manual for the bifurcation.

The Confession

I come from a family of builders. My grandfather built houses. My other grandfather was a smith. My dad is a carpenter. The material changed with me. I ended up in software engineering, but the instinct stayed the same: make something real with your hands.

After studying software engineering and a stint at Microsoft, I moved into data. I led large data teams, consulted on architecture, helped enterprises figure out their data strategies. For years, I was the guy in the room with the whiteboard, not the IDE.

Then my daughter needed a game.

I built her a small iOS app, a casual kids’ game. Over four years it grew into a surprisingly large codebase. But that hobby project did something unexpected: it put me back in front of code. And when AI coding tools started showing up, that little frog game I developed form y daughter became my first laboratory.

First Contact: The Codex Era

It was the very early days of ChatGPT. I had a Pro subscription and access to Codex, so I started there. My first experiment was adding shader effects to the game: nicer water, visual polish, things I probably would not have attempted on my own.

It worked, more or less. The code Codex produced got me moving. I would describe what I wanted, get a block of code back, paste it in, and then start fixing, because fixing was always part of the deal. Variable names did not match my project, logic was close but not quite right, and patterns that looked clean in isolation clashed with the existing codebase. Every generated snippet needed a manual pass.

One time, Codex produced a shader effect that looked beautiful in the preview. I shipped it. On device, it triggered a critical GPU issue on the iPhone and crashed the game. Shader effects are written in OpenGL Shading Language, a C-like syntax. There is no stepping through it with a debugger, no breakpoints, no stack traces. When a shader fails, the screen goes black and the app dies. I spent hours isolating the problem by commenting out lines one at a time. That was my first real lesson: AI-generated code can look correct and still be silently dangerous.

But here is the thing: even with all that correction, I was faster. Not dramatically. Maybe 25% faster than writing everything from scratch. More importantly, I started trying things I would not have touched before: a particle system, a new animation layer, small things individually that added up.

The outputs stayed modest: a function here, an effect there, nothing structurally ambitious. Codex was a useful assistant for isolated tasks, but it had no memory of my project, no sense of architecture, no understanding of how one piece connected to another.

Still, something had shifted. I was building again. The AI was not good enough to trust with anything big, but it was good enough to move.

The Valley of Despair

The tools kept getting better. I moved to Cursor, which gave me more context-aware suggestions and inline editing. I started building larger software, not game features anymore but full web applications. And with every improvement in the models, the temptation grew.

At some point, it felt so powerful that I started handing over more and more trust, too much too fast. I stopped reading the generated code carefully. I just accepted, ran, fixed the error, accepted again. It became a factory: features were landing, screens were appearing, it felt productive.

Then I looked at what I had actually built.

The code was a mess: no layered architecture, no class hierarchy, no reusable patterns. Business logic was tangled into UI components. Single class files had grown to 2,000 lines and beyond. Everything was coupled to everything. It worked, technically, but nobody had designed it; it had just piled up.

The worst part: once you are deep into a codebase like that, even AI cannot help you get out. The models choke on the complexity they created, context windows fill up with spaghetti, suggestions start conflicting with each other. You ask the AI to refactor and it produces more of the same.

One moment captured it perfectly. I asked the AI to fix a problem in a file it had written. Its response started with: “This is a very large file.” Yes. Yes it is. You wrote it.

That was the low point. I had outsourced not just the typing but the thinking. And the result was code that nobody, not me and not the AI, could maintain. I had done what every software team fears most: I had produced legacy, code that is too broken to extend and too expensive to replace. I was stuck.

A sidebar by Karl Ivo Sokolov: Technical debt after AI

Carrying broken legacy is measurable; benchmarks suggest rewriting a tangled codebase costs 2-4x the original build. AI without architectural discipline accelerates the rate at which liabilities accumulate, because generation outruns review. Mario’s 2,000-line single-class file is the visible end of an invisible balance sheet.

What AI changes is the math on rewrite-versus-encapsulate. For most non-strategic legacy, ring-fencing behind an agent-callable interface is now the rational default. The term comes from finance, where ring-fencing structurally separates a vital asset to constrain its blast radius and contain its cost line. Applied to legacy software: leave the working backend in place, define a clean API surface, let agents read and write through it. The legacy runs on forecastable cost, the surrounding system gets agent-native economics, and the rewrite question can be deferred until the math justifies it.

Where the legacy embeds rare domain expertise a rewrite team can capture cleanly, rewrite remains the right answer; the scarce good there is people, not code. For most enterprise legacy, the people who knew the domain are gone, the documentation is partial, and the working system is the documentation. Ring-fence first; rewrite when you have to.

When you do rewrite: in our recent migrations (SQL Server to Databricks, Oracle Forms to Angular and Postgres, SAS to Snowflake) total cost reductions run 40-50%, because pure coding is only half the work. SMEs, business analysts, QA, and program management still have to be paid. That figure is today’s reality, late April 2026, and will only improve.

The Shift: Claude Code and the Agent Mindset

I had to start from scratch on the messy codebase, no salvaging it. I moved back to VS Code, dropped Cursor, and started working with Claude.

But this time I did something different. Instead of just asking for code, I told Claude how to build. I specified the architectural patterns. I defined which frameworks to use, how to decouple layers, where boundaries between modules should be. I stopped treating AI as a code generator and started treating it as a developer who needs clear engineering standards.

The real breakthrough came with agents. I created my first specialized agent, an architect. Before writing any feature, I would run the idea past the architect agent. It would evaluate the approach against my defined patterns, flag violations, suggest alternatives. It was not generating code yet but thinking about structure.

Next came a frontend designer agent, then a testing agent, then others, each with a single responsibility and a clear scope. The game changer was my CI/CD agent. It runs in five steps, each requiring my explicit permission before moving on. It performs schema diffs between databases, checks for breaking changes between production and development environments, audits secrets, and validates deployment readiness. It does in minutes what used to take a team hours of manual checklist work.

At some point I realized I had stopped coding. I was managing.

A sidebar by Karl Ivo Sokolov: The unit economics of an agentic-first team

A loaded twenty-person EU engineering team, mixed seniority and roles, costs the business roughly €2.0 to €2.5 million per year all-in. Mario’s setup, run by one capable builder, restructures that cost line rather than eliminating it. The Anthropic bill at current Max-subscription pricing is a low four-figure monthly outlay per power-builder. We expect that figure to rise meaningfully over the next 12 to 24 months as flat-rate plans reprice toward consumption, possibly 2-3x current rates. (Expert opinion; take it as such.) Even at 3x current rates, the AI cost remains two orders of magnitude below the comparable team cost.

What does not disappear is the surrounding human work. Subject-matter experts, business analysts, QA leads, and program managers are still on the ledger. Their work shifts from meetings into writing markdown specifications and reviewing agent outputs; their time is still paid for. The honest median, planned conservatively across multiple engagements rather than picked from one heroic outcome, is around 60% total cost reduction against the comparable team, with 5-8x the feature velocity. The velocity number includes small polish features that previously got cut from the backlog because they did not justify a developer’s attention. Those features now ship, and the cumulative effect on product quality is genuinely novel to the agentic operating model.

This is a feature of the approach, not a limitation. The codified domain expertise the SMEs hold becomes more valuable as the surrounding routine work compresses, which is the same dynamic explaining why professional certifications in finance and accounting (the CMA, the CFA, and their analogues) are seeing rising rather than declining demand. AI does not flatten the org chart so much as steepen it: the people whose judgment the system depends on get more leverage, and the rest of the work consolidates onto fewer, more capable operators.

60% cost reduction at 5-8x feature velocity is the planning number a CFO can rely on, not a hero-case to hope for.

One lesson I learned the hard way: commit early, commit often. When you produce software at the pace of a 20-person team, mistakes scale just as fast. One wrong prompt can cascade through your entire codebase before you notice. I learned this when a single bad instruction cost me an entire night. I had not committed in Git. The codebase was corrupted and I had no clean state to revert to. Hours of work, gone. One prompt, one night lost.

After that, I treated Git commits like breathing. Every meaningful change, no matter how small, gets committed before the next prompt.

What I Built

With the agent approach in place, real software started to emerge.

The first serious application was a FastAPI-based financial management platform for my own consulting business. Invoices, expenses, accounting, cashflow forecasts. AI handles categorization and pattern detection, but the core is a proper SaaS application with layered architecture, clean separation, and maintainable code. It runs my business today.

Next came Vane Loop. For years this was a consulting framework that lived on slides and in workshops. With the agent workflow, I turned it into a working web platform. Several large European organisations already use it. That jump from PowerPoint to production would not have happened if I were still copy-pasting code from ChatGPT.

My daughter got hungry for more games. She had seen what the frog game became and wanted new ones. But time for hobby coding is limited when you run a consulting practice. So I asked myself: what if I push this further? Not just instructing individual agents, but building a fully autonomous software team. I had managed real engineering teams for years. I knew what the roles looked like, how handoffs worked, where things break down. What if I modelled that entire structure in AI agents that talk to each other?

That is how AI Team was born. The idea is simple but ambitious: you describe a feature, and a coordinated team of specialized agents takes it from specification to tested, shipped code.

The workflow mirrors how a real team operates. A Requirements Engineer defines the feature spec. A Principal Designer then discusses the design with the Requirements Engineer, going back and forth until the look and interaction model are locked in. A Proxy Product Owner breaks the spec into tasks. An Architect adds structural guidance. An Implementation Engineer writes the code. A Test Manager derives test cases from the original spec, then hands off to unit and UI automation testers. The agents talk to each other, challenge each other, and hand off in sequence. Nothing gets skipped.

The first team I built targets Swift for iOS. You install it with npx aiteam, it writes a routing manifest into your project, and Claude Code picks up the entire team from there.

My goal: give my daughter a new game every week. She specifies what she wants. The agents implement it.

The first result is already on the App Store, including Game Center integration so that I can play against her. It was built fully autonomously by the AI Team workflow. Later I added Game Center support so my daughter and I can play against each other. That feature was also designed by the agents. I had to step in a few times, but the turnaround was remarkably fast. Tests are covered too. Every user story is verified by the UI automation agent, which tests the app the way a human would by driving Apple’s XCUITest framework.

I open-sourced AI Team because I believe great agent prompts are a community resource. I have a long list of ideas for new teams and roles, but honestly, this is bigger than what one person can build. I am hoping others will pick it up and contribute.

The Numbers Behind This Shift

By Karl Ivo Sokolov

Mario’s piece reads as a personal journey, but the unit economics underneath it are worth pricing explicitly, because they are the reason this transition is durable rather than a productivity-tool trend.

Consider the three artefacts. The FastAPI consulting platform that runs Mario’s business, with invoice management, expense tracking, accounting, and cashflow forecasting, would be a €250,000 build at EU mid-market agency rates, plus €12,000 to €30,000 per year in maintenance. Mario built and runs it himself, with his own time and the AI bill described above as the only direct costs.

Vane Loop is the more interesting case. Priced narrowly at agency rates for the deterministic portions of the platform, it would land somewhere between €500,000 and €800,000. But that pricing is misleading, because Vane Loop without foundation models is not a smaller version of the same product. It is a different product that no one would build, because the value proposition only exists when AI exists. Auto-generated SWOT analyses, augmented decision support across 100 use cases, AI-driven recommendations from a 200-question stance assessment with 5-10 document attachments per question; these are not features that can be specified deterministically and built by a 2022-era engineering team. The economics of Vane Loop run on a different logic from agency-fee comparisons. Without foundation models, the product simply does not exist; with them, a single architect can build it inside a market window that closes for slower competitors.

The iOS game and the AI Team open-source framework round out the portfolio at perhaps €200,000 to €350,000 of equivalent commissioned work, with AI Team being the kind of orchestration layer that would normally appear as a contracted R&D engagement rather than an agency build.

Total counterfactual at agency rates, treating Vane Loop conservatively at €500,000 for what is buildable that way: roughly €950,000 to €1.1 million for what one builder shipped. Actual all-in cost, including Mario’s time at a generous self-imputed rate plus the AI bill: a small fraction of that. The order-of-magnitude gap matters less than the question of who this transition unlocks.

The bifurcation in the foreword cuts in this exact spot. Horizontal generic SaaS is repricing because the build cost has collapsed; the buyer’s question is no longer “is this worth €60 per seat per month?” but “what premium am I paying over the bespoke alternative I could now commission?” The premium has to be defended on security, compliance, integration, and domain coverage rather than on the difficulty of the build. Vertical bespoke is the side that grows, because the long tail of business problems too small to commission a custom build for is now economically buildable.

The neighborhood case is the cleanest illustration. Imagine a single-location bike, ski, and rental shop in an Austrian village, the kind of business that turns over €1-2 million a year and has never written a custom line of code. The owner has lived for 20 years with off-the-shelf retail software that does not understand snow seasonality, does not integrate with the village community calendar, does not price end-of-season clearance dynamically, and does not know which workshop parts are in stock when scheduling a service appointment. None of those problems were worth a custom build at historical rates; each would have cost €30,000 to €60,000, and the shop would never have approved any of them.

In 2026, all four become a few weeks of agentic build, possibly by a freelancer who knows the shop, possibly by the owner’s son using Claude Code. A custom local CRM tied to the community calendar (Joe Reis observed on LinkedIn that CRM is the vibe-coding hello-world, which lands precisely because it is true) plus a workshop scheduler that knows the parts inventory plus a snowfall-aware rental forecaster plus dynamic shoulder-season clearance pricing. The total counterfactual at agency rates is €150,000 to €240,000 of bespoke software no one was ever going to commission. The actual cost is a freelancer’s 6-week engagement.

Multiply that village shop by every small and mid-sized business in Europe whose operations have been ignored by SaaS because the unit economics never worked, and you arrive at the picture of the application layer in 2026: commoditizing at the top, exploding at the bottom, and far from finished.

Recommendations for Builders

After months of building this way, here is what I have learned.

Stay in control. Never fully trust the output. Review everything. The moment you stop reading the code is the moment it starts rotting.

Decouple testing from implementation. AI has a tendency to write unit tests that simply confirm what the code already does. The tests go green, but they prove nothing. In AI Team, the Test Manager derives test cases from the user stories, not from the code. The testing agents never see the implementation. If the functionality is broken, a red light is a reward. It means the tests are actually doing their job.

Describe problems, don’t prescribe fixes. Never tell an agent “there is a bug, fix it.” Describe the wrong behaviour. Let the agent analyze the problem and build its own context first. In AI Team, problems get routed from the testing agents to the Implementation Engineer, but the fix only starts after the Proxy Product Owner has written a proper bug report. The agent analyzes first, then fixes. This alone eliminated most of the “fix one thing, break two others” loops I used to get stuck in.

Fewer agents, sharper focus. You will not get better code by having 80 agents. Eight to ten well-crafted, focused agents are enough. If your project grows beyond that, split into separate teams with their own repositories. A frontend team and a backend team, just like in a real company.

Commit early, commit often. I said this already in my story, but it is the one mistake I see everyone make. A good rule: never send a second prompt without committing first. If that feels excessive, you have not yet lost a night to a bad prompt.

Run periodic codebase health checks. Ask an agent to audit the project structure, flag files that have grown too large, identify dead code, and check that architectural boundaries are still intact. Codebases drift. AI-generated codebases drift faster.

Keep a Book of Standards. Define your architectural patterns, naming conventions, folder structures, and framework choices in a single document that every agent reads. Without it, each agent invents its own conventions and you end up with a codebase that looks like it was written by ten people who never talked to each other. Because it was.

Invest tokens in context, not in carrying it. In AI Team, every agent generates markdown documentation before handing off. The developers receive a full specification of the app, but they restart with a fresh context window for every user story they implement. This means spending tokens on context creation rather than dragging an ever-growing conversation forward. The context each agent needs to carry stays manageable, and that increases the likelihood they get it right on the first shot. Yes, the Anthropic bill is real. But compared to hiring a development team, it is a rounding error.

Invest time in prompts, not in shouting. A well-crafted prompt solves more problems than ten frustrated follow-ups. Being mean to agents never helps. Being precise always does. Also, when the machines eventually take over, you will want to be on the polite list.

Review at the spec level, not just the code level. The approach of writing specifications before implementation gives you a checkpoint. You can read what the agents understood, what they plan to do, and whether their overall idea is correct or broken. You intervene at the meta level, before a single line of code is written. This is faster and cheaper than debugging after the fact.

Treat agent prompts like production code. Version them in Git, review changes, iterate. A prompt regression is just as dangerous as a code regression. You would not deploy untested code. Do not deploy untested prompts.

Build a smoke test you run after every session. A single script that boots the app, hits the critical paths, and confirms nothing is on fire. You need this because AI can break things in places you did not touch and did not think to check.

Use structured handoff formats between agents. Markdown specs, YAML task lists, not free-form conversation. The more structured the handoff, the less room for misinterpretation. Agents are like junior developers: they do exactly what you tell them, which is terrifying when you are vague.

Hallucinations are real. Learn to live with them. You will catch the AI inventing APIs that do not exist or referencing frameworks it confused with something else. This is frustrating, and you need to verify everything. But here is the other side: without hallucinations, you get no creativity. A purely deterministic system only recombines what it has seen. The same property that makes AI unreliable also makes it surprisingly inventive. Treat hallucinations as a trait of the medium, not a defect. Check the output, but do not wish away the thing that makes it useful.

When the AI does something you did not ask for, check your prompt first. It is tempting to blame the tool. But most of the time, the prompt invited it. If you write “I see a bug,” the AI will try to fix it immediately. That is what you asked for, even if you meant “analyze this first.” Be explicit about what you want: investigate the root cause, write up the findings, then propose a fix. It feels more expensive upfront but it eliminates the loops where the AI patches symptoms while the real problem keeps coming back.

Start small, prove the workflow, then scale. Do not begin with your most complex project. Pick something contained. A utility app. A single API. Get comfortable with the agent handoffs, learn where you need to intervene, and then bring the approach to bigger things.

AI powered coding is real

My grandfather laid bricks. My dad cuts wood. I write prompts. Somewhere along the way that became building too. And the best part is that my daughter now sits next to me, tells me what game she wants, and a week later it is on her iPad. That alone was worth every crashed shader, every lost night, and every 2,000-line file I had to throw away.

The bifurcation will sort builders from custodians, and Mario has already made his choice.

Karl Ivo Sokolov

Links

• AI Team on GitHub

• Flower Power Tic Tac Toe on the App Store

• Vane Loop

Karl Ivo Sokolov is Managing Director Data & AI at Specific-Group Austria, where he leads international teams across eight European countries. His work focuses on building robust data products and guiding enterprises through the modernization of complex data environments spanning both legacy and modern platforms. Beyond his role at Specific-Group, Karl Ivo also serves on the Global Board of Directors of the U.S. Institute of Management Accountants (IMA), contributing to the advancement of data-driven decision-making in finance and management accounting worldwide.

Mario Meir-Huber helps organizations turn scattered data initiatives into governed, business-driven Data Products. A former Head of Data and ex-Microsoftie, he has built Data Products across European companies. He created the GAP (Governance–Architecture–People) model and the DRIVE lifecycle framework and applies them in real engagements. Mario lectures at WU Vienna and TU Wien, teaches on LinkedIn Learning and speaks at events like WeAreDevelopers, Data Modelling Zone, London Tech Week, Data Science Conference and GITEX. He co-authored Handbook Data Science & AI and is writing The Data Products Series.

Data & AI Cognitive Architecture: A Federated Model

By Mustafa Qizilbash

Monograph Structure Overview

1. Structural Failure of Enterprise Data Architecture: Why modern data architectures lose coherence as they scale.

2. Foundations of Federated Enterprise Architecture: How federation separates ownership from control to sustain alignment.

3. DAC Architectural Topology: The structural model that enforces how data enters, moves, and is consumed.

4. The Integration Hub: Structural Control of Ingestion: A single, governed entry point that eliminates ingestion-driven divergence.

5. Enterprise and Business Zones: Federated Ownership Model: How domain autonomy is balanced with enterprise-wide structural control.

6. Virtualization Layer: Unified Serving Plane: A controlled access layer that standardizes consumption and prevents fragmentation.

7. Absorbing AI Within Federation: How AI operates within architectural boundaries without creating parallel systems.

8. Governance Blueprint and Enforcement: Governance embedded as structure, not policy or coordination.

9. Operational and Economic Implications: How structural control enforces accountability, efficiency, and cost discipline.

10. DAC Adoption and Architectural Doctrine: How to transition from fragmented systems to enforced architectural coherence.

Monograph Flow Categorization

• 1–3 → Structural breakdown of modern enterprise data architecture

• 4–6 → Foundational model for federated coherence

• 7–10 → Enforcement mechanisms within DAC architecture

• 11 → Appendix
  

1. Structural Failure of Enterprise Data Architecture

Enterprise data architecture is failing. Not because organizations lack investment, or because the problem is inherently too complex, but because the structure required to hold these ecosystems together is either weak or absent. As data landscapes stretch across domains, platforms, and workloads, coherence simply does not sustain itself. Below diagram show the potentiate challenges faced by different persona(s) in an organization.

Over the past decade, there has been no shortage of progress. Cloud-native platforms, lakehouse patterns, distributed processing, governance tooling - none of this is immature anymore. And yet the outcomes tell a different story. Environments expand, but alignment erodes. Governance becomes more visible, more documented, more discussed - but less effective in practice. Analytical capability grows, but trust lags behind, sometimes significantly. These are not isolated execution failures. They point to something more fundamental.

Most architectures are built to enable. They focus on access, flexibility, integration. What they don’t do - at least not in any enforceable way - is constrain behavior. How data enters the system varies. How it moves across domains is often negotiated rather than controlled. Consumption evolves based on local needs, not architectural discipline. Once those boundaries are left open, divergence doesn’t creep in - it’s introduced immediately, at the edges, and then carried forward. You end up with a landscape where each environment operates on its own terms. Ingestion patterns differ. Transformation logic reflects local interpretation. Consumption interfaces evolve independently. What looks like a distributed system is, in reality, a collection of loosely related systems.

Data stops behaving like an enterprise asset and starts behaving like a set of localized artifacts. Definitions drift. Lineage breaks down. Governance, at that point, becomes observational - it reports on divergence rather than preventing it.

None of the dominant architectural models have fully resolved this. Centralized approaches maintain control, but at the cost of adaptability. They struggle when distribution becomes a requirement rather than an exception. Decentralized models swing in the opposite direction - ownership is distributed, flexibility increases, but discipline fades and divergence accumulates quickly. Federated models attempt to balance the two, but in practice they depend on alignment and shared standards that are rarely enforced. Without hard boundaries, federation tends to erode over time into decentralization. The pattern underneath all of this is consistent. Architecture defines intent. It does not ensure adherence. Principles, patterns, and policies assume coordination across teams. At small scale, that might hold. At enterprise scale, it doesn’t.

Fragmentation starts early - at ingestion. Multiple pipelines are built to the same source systems, each slightly different. Extraction logic varies. Refresh cycles don’t align. Even interpretation of the same source data begins to diverge. That variation isn’t corrected - it propagates. From there, it compounds. Data is replicated across platforms to serve different workloads. Each environment reshapes it according to its own objectives. Over time, transformation logic drifts. Metadata becomes inconsistent, sometimes incomplete. Lineage exists in fragments, if at all. What you’re left with is not a shared foundation, but a series of parallel interpretations. As the system grows, so does the problem. Every additional pipeline, every replication point, every transformation layer introduces more divergence. There is no natural mechanism that pulls the system back toward alignment.

Capability increases. Structural integrity weakens.

At some point, the symptoms become hard to ignore. Core business entities - customers, products, transactions - start appearing in multiple forms that don’t reconcile cleanly. Alignment is no longer a design property; it becomes an operational burden. Teams coordinate, negotiate, and manually reconcile, often under time pressure. The absence of an authoritative dataset isn’t accidental - it reflects a lack of structural control upstream. Operationally, the impact is immediate. Ingestion inconsistencies lead to different data states across environments. Independent transformations produce outputs that cannot be directly compared. Metadata gaps obscure lineage, making it difficult to understand how data was derived. Reconciliation becomes routine, not exceptional.

Then AI enters the picture and accelerates everything. AI workloads demand rapid iteration, repeated training, and access to data across domains. When the governed paths can’t keep up, teams build around them. Parallel pipelines appear. Data is copied, reshaped, and used outside existing structures. Those parallel paths don’t stay temporary. They harden. Feature engineering happens in isolation. Training pipelines evolve independently. Over time, what was once a single architecture starts to look like multiple ecosystems, each with its own logic, its own data, its own assumptions.

This has direct consequences for decision-making. The same entity is defined differently depending on where you look. Analytical outputs vary because the underlying data and transformations differ. Consistency becomes situational. Trust becomes conditional. Machine learning systems amplify the problem further. Models inherit the assumptions embedded in their training data. Move them to another environment, and those assumptions don’t always hold. Results shift. Reproducibility breaks down. Without consistent lineage and metadata, it’s difficult to even understand why. Governance doesn’t disappear - but it weakens. Data spans multiple systems, each with its own policies, its own gaps. Visibility fragments. Enforcement becomes partial at best. What remains is coordination - manual, ongoing, and increasingly expensive as the system grows. Cost follows the same pattern. Duplicate pipelines, redundant storage, repeated transformations - they accumulate quietly. Multiple teams pay to maintain variations of the same data. Attribution becomes fuzzy because ownership is obscured by duplication.

These effects reinforce each other. Divergence leads to duplication. Duplication increases inconsistency. Inconsistency reduces trust. Reduced trust drives more localized data creation. The system doesn’t stabilize - it drifts further.

And recovery is not straightforward. Aligning datasets means coordinating across teams with entrenched dependencies and differing interpretations. Transformation logic has to be reconciled, often at the expense of existing workflows. The longer divergence persists, the more expensive it becomes to correct. At a certain point, incremental fixes stop working. Restoring coherence requires structural intervention - redefining pipelines, standardizing transformations, consolidating datasets. These are not small changes. They cut across both operational and analytical layers.

The underlying issue is not scale, or distribution, or even the introduction of AI. It’s the absence of control over how data enters, moves, and is consumed. Policies set expectations, but they don’t enforce behavior. In distributed environments, that distinction matters. If coherence is the goal, control cannot remain optional. It has to be built into the architecture itself. Data entry follows defined validation and standardization rules. Movement across domains is governed, not negotiated. Consumption happens through consistent, controlled interfaces. These are not guidelines - they define how the system operates. This is where Data & AI Cognitive (DAC) Architecture comes in. Not as another variation of existing models, but as a structural correction. It defines control points that are not negotiable. Domains retain autonomy, but only within those boundaries. They don’t decide how data enters, how it moves, or how it is exposed. That is defined once - and enforced

.For a long time, enterprise data architecture has prioritized enablement over control. That trade-off doesn’t hold at scale. Structural control isn’t an enhancement layered on top. It’s the condition required for the architecture to function at all. This didn’t happen overnight. It’s the result of how the industry has responded to scale. Centralized models enforced control but couldn’t keep up with demand. Decentralized approaches restored responsiveness but lost discipline. Federated models tried to bridge the gap, but relied too heavily on coordination. Without hard boundaries, they drifted.

What we see now is the outcome of that progression - capability without constraint. DAC doesn’t introduce a new direction as much as it closes a gap that has been there all along.

2. Foundations of Federated Enterprise Architecture

As enterprise data environments expand across domains, platforms, and workloads, the problem shifts. It is no longer about consolidation. The real issue is whether coordinated operation can be sustained across distributed systems without losing structural integrity. Multiple sources, multiple platforms, multiple domain environments - they all need to coexist. But coexistence without constraint quickly turns into divergence. Centralizing everything into a single platform doesn’t resolve this. It simply collapses under scale and diversity. Something else is required.

Federation addresses that gap. It distributes responsibility, but it does not distribute control indiscriminately. It defines where centralization must remain and where decentralization is acceptable. The distinction matters. Centralization governs structure. Decentralization governs execution. Federation only holds if that boundary is enforced. Without enforcement, it degrades. At the centre of this model is a separation that is often misunderstood - ownership is not the same as governance. Domains own their data, their transformations, their workloads. They execute. But they do not define how data enters the enterprise, how it is classified, or how it is exposed across domains. Those are architectural concerns. If domains take control of them, the structure doesn’t stretch - it dissolves.

That separation forces a more uncomfortable question: what is allowed to vary, and what is not. Domain implementations can and should adapt. Structural rules cannot. Governance here is not advisory, and it is not negotiable. It defines operating conditions. If those conditions can be bypassed through local decisions, they are not governance - they are suggestions. A federated architecture, in practice, is a centralized blueprint with decentralized execution layered on top. The blueprint is not abstract. It defines metadata standards, access control, classification, lifecycle, and the allowed movement of data. These are enforced constraints. Domains operate within them, not around them. What the blueprint does not do is dictate how domains build internally. That remains flexible. It defines interaction, not implementation. Domains can optimize for their own workloads, choose their modeling approaches, structure their transformations as needed - but they cannot alter how data enters the system, how it flows, or how it is exposed.

That boundary is where most implementations fail.

Structural enforcement is not achieved through documentation or alignment sessions. It is achieved through topology. The architecture itself determines how data can move and where control is applied. If movement is unrestricted, autonomy leaks into structural behavior, and fragmentation returns - quietly at first. This is where zones come in. Not as logical groupings or organizational constructs, but as enforcement points. Each zone has a defined role, explicit constraints, and limited operations. Data doesn’t move freely between them. It transitions through controlled paths. Zones introduce order into what would otherwise be an open system. They force data through defined stages. They preserve lineage by design, not by afterthought. They limit replication by constraining where and how data can be copied or transformed. The system becomes predictable - not in what domains do internally, but in how data behaves across the architecture.

Certain controls sit at the core of this structure, and they are not optional.

Ingestion is one of them. If domains are allowed to connect independently to source systems, variation is introduced immediately. Different extraction logic, different refresh cycles, different interpretations. Once that happens, downstream consistency is already compromised. Ingestion defines the initial state of data. If that state is not controlled, nothing downstream will correct it. Movement is another. Data does not move arbitrarily across zones. It follows defined pathways. Those pathways enforce validation, preserve lineage, and constrain transformation. Uncontrolled movement leads to duplication and breaks traceability. At that point, the system cannot explain itself. Exposure follows the same pattern. Domains cannot define their own mechanisms for sharing data externally. Exposure must be mediated through controlled interfaces. Without that, interoperability degrades. Cross-domain usage becomes unreliable - not because the data isn’t there, but because it means different things in different places.

Within these boundaries, domains still have room to operate. They define their models, run their transformations, execute their workloads. But that autonomy is contained. It does not extend into structural behavior. Domains can optimize - but they cannot redefine the system. This is the tension federation manages. Autonomy where variation is acceptable. Control where consistency is required. The architecture draws that line. If it doesn’t, teams will - independently, and inconsistently. Metadata holds the system together. Not as an add-on, but as structure. It defines how data is described, how it is traced, how it is understood across domains. Without consistent metadata, relationships between datasets degrade. Lineage becomes partial. Interpretation becomes ambiguous. The system fragments, even if the topology remains intact.

Cost behaves the same way if left unmanaged. Distributed environments multiply consumption points - pipelines, storage, compute. Without enforced visibility and attribution, duplication grows unchecked. Teams replicate data because they cannot see or trust what already exists. Costs rise, but more importantly, so does inconsistency. Cost control, in this context, is structural. It has to be embedded. Semantic consistency is a different kind of constraint. Domains do not need to implement identical models, but they cannot redefine shared business concepts independently. Where data crosses domains, meaning must remain aligned. Without that, structural control is insufficient. The system may be well-governed technically, but it will still fail operationally.

The strength of federation is not in distributing ownership. That part is easy. It lies in enforcing the conditions that prevent that distribution from turning into fragmentation. Control over entry, movement, and exposure is what stops the system from drifting. Governance is not applied from the outside. It is built into how the system works. Adoption does not happen arbitrarily. There is an order, whether acknowledged or not. Structural control has to come first. Ingestion authority is established before anything else. Data enters through governed pathways, or not at all. Then zonal boundaries are defined - where transformation occurs, where control is applied. Movement between zones is constrained next. Only after these are in place does domain autonomy expand.

Reverse that sequence, and the outcome is predictable. Autonomy fills the gaps. Decentralization takes over. What looks like federation on paper behaves very differently in practice. Skipping steps doesn’t produce a partial federation. It produces a decentralized system with additional overhead.

Federation, when it works, allows distributed environments to scale without losing coherence. It removes reliance on coordination and replaces it with structural enforcement. Domains remain independent in execution, but they do not diverge structurally. The architecture described here builds directly on that premise. Control over how data enters, moves, and is consumed is fixed. Non-negotiable. Domain execution remains flexible, but only within those constraints. That balance is what allows modern workloads - including AI - to operate without breaking consistency, lineage, or governance. Architectures that avoid enforcing these conditions don’t fail immediately. They drift. And once that drift sets in, recovery is expensive. Federation, implemented properly, is not one option among many. At this scale, it is the only model that holds.

3. DAC Architectural Topology

A federated architecture does not operate through principles or governance intent. It requires an explicit topology that determines how data enters the system, how it is organized across domains, and how it is consumed. Without defined and enforced topology, federation cannot sustain coherence. Pipelines proliferate, semantics diverge, and governance becomes reactive rather than controlling. DAC defines this topology through structural boundaries that are enforced across the entire architecture. These boundaries are not conceptual. They determine how data flows, where it is transformed, and how it is accessed. The topology establishes a controlled system in which every data operation occurs within defined constraints

.At the outer edge of the architecture are enterprise source systems. These include operational databases, enterprise applications, and external data providers. They evolve independently and expose heterogeneous integration mechanisms. Allowing analytical environments to connect directly to these systems introduces uncontrolled ingestion pathways. Each connection creates a new pipeline, often duplicating logic, bypassing governance, and introducing variation at the point of entry. Over time, this results in pipeline proliferation and fragmented control. DAC eliminates this condition by enforcing a single ingestion authority: the Integration Hub. All data entering the architecture must pass through this boundary. Direct connections from analytical environments to source systems are not permitted. Each source is integrated once through a governed process, ensuring that ingestion logic is defined, controlled, and consistent. The Integration Hub is not an integration convenience. It is a structural control point. All physical data movement into the architecture is governed through this boundary. Independent pipeline creation outside this control is not allowed. This eliminates variation at the point of entry and establishes a consistent foundation for all downstream operations.

Within the Integration Hub, data progresses through defined layers that enforce traceability and standardization. The landing layer preserves raw data in its original form, ensuring that source-level traceability is maintained. The staging layer applies structural normalization, validation rules, metadata registration, classification, and quality controls. Data that does not meet these conditions is rejected and does not proceed further into the architecture. Entry is conditional, not permissive.

Once ingested and validated, data is made available to analytical environments organized into two structurally distinct zones: the Enterprise Zone and Business Zones. These zones define how data is transformed and consumed within the architecture. They are not independent systems. They are governed environments operating within a shared topology.

The Enterprise Zone provides an integrated analytical environment aligned to shared models and reference data. It supports cross-domain analytics and enterprise-level decision-making. Data within this zone is structured to maintain consistency across domains, ensuring that shared business concepts are aligned and accessible. Business Zones are domain-aligned environments where domains control transformations, models, and performance optimization within architectural constraints. Domains operate independently within these zones, but they do not control structural behavior. Their autonomy is limited to execution within defined boundaries. The relationship between zones is controlled. Direct dependencies between zones are not permitted. Zones cannot extract or replicate datasets through independent pipelines. This restriction prevents uncontrolled data movement and ensures that transformations remain traceable. Zones operate as coordinated environments, not as independent systems exchanging data through unmanaged pathways.

Data consumption and physical data movement are separated within the topology. Cross-domain consumption occurs through the Virtualization Layer, which serves as the unified interface for all analytical access

.This layer operates in read-only mode and decouples consumers from underlying storage. Consumers do not access data directly from zones. They access it through controlled interfaces that enforce consistency in how data is interpreted. This separation is structural. Logical access is provided through virtualization, while physical data movement is governed independently. When data must be materialized across zones, it is routed back through controlled pathways rather than being transferred directly. This ensures that lineage is preserved and that duplication remains visible and governed. The Virtualization Layer functions as the exit point of the architecture. It enforces access control, data masking, and classification policies consistently across all consumers. By operating as a single serving interface, it prevents uncontrolled extraction and unmanaged replication. Consumers interact with data through defined interfaces, not through direct access to storage.

Control within the topology is enforced at three critical points: ingestion, cross-zone movement, and consumption. Data does not enter the architecture without passing through the Integration Hub. It does not move across zones without controlled pathways. It is not consumed outside the Virtualization Layer. These constraints define how the system operates. They are not optional. Metadata continuity spans all layers of the topology. From ingestion through transformation to consumption, lineage, ownership, and processing logic are recorded consistently. This provides a complete and traceable view of how data is derived and used across the enterprise. Without continuous metadata, structural control cannot be maintained.

Governance within this topology is enforced structurally

.Cost is embedded within the topology as a governed attribute. Enterprise-level environments operate as shared platforms, while domains are accountable for their own resource consumption within Business Zones. This separation ensures that resource usage is visible and attributable. Without it, duplicated pipelines, redundant storage, and unnecessary compute become untraceable, leading to uncontrolled cost expansion. The architecture separates storage, compute, and governance into distinct planes. Storage manages datasets, compute executes workloads, and governance operates across both through metadata and control mechanisms. This separation allows each plane to evolve independently without disrupting the structural integrity of the system. It ensures that control remains consistent even as technologies change.

Adoption of this topology requires enforced sequencing. Ingestion control must be established first, ensuring that all data enters the system through governed pathways. Zonal boundaries must then be defined to structure how data is transformed and accessed. The unified consumption layer is introduced after these controls are in place. If this sequence is not followed, domains establish independent pipelines and consumption patterns before structural boundaries exist, and the architecture reverts to decentralized behavior. This topology is the structural backbone of DAC. It defines how data flows through the system, where control is applied, and how consistency is maintained across distributed environments. It enables analytical scale without sacrificing traceability or governance. Federation without enforced topology does not function as architecture. It remains a conceptual model dependent on coordination and alignment. Topology makes federation operational by embedding control within the structure of the system.

Structure determines behavior. Without enforced topology, architecture does not govern.

4. The Integration Hub: Structural Control of Ingestion

In a federated architecture, structural control begins at ingestion. If the point of entry is not governed, no downstream mechanism can restore coherence. Variation introduced at ingestion propagates across every subsequent transformation, replication, and consumption path. Control applied later can only observe divergence, not prevent it. For this reason, DAC establishes ingestion as a non-negotiable control point and enforces a single ingestion authority: the Integration Hub

.The Integration Hub is the mandatory boundary through which all data enters the architecture. It is not an optional integration layer or a convenience mechanism. It defines the only permissible path from source systems into the governed environment. Direct connections from analytical platforms, domain environments, or AI systems to enterprise sources are not permitted. Each source is integrated once through a centrally visible and governed process, ensuring that ingestion logic is defined, consistent, and controlled at the architectural level. This constraint eliminates uncontrolled pipeline creation at the point of entry. Without a single ingestion authority, multiple teams establish independent connections to the same sources, introducing variation in extraction logic, refresh cadence, filtering conditions, and interpretation. These differences are embedded into datasets before any transformation occurs, making alignment downstream impractical. The Integration Hub removes this condition by enforcing a single, canonical ingestion pathway for each source.

Control at this boundary extends beyond initial ingestion. All physical data movement across the architecture is routed through the Integration Hub and re-ingested under the same governance conditions. Data does not move directly between zones or environments through independent pipelines. Any materialization or redistribution of data is treated as a new ingestion event, subject to the same validation, metadata registration, and control mechanisms. This ensures that movement does not introduce untracked divergence.

Within the Integration Hub, ingestion is structured through defined layers that enforce traceability and standardization. The landing layer preserves raw data in its original form, maintaining an immutable record of source input. This layer provides the basis for traceability, allowing all downstream datasets to be linked back to their origin without loss of fidelity. It is not used for transformation or consumption. Its purpose is preservation and reference. The staging layer enforces structural normalization and validation. Data is transformed into standardized formats, quality checks are applied, and metadata is registered. Structural inconsistencies, missing attributes, and quality violations are detected and resolved at this stage. Data that does not meet defined conditions does not proceed further into the architecture. Entry is conditional, and acceptance is governed by enforceable rules rather than discretionary decisions. Centralizing these layers prevents duplication of ingestion preparation across domains. Domains do not define their own landing or staging processes. They do not implement independent validation logic or metadata definitions. These responsibilities are architectural and must remain consistent. If domains replicate ingestion preparation, structural alignment is lost at the earliest stage, and divergence becomes embedded in every downstream process

.A critical operational principle within the Integration Hub is ingestion ownership. The first domain or consumer requesting a dataset assumes ownership of its ingestion pipeline. This ownership includes responsibility for maintenance, performance, and cost. Subsequent consumers do not establish new ingestion pipelines for the same source. They consume the existing pipeline. This eliminates duplication at the ingestion level and ensures that each dataset has a single controlled entry point into the architecture. Ownership is not static. It may transfer as organizational structures or usage patterns evolve. However, transitions are governed and must preserve continuity of control. Changes to ownership do not result in new pipelines or alternative ingestion pathways. They reassign responsibility within the existing structure, maintaining consistency in how data enters the system.

The Integration Hub is also the enforcement point for data quality and metadata continuity. Validation at ingestion ensures that structural anomalies are identified before data is propagated. Metadata - covering schema, lineage, ownership, classification, and processing context - is registered as part of the ingestion process. Data that is not registered within this framework does not exist within the governed architecture. It cannot be consumed, moved, or transformed through controlled pathways. This establishes metadata as a structural requirement rather than a supplementary artifact. Every dataset entering the architecture carries defined attributes that enable traceability, governance, and interoperability. Without this continuity at ingestion, downstream processes cannot maintain consistent lineage or enforce control.

Cost accountability is embedded at this boundary. Ingestion pipelines are centrally defined and owned, making resource consumption visible and attributable. Compute, storage, and data transfer costs associated with ingestion are linked to specific datasets and owners. Without this structure, independent pipelines proliferate, and resource usage becomes opaque. Cost increases without corresponding visibility, making optimization reactive and incomplete. The Integration Hub also defines constraints for AI workloads. Data science environments, model training pipelines, and experimentation platforms are not permitted to extract data directly from source systems. All datasets used for experimentation and training must originate from governed ingestion. This ensures that models are built on consistent, traceable data and that training pipelines do not introduce parallel ingestion pathways. Without this constraint, AI environments create isolated data ecosystems that diverge from the governed architecture.

By enforcing a single ingestion authority, the Integration Hub preserves both structural and semantic consistency. All downstream transformations, models, and analytical outputs are derived from data that has passed through a consistent entry process. This establishes a stable foundation for the entire architecture. Variation is controlled at the source rather than corrected after propagation.

The absence of this control leads to immediate and irreversible divergence. Independent ingestion pipelines introduce differences that cannot be reconciled without reprocessing data from origin. Downstream governance mechanisms - whether applied in transformation layers, semantic models, or consumption interfaces - operate on data that is already inconsistent. They cannot restore alignment because the inconsistency is embedded at entry. Architectural approaches that allow distributed ingestion attempt to manage this condition through standards, guidelines, or coordination. These mechanisms do not enforce behavior. Teams implement ingestion according to local interpretations, and variation accumulates. Over time, multiple representations of the same source data emerge, each governed independently. The architecture loses its ability to define a consistent baseline.

Centralized ingestion without structural enforcement does not resolve this problem. If ingestion processes are centralized but allow variation in implementation, the same divergence occurs within a single environment. Control must therefore be explicit and enforced. The Integration Hub does not provide centralized capability alone. It enforces uniform behavior at the point of entry. As data ecosystems expand and new sources are integrated, the importance of this boundary increases. Each additional source introduces potential variation. Without enforced ingestion control, the rate of divergence scales with the number of sources and consumers. The Integration Hub contains this growth by ensuring that each new source is integrated once and governed consistently from the outset.

Adoption of this model begins at the ingestion boundary. Structural control must be established before downstream environments are expanded or federated. If domains are allowed to define ingestion independently before this boundary is enforced, multiple pipelines become embedded in operational workflows. Retrofitting control becomes disruptive and often incomplete. Establishing ingestion authority first creates a stable foundation for subsequent architectural elements. Zones, virtualization layers, and cross-domain controls depend on consistent input. Without controlled ingestion, these elements operate on divergent datasets and cannot maintain coherence.

The Integration Hub is not a supporting component within the architecture. It is the control point that anchors governance, lineage, and consistency across the system. It defines how data enters, how it is registered, and how it becomes part of the governed environment.

Architectures that do not enforce ingestion control cannot sustain coherence under scale. They inherit divergence at entry and propagate it across every layer. The Integration Hub addresses this by making ingestion a structural operation, governed by explicit and non-negotiable constraints.

5. Enterprise and Business Zones: Federated Ownership Model

A federated architecture requires explicit separation between domain execution and enterprise control. Without this separation, organizations are forced into unstable extremes. Centralizing all analytical activity within a single environment constrains adaptability and limits domain responsiveness. Allowing domains to operate independently without structural boundaries removes discipline and produces divergence. A federated model must therefore define where autonomy is permitted and where it is constrained. DAC establishes this through a zonal structure that enforces ownership boundaries while preserving coherence across the system

.The zonal model is composed of two distinct constructs: the Enterprise Zone and Business Zones. These are not infrastructural partitions or deployment patterns. They define how ownership is distributed and how control is applied. Each zone represents a governed environment with specific responsibilities, constraints, and operating conditions. Together, they establish a system in which domains can execute independently without altering the structural behavior of the architecture. The Enterprise Zone is the corporate analytical layer. It is the environment where cross-domain integration occurs and where consistency across the enterprise is enforced. Datasets within this zone are aligned to shared models, governed reference data, and enterprise-level definitions. This is not an optional consolidation layer. It is the location where enterprise coherence is materialized. Analytical outputs that require cross-domain alignment, regulatory reporting, or enterprise-wide intelligence depend on this environment. In this zone, variation is not permitted. Consistency is mandatory.

The Business Zones represent domain-aligned environments where operational ownership resides. Each domain controls its datasets, transformation logic, and analytical workloads within its assigned zone. This control enables domains to operate with the flexibility required to meet their specific objectives. However, this autonomy is bounded. Domains do not define structural rules. They execute within them. Metadata standards, access control policies, lifecycle governance, and structural constraints are enforced at the architectural level and apply uniformly across all Business Zones. This separation between ownership and governance is the defining characteristic of the federated model. Domains own execution. They do not own structural control. They cannot define how data enters the system, how it is exposed to other domains, or how it moves across the architecture. These elements are enforced centrally and apply consistently regardless of domain-specific requirements.

If domains assume control over these aspects, the architecture transitions from federated to decentralized behavior.

The boundary between zones is strictly governed. Business Zones do not establish direct ingestion pipelines to source systems. They do not exchange data through independent mechanisms. All data entering the architecture or moving between zones must pass through the Integration Hub. This constraint ensures that ingestion and inter-zone movement remain controlled and traceable. Without it, domains create parallel pathways for data movement, introducing duplication and breaking lineage. Data produced within a Business Zone does not become available to other zones through direct access. It must be published through governed pathways. Publication is a controlled operation that registers datasets, enforces metadata standards, and ensures that outputs conform to structural requirements. Once published, data can be accessed by other zones under controlled conditions.

Zones operate as coordinated environments, not as independent systems exchanging data through unmanaged pathways.

The Enterprise Zone consumes data through these governed pathways. It does not extract data directly from Business Zones. Cross-domain datasets are constructed from published data that has been validated and registered within the architecture. This ensures that enterprise-level outputs are derived from controlled inputs. It prevents the introduction of untracked dependencies between zones and maintains consistency in how data is integrated. Logical access across zones is provided through the Virtualization Layer. This layer enables read-only access to datasets without requiring physical replication. Consumers access data through defined interfaces that enforce access control, masking, and classification policies. This approach decouples consumption from storage, allowing domains and enterprise workloads to access shared data without introducing duplication.

Physical data movement is governed independently of logical access. When datasets must be materialized across zones, the movement occurs through controlled pathways that preserve lineage and metadata continuity. Direct replication between zones is not permitted. This separation between logical access and physical movement prevents dependency sprawl and ensures that all materialization remains visible within the architecture. Cost governance reinforces ownership boundaries within this model. The Enterprise Zone operates as shared infrastructure, supporting cross-domain workloads and enterprise-level analytics. Its cost structure is centralized. Business Zones, by contrast, are accountable for their own resource consumption. Compute, storage, and processing costs incurred within a Business Zone are attributed to the owning domain. This separation ensures that domains are responsible for the efficiency of their operations and prevents cost from becoming opaque as the architecture scales.

Without this distinction, cost attribution fails.

Domains replicate datasets and execute transformations without visibility into existing assets, increasing consumption without accountability. The zonal model enforces cost ownership in alignment with execution ownership, ensuring that resource usage is both visible and controlled. Semantic consistency is maintained through shared metadata contracts and governed reference datasets. Domains are permitted to define local models within Business Zones, reflecting their specific analytical needs. However, when data is shared across domains or consumed within the Enterprise Zone, it must align with enterprise-defined semantics. Common business concepts, reference data, and classification rules are enforced to ensure interoperability.

Without this alignment, cross-domain analytics produce inconsistent results, even when structural constraints are maintained.

This model allows variation where it is appropriate and enforces consistency where it is required. Domains retain the flexibility to optimize for performance and use case requirements within their zones. They cannot redefine shared business concepts or alter structural rules that govern interaction with the rest of the architecture. The boundary between local optimization and enterprise consistency is explicitly defined and enforced. AI workloads operate within the same zonal structure. Domain-specific models are developed within Business Zones using governed datasets that originate from controlled ingestion. These models reflect domain-level requirements and operate within the constraints of the zone. Cross-domain models, which require integrated datasets and consistent semantics, are developed within the Enterprise Zone. This separation ensures that model training and execution remain aligned with the structural boundaries of the architecture. No independent AI environments are permitted outside this structure. Data science platforms, experimentation pipelines, and model training systems operate within Business Zones or the Enterprise Zone, depending on their scope. They inherit the same governance, metadata, and cost controls as other workloads. This ensures that AI does not introduce parallel data ecosystems that bypass architectural constraints.

Adoption of the zonal model requires structural enforcement. Business Zones must be defined along domain boundaries, establishing clear ownership of data and workloads. Ingestion must be controlled through the Integration Hub before domains are allowed to operate independently. The Enterprise Zone must be established to support cross-domain integration and ensure consistency. If domains are granted autonomy before these controls are in place, they establish independent ingestion and data exchange mechanisms. The architecture then reflects decentralized behavior rather than federation. This sequence is critical. Structural boundaries must be established before execution autonomy is expanded. Without enforced boundaries, domains optimize for local requirements and introduce variation that cannot be reconciled later. Federation cannot be achieved incrementally through coordination. It requires structural definition from the outset.

The zonal model defines how ownership is distributed and how control is maintained within a federated architecture. It allows domains to operate independently while ensuring that their operations do not alter the structural behavior of the system. It replaces reliance on coordination with enforced boundaries that define how data is managed across domains. DAC enforces this federated ownership model through explicit constraints. Domains execute independently within Business Zones, but they remain structurally bound by enterprise-defined rules. Data entry, cross-domain movement, and consumption are governed centrally and applied consistently across all zones.

Architectures that do not enforce these boundaries cannot sustain coherence under scale. Domain autonomy without structural constraint results in divergence. Centralized control without domain autonomy limits adaptability. The federated ownership model resolves this by separating execution from structural control and enforcing the boundary between them.

Scale amplifies both capability and risk. Without enforced ownership boundaries, that risk materializes as fragmentation. The zonal model ensures that expansion does not alter the structural integrity of the architecture.

6. Virtualization Layer: Unified Serving Plane

In a federated architecture, structural control must extend beyond ingestion and transformation to the point of consumption. Governing how data enters the system is insufficient if consumers can extract, replicate, or reinterpret data outside controlled pathways. Consumption is where divergence re-emerges if it is not constrained. DAC addresses this by enforcing a single, unified serving plane: the Virtualization Layer

.The Virtualization Layer is the mandatory interface through which all data consumption occurs. Analytical tools, reporting platforms, applications, and AI workloads access data exclusively through this layer. Direct access to storage within Enterprise or Business Zones is not permitted. This constraint is structural. It ensures that consumption follows a controlled and observable pathway, eliminating the ability for consumers to bypass governance by interacting directly with underlying data stores. This design removes the variability that typically arises at the point of access. In unconstrained environments, consumers extract data into local contexts, reshape it according to specific needs, and persist independent copies. Each extraction introduces a new version of the dataset, often without lineage, governance, or visibility. Over time, consumption patterns create parallel data ecosystems that diverge from the governed architecture.

The Virtualization Layer eliminates this condition by defining a single, enforced access mechanism.

Data within this layer is exposed through read-only, virtualized representations. Consumers interact with logical views rather than physical storage. These views present data in a form aligned with metadata definitions, access policies, and classification rules. The underlying datasets remain unchanged. Consumers cannot modify data through the interface, nor can they persist derived copies within the governed architecture through this pathway. The read-only constraint is not a usability decision. It is a structural requirement. By preventing write operations, the architecture ensures that consumption does not introduce new data states or uncontrolled transformations. Consumers are restricted to retrieval and computation within governed boundaries. This preserves the integrity of core datasets.

Cross-domain analysis is executed through federated query mechanisms within the Virtualization Layer. Data remains physically located within its respective zone, and queries are resolved by combining datasets logically at execution time. This approach avoids the need for physical consolidation while enabling integrated analysis across domains. Domain autonomy is preserved, as datasets are not moved or duplicated for the purpose of consumption. This separation between logical access and physical movement is fundamental. Logical access is provided through virtualized views. Physical movement occurs only through governed pathways defined elsewhere in the architecture. When data must be materialized - whether for performance optimization, regulatory requirements, or persistent storage - it is transferred through controlled mechanisms that preserve lineage and metadata continuity. Consumption interfaces are not used for replication.

The Virtualization Layer enforces governance dynamically at the point of access. Access control policies determine which datasets and attributes are visible to each consumer. Data masking is applied based on classification rules, ensuring that sensitive information is protected. These controls are evaluated at query execution time, providing consistent enforcement regardless of the consuming application or workload. Governance is embedded in the access mechanism rather than applied externally. All access through this layer is observable. Queries, dataset usage, and access patterns are recorded centrally. This provides a complete view of how data is consumed across the enterprise. It enables monitoring of usage trends, detection of anomalous access patterns, and attribution of consumption to specific domains or users.

Visibility is a direct consequence of enforcing a single consumption pathway.

Semantic consistency is maintained through controlled exposure. Data presented through the Virtualization Layer conforms to shared metadata definitions when cross-domain alignment is required. Domain-specific variations remain internal to Business Zones and are not exposed unless they meet defined structural and semantic conditions. This ensures that consumers interacting with shared datasets receive consistent representations, even when underlying implementations differ. The Virtualization Layer defines a single consumption pathway for all workloads. Business intelligence tools, reporting systems, and machine learning pipelines access data through the same interface. There are no alternative mechanisms for retrieving data within the governed architecture.

This constraint is particularly significant for AI workloads. Data science environments often require flexible access to diverse datasets for experimentation and model training. In the absence of control, these environments establish independent data acquisition pipelines, creating isolated feature stores and training datasets that diverge from governed sources. The Virtualization Layer prevents this by enforcing that all data used for experimentation and training is accessed through the same controlled interface or originates from governed ingestion pathways. By integrating AI workloads into the same consumption model, the architecture ensures that models are built on consistent, traceable data. Feature engineering, model training, and evaluation operate within the same governance, metadata, and access control frameworks as other analytical processes. This eliminates the creation of parallel data ecosystems specific to AI.

The Virtualization Layer also decouples consumption from underlying infrastructure. Consumers interact with logical representations of data rather than with physical storage or compute resources. Changes in storage technologies, data formats, or processing engines do not affect the consumption interface. This abstraction allows the architecture to evolve without disrupting analytical workloads. Stability at the interface preserves continuity for consumers even as the system changes internally. Adoption of the Virtualization Layer follows the establishment of ingestion control. Once data entry is governed through the Integration Hub, consumption must be constrained to prevent divergence at the point of use. If consumers are allowed to access data directly from storage before this layer is enforced, independent extraction patterns become embedded in workflows.

The sequence is therefore critical.

Ingestion establishes a controlled foundation. The Virtualization Layer ensures that this control is maintained during consumption. Together, they define the boundaries within which data operates. Skipping this sequence results in architectures where control exists at entry but is bypassed during use. The Virtualization Layer is not an optional enhancement. It is the control point that governs how data is accessed and interpreted across the architecture. Without a single, enforced, read-only consumption pathway, coherence cannot be sustained. Consumers will extract and replicate data to meet local requirements, reintroducing fragmentation regardless of upstream controls.

Architectures that rely on multiple access mechanisms cannot maintain consistent governance.

Each pathway introduces variation in access control, data interpretation, and visibility. Over time, these variations accumulate, and the architecture loses its ability to provide a unified view of data consumption. The Virtualization Layer eliminates this condition by enforcing a single interface. All data access is mediated, observable, and governed. Logical access is decoupled from physical storage. Replication is controlled. Semantic consistency is maintained at the point of exposure. Control at consumption completes the structural model.

Data enters through governed ingestion, moves through controlled pathways, and is accessed through a unified serving plane. Removing control at any of these points reintroduces divergence. The Virtualization Layer ensures that consumption remains within the boundaries defined by the architecture, preserving coherence as the system scales.

7. Absorbing AI Within Federation

Artificial intelligence does not require a separate architectural domain. It introduces new processing patterns, not new structural requirements. In a federated architecture, AI is treated as a workload class operating within the same boundaries that govern all analytical processes. This is not a design preference. It is a structural condition. If AI operates outside the architecture, it introduces independent ingestion, storage, and transformation pathways that undermine coherence. The perception that AI requires dedicated platforms has led organizations to create isolated environments for experimentation, model training, and feature engineering. These environments are typically provisioned to provide flexibility and speed, but they do so by bypassing established controls. Data is extracted from governed systems, copied into local environments, and reshaped independently. These actions introduce new versions of datasets that are not aligned with the architectural model.

The consequence is immediate divergence. Independent extraction pipelines introduce variation at the point of data acquisition. Feature engineering processes operate on datasets that are no longer synchronized with governed sources. Models are trained on data that cannot be traced consistently across environments. These conditions are not temporary. Once established, they become embedded in operational workflows. The structural issue is not the presence of AI workloads. It is the introduction of parallel data ecosystems to support them.

Dedicated AI platforms replicate ingestion, storage, and governance capabilities already defined within the architecture.

Each replication introduces a separate control surface, weakening visibility and consistency. Over time, the architecture no longer represents a single system. It becomes a set of loosely connected environments, each maintaining its own data and logic. DAC eliminates this condition by requiring that all AI workloads operate within the existing federated structure. There are no separate AI environments. Enterprise and Business Zones provide the execution contexts for all AI development, training, and inference. This ensures that AI operates on governed datasets and remains subject to the same structural constraints as other analytical processes. The placement of AI workloads follows the same zonal logic that governs other operations. Cross-domain models are developed within the Enterprise Zone, where integrated datasets and aligned semantics are available. Domain-specific models are developed within Business Zones, where local data and transformations are controlled by the domain.

This separation ensures that model scope aligns with data scope.

Cross-domain dependencies required for AI are resolved through governed pathways. Data required from multiple domains is integrated through the same controlled mechanisms that support enterprise analytics. Direct access between zones for the purpose of model training or feature extraction is not permitted. Allowing such access would bypass structural controls and reintroduce untracked dependencies. All datasets used for AI - whether for training, validation, or feature engineering - originate from governed ingestion. Data scientists do not extract data directly from source systems or establish independent acquisition pipelines. They access datasets through the same controlled pathways available to other consumers. Feature engineering operates within zone boundaries. Features are derived from governed datasets and remain linked to their source data through metadata and lineage. Independent feature stores that exist outside the architecture are not permitted. When features are treated as separate, unmanaged assets, they become disconnected from source data and cannot be reused consistently across models. By enforcing feature development within zones, features remain governed and interoperable.

Features are registered as data assets within the architecture.

Their definitions, transformations, and dependencies are captured in metadata alongside other datasets. This ensures that feature reuse is controlled and that changes to underlying data can be traced through to dependent models. Model artifacts are also managed within the zonal structure. Models, training datasets, evaluation metrics, and configuration parameters are stored in the zones where they are developed. There are no separate storage environments for AI artifacts. This ensures that models remain connected to the datasets and transformations from which they are derived. Separation of model artifacts from governed data introduces gaps in lineage and limits visibility into model behavior. AI workloads do not introduce independent storage layers. Data scientists operate directly on datasets that exist within the architecture. They do not create persistent copies of data in isolated environments for the purpose of experimentation. This constraint prevents duplication and ensures that all data used in AI workflows remains governed. It also eliminates the need to reconcile differences between local and governed datasets.

Inference follows the same architectural pathways as other analytical outputs. Predictions generated by models are exposed through the Virtualization Layer. They are subject to the same access control, masking, and classification policies as any other dataset. Independent mechanisms for distributing model outputs are not permitted. This approach ensures that AI outputs remain integrated within the architecture. Predictions can be traced back to the models that produced them, the features used during training, and the source data from which those features were derived. This continuity is essential for auditability, compliance, and operational reliability. Metadata continuity extends to all components of the AI lifecycle. Models, features, training datasets, and transformation logic are registered alongside other architectural elements. This provides a complete lineage from source data through feature engineering to model outputs.

The AI lifecycle - development, training, deployment, monitoring, and retraining - operates entirely within the zonal structure. Each stage is governed by the same controls that apply to other workloads. Alternative lifecycle pathways, where models are developed or deployed outside the architecture, are not permitted. Such pathways introduce inconsistencies that cannot be reconciled within the governed system. As AI adoption scales, the pressure to create independent environments increases. Teams seek flexibility to experiment and iterate rapidly. Without structural constraints, this results in the proliferation of isolated environments, each with its own data and models. These environments may deliver short-term capability, but they introduce long-term fragmentation that cannot be contained through governance. Embedding AI within the federated architecture addresses this pressure by providing controlled flexibility. Domains retain the ability to develop and optimize models within their zones. Cross-domain capabilities are supported through the Enterprise Zone. All operations occur within defined boundaries that preserve coherence.

These constraints are not restrictive.

They define the conditions under which AI can operate without introducing divergence. Removing them results in parallel systems that compete with the architecture rather than extending it. AI does not introduce new structural requirements. It amplifies existing weaknesses. Where ingestion is uncontrolled, AI accelerates data extraction. Where movement is ungoverned, AI introduces new dependencies. Where consumption is unconstrained, AI creates isolated outputs.

The presence of AI exposes whether structural control exists.

DAC addresses this by embedding AI within the same enforced structure that governs all data operations. AI workloads do not bypass the architecture. They operate within it. This ensures that as AI capabilities expand, they do so within a system that preserves consistency, traceability, and control.

Artificial intelligence does not break architecture. It reveals whether architecture was ever enforced.

8. Governance Blueprint and Enforcement

In DAC, governance is not defined through policy, documentation, or procedural oversight. It is embedded in the structure of the architecture. Data cannot enter, move, or be consumed unless it passes through enforced control points. Governance does not depend on adherence. It is executed through the way the system operates. If governance can be bypassed, it does not exist

.The Governance Blueprint defines the structural constraints that apply across all layers of the architecture: the Integration Hub, Enterprise Zone, Business Zones, and the Virtualization Layer. These constraints are not recommendations. They define how metadata is managed, how data is classified, how access is controlled, how quality is enforced, and how lifecycle and compliance requirements are applied. Domains operate within these constraints, but they do not modify them. Governance is uniform, regardless of domain-specific implementation. At the centre of this blueprint is a mandatory metadata spine. No dataset, pipeline, transformation, or analytical artifact exists within the architecture without metadata registration. Metadata captures definitions, ownership, lineage, classification, and quality indicators. This registration occurs at ingestion and is maintained continuously across all zones. It extends beyond datasets to include AI artifacts such as models, features, and training lineage.

Metadata is not descriptive. It is the mechanism through which governance is enforced.

Data that is not registered in metadata is not part of the architecture. It is not visible through governed interfaces, cannot be accessed, and cannot participate in analytical processes. This condition is absolute. It ensures that all data within the system is traceable and governed from the point of entry through to consumption. Access control is enforced centrally across all layers through a unified identity and authorization model. Policies are applied consistently at ingestion, storage, transformation, and consumption. There are no alternative pathways for access. Direct interaction with data outside governed interfaces is not permitted. This ensures that all data access is mediated, controlled, and observable. The enforcement of governance is inseparable from the topology of the architecture.

The system defines where data can enter, how it can move, and how it can be consumed. These pathways are fixed. Domains cannot redefine them. The Integration Hub governs ingestion and cross-zone movement. The Virtualization Layer governs consumption. Data operations that occur outside these pathways are not supported. Governance is enforced through structure, not through oversight. Data quality is enforced as a condition of entry. At ingestion, datasets are validated against structural and semantic requirements. Data that does not meet these conditions is rejected and does not enter the governed environment. This establishes a baseline of consistency that is preserved across all zones. Domains may apply additional quality controls within their environments, but they cannot reduce or bypass enterprise-defined standards. This approach ensures that quality is not treated as a downstream concern.

It is embedded at the point where data becomes part of the architecture. Once admitted, datasets carry their quality attributes through all subsequent transformations and consumption processes. This continuity allows quality to be evaluated consistently across the system. AI artifacts are governed under the same structural model as datasets. Models, features, and training datasets are treated as governed assets. They must be registered in metadata, linked to their source data, and traceable across the lifecycle. Independent management of AI artifacts outside the architecture is not permitted. This ensures that model behavior can be understood, reproduced, and governed in the same way as other analytical outputs.

The governance model extends to deployment.

Promotion of datasets, transformations, or models into production environments is gated by structural validation. Metadata completeness, lineage traceability, access control enforcement, and data quality must meet defined conditions. Artifacts that do not satisfy these requirements do not progress. Deployment is not a discretionary process. It is controlled by enforceable criteria. Observability is inherent to this design. Because all ingestion, transformation, and consumption occur within governed pathways, the architecture provides a complete and continuous view of data activity. Dataset usage, transformation dependencies, access patterns, and model interactions are visible without additional instrumentation. This visibility is a direct result of enforcing a single structural model for all operations. The elimination of bypass mechanisms is a defining characteristic of the Governance Blueprint.

Shadow pipelines, direct data extraction, unmanaged storage environments, and independent transformation processes are not supported. The architecture does not provide pathways for activity outside governed boundaries. All data operations must occur within the defined structure. This removes the need for reactive governance measures. When bypass is not possible, enforcement becomes inherent. Governance is no longer dependent on detecting and correcting deviations. It prevents them.

Adoption of this model requires establishing non-negotiable conditions at the outset. Metadata registration must be enforced for all data assets. Access control must be applied through a unified model. These elements define the baseline for governance. Without them, governance remains procedural and cannot scale as complexity increases. Deployment gating follows as a structural extension of these conditions. By enforcing validation before promotion, the architecture ensures that only governed and compliant artifacts become operational. This prevents the introduction of inconsistencies into production environments and maintains integrity across the system.

The Governance Blueprint defines governance as infrastructure.

It establishes the boundaries within which all data activity occurs. Data flows, analytical processes, and AI workloads are governed because the architecture enforces how they operate. There is no distinction between governance and system behavior. As the architecture scales, these controls remain consistent. Additional domains, datasets, and workloads operate within the same structural constraints. Autonomy is preserved within zones, but structural integrity is not compromised. Governance does not degrade as complexity increases because it is not dependent on coordination or compliance. Architectures that treat governance as policy cannot maintain this consistency. Policies rely on interpretation and adherence. As systems expand, adherence varies, and governance weakens. Structural enforcement removes this variability by defining what is possible within the system.

The Governance Blueprint ensures that governance is not an overlay applied to the architecture. It is the architecture.

9. Operational and Economic Implications

Architecture determines how systems behave under load and how cost accumulates as they scale. In DAC, operational responsibility and economic accountability are not managed through reporting, allocation exercises, or governance committees. They are enforced by structure. Every component has a defined role, every action occurs within a controlled pathway, and every cost is attributable to a specific owner. If responsibility and cost cannot be traced structurally, the architecture has already lost control.

The topology establishes ownership by location, not by agreement. The Integration Hub owns ingestion and cross-zone movement. The Enterprise Zone maintains enterprise-level datasets and cross-domain capabilities. Business Zones own domain transformations, models, and analytical workloads. The Virtualization Layer governs consumption. These responsibilities are fixed. They do not shift based on project needs or organizational preference. Structural position determines accountability, and that accountability is enforced through how the system operates. This eliminates ambiguity in operational ownership. Teams are responsible for the data assets, pipelines, and compute they execute within their zones. They do not negotiate ownership at runtime, and they do not inherit responsibility for operations outside their boundary. Cross-zone interactions do not rely on coordination between teams. They are resolved through governed pathways that define how data moves and how responsibility is maintained.

The primary constraint that anchors both operational discipline and cost control is singular ingestion.

Data enters the architecture once through the Integration Hub. Independent pipelines to the same source are not permitted. This removes the most common source of duplication and divergence. When multiple pipelines exist, each introduces its own extraction logic, refresh cadence, and processing cost. These differences propagate across the system, creating both inconsistency and unnecessary expenditure. Singular ingestion enforces a shared foundation. Downstream environments consume governed datasets rather than recreating ingestion. Domains do not replicate pipelines to meet local requirements. They operate on the same data entry point, ensuring that both structure and cost remain consistent. This constraint defines how data acquisition is performed across the architecture.

Cost follows directly from this model. The domain that initiates ingestion assumes ownership of the associated infrastructure. This includes compute for extraction and transformation, storage for landing and staging, and ongoing operational overhead. There is no shared or unallocated ingestion cost. If a domain requires higher ingestion frequency, additional validation, or increased data volume, it bears the corresponding cost. Responsibility is explicit. Subsequent consumers do not establish new ingestion pipelines. They consume existing datasets through governed pathways. Cost sharing between domains is handled through financial mechanisms external to the architecture. It does not result in duplication of pipelines or data. The architecture enforces reuse; cost distribution is managed outside the data flow. This establishes two non-negotiable conditions.

Data is ingested once, and all consumption is attributable. There is no concept of free data acquisition within the system. Every dataset has a defined owner, and every unit of compute or storage consumed can be traced to a specific domain or enterprise function. Without this alignment, duplication persists because the cost of redundancy is not visible to those creating it. Within Business Zones, domains operate under defined compute and storage allocations. All transformation processes, model training activities, and analytical workloads consume resources that are owned and funded by the domain. There is no unrestricted central compute pool that domains can draw from without accountability. Resource usage is directly tied to domain responsibility.

This constraint enforces operational discipline.

Domains must optimize their workloads because inefficiency results in direct cost impact. They cannot externalize the cost of excessive computation, redundant transformations, or unnecessary storage. The architecture ensures that operational decisions have immediate economic consequences. The Enterprise Zone operates under a different model. It functions as shared infrastructure supporting cross-domain integration, enterprise analytics, and regulatory requirements. Its cost is distributed across the organization, but its scope is constrained. It is not a general-purpose environment for domain workloads. It exists to maintain enterprise-level coherence and support use cases that require integrated data. This separation prevents the Enterprise Zone from becoming a default execution environment.

Data movement is governed with the same level of control. Physical transfer of data between zones occurs only through the Integration Hub and requires justification. Movement is not initiated as a convenience. It is a controlled operation that preserves lineage and ensures visibility. Uncontrolled replication between zones is not permitted. Logical access through the Virtualization Layer is the default mechanism for data consumption. Consumers access data without creating physical copies, reducing storage duplication and data transfer costs. This separation between logical access and physical movement ensures that replication occurs only when necessary and under governed conditions. When this constraint is not enforced, storage duplication increases without visibility. Data is replicated across environments to meet local needs, and each copy incurs storage and maintenance costs. Transfer costs accumulate as data is moved repeatedly across systems. These costs are often hidden because they are distributed across teams and environments. The architecture eliminates this by enforcing controlled movement and logical access.

AI workloads follow the same operational and economic model. Training, feature engineering, and inference consume compute and storage resources within the zones where they are executed. Domains are accountable for the cost of domain-specific models, while enterprise-level models operate within the Enterprise Zone under shared cost structures. There are no separate AI infrastructure environments that operate outside this model. This constraint prevents the accumulation of untracked cost associated with isolated data science environments. When AI workloads operate independently, they duplicate datasets, create separate storage layers, and consume compute resources without visibility. By embedding AI within the same structural boundaries, all resource usage remains attributable and governed.

Operational efficiency emerges from these constraints. Ingestion, transformation, and consumption follow defined pathways, eliminating ad hoc integration patterns and parallel systems. There is no need to reconcile multiple pipelines, align duplicated datasets, or manage conflicting transformations. The architecture reduces operational overhead by preventing the conditions that create it. Reliability is structured rather than managed. Ingestion pipelines within the Integration Hub operate as shared or dedicated infrastructure with defined service levels. Downstream environments depend on these pipelines, so improvements to ingestion propagate across the system. Stability at the point of entry ensures stability throughout the architecture. Audit and compliance are inherent properties of the system.

All datasets, transformations, and models are registered within the metadata framework, maintaining complete lineage from source to consumption. Audit processes do not require reconstruction of data flows across independent systems. Traceability is embedded in the architecture, enabling continuous compliance without additional effort. These constraints introduce trade-offs. Centralized ingestion restricts direct access to source systems. Unified consumption through virtualization may introduce latency compared to direct data access. Controlled data movement limits the ability to replicate datasets for local optimization. These limitations are not unintended consequences. They are enforced conditions that prioritize structural integrity, governance, and cost accountability over unrestricted flexibility.

Flexibility within this architecture is not eliminated. It is constrained to defined boundaries. Domains retain control over their transformations, models, and analytical processes within Business Zones. They can optimize execution, adjust processing logic, and evolve their workloads. What they cannot do is alter the structural rules that govern how data enters, moves, or is consumed. Adoption of this model begins with cost attribution at ingestion and enforcement of compute allocation within zones. Without clear ownership of cost, domains continue to create redundant pipelines and duplicate datasets because the economic impact is not visible. Once cost aligns with responsibility, duplication becomes self-limiting. Structural constraints reinforce themselves through economic accountability.

This alignment creates a system where operational discipline is not enforced through governance intervention.

It is a direct consequence of how the architecture is constructed. Teams operate within defined boundaries, costs are visible and attributable, and deviations are not possible without structural change. As the architecture scales, these properties remain stable. Additional domains, datasets, and workloads do not introduce uncontrolled cost or complexity because they operate within the same constraints. The system expands without altering its operational behavior.

In DAC, operational discipline and economic accountability are not outcomes that must be achieved through management. They are enforced properties of the architecture itself.

10. DAC Adoption and Architectural Doctrine

DAC adoption does not begin from a defined baseline. It is introduced into environments where architectural control has already been fragmented through independent ingestion pathways, uncontrolled data movement, and inconsistent consumption patterns. These conditions are not transitional artifacts; they are embedded in operational systems and cannot be removed prior to adoption. As a result, transition does not proceed through replacement, but through the progressive enforcement of structural constraints within a partially governed environment. Controlled and uncontrolled states coexist, and the architecture emerges through the reduction of permissible variation over time. This requires explicit sequencing, sustained enforcement, and acceptance of intermediate states that do not yet reflect full coherence. The objective is not immediate alignment, but directional convergence toward a system where structural control is uniformly applied.

Adoption is a structural intervention, not a deployment exercise.

New platforms do not create alignment. Additional tooling does not restore coherence. The architecture is established by enforcing constraints on how data enters, moves, and is consumed. Systems continue to operate during this process, but they do so under progressively restricted conditions. The architecture does not adapt to existing behavior. Existing behavior is constrained until it conforms. The system enters a dual-state condition. One part begins to operate under enforced structure. Another continues under legacy patterns. These states coexist. Data flows across both. Dependencies span both. This is not an exception. It is the expected state of any transition. The objective is not immediate uniformity, but directional control. The governed portion must expand. The uncontrolled portion must contract. Expansion of uncontrolled behavior is the primary risk.

If new workloads are allowed to replicate existing patterns, the architecture diverges faster than it converges. Independent ingestion continues. Unmanaged consumption persists. Parallel data paths multiply. The governed system becomes peripheral. This condition cannot be reversed through later alignment. It must be prevented at the point of introduction. Containment is therefore mandatory. Legacy systems may remain, but they are isolated. They do not define new patterns. They are not extended into new workloads. All new data activity must conform to enforced constraints from the outset. This creates a structural asymmetry. The past is tolerated. The future is controlled. This sequencing is not optional.

Entry is enforced first. Access is constrained next. Movement is governed after these controls are stable. Domain execution follows within defined boundaries. Analytical workloads, including artificial intelligence, are aligned last. Any deviation introduces alternative pathways that persist beyond the transition and undermine structural control. Enforcement must be absolute. Constraints applied selectively do not hold. The system will converge toward the least restrictive behavior available. If exceptions are introduced, they become alternative pathways. Once established, they are reused and expanded. Control becomes negotiated rather than enforced, and fragmentation resumes.

Exceptions are therefore temporary and governed. They are defined explicitly and scheduled for removal. Persistent deviation is not an operating model. It is structural failure. The architecture does not accommodate alternative pathways that achieve equivalent outcomes. There is one way data enters, a controlled way it moves, and a defined way it is consumed. Adoption is measured by convergence. The proportion of workloads operating within enforced constraints must increase continuously. The proportion operating outside must decrease. Absolute completion is not required at each stage, but direction is. If uncontrolled elements persist without reduction, the architecture is not converging. It is accumulating additional layers without resolving structural inconsistency.

This transformation is not driven by tooling.

Technologies may vary. Platforms may evolve. What changes is behavior. All systems must conform to the same structural rules regardless of implementation. The architecture governs how data flows, not which tools are used to process it. Completion is defined by elimination. Uncontrolled ingestion is removed. Unmanaged movement is eliminated. Ungoverned consumption is no longer possible. At this point, the system operates under a single set of enforced pathways. Federation becomes operational, not conceptual.

DAC defines both the constraint and the outcome. It does not describe how systems should behave. It enforces how they must behave. Systems that operate within these constraints remain coherent as they scale. Systems that do not are not partially aligned. They are outside the architecture.

There is no intermediate state. Structural control either exists or it does not.

11. Appendix with all diagram

Who is Mustafa Qizilbash:

Mustafa Qizilbash is a Data and AI architecture professional, with more than 25+ years of experience, focused on redefining how modern enterprises design and operationalize data ecosystems. His work centres on creating practical architectural models that balance decentralized domain ownership with centralized governance—enabling organizations to scale data, analytics, and AI in a controlled yet flexible manner. Mustafa’s contributions to the field include Five books on Data & AI, over 1000+ publications, and Three patents, mentioned below. With experience spanning more than Seven countries, he has worked with leading global enterprises such as Oracle, IBM, Teradata, Petronas, Hong Leong Bank and many others.

He is the originator of the Data & AI Cognitive (DAC) Architecture, a structural approach to integrating AI within enterprise data platforms, along with the Value Discovery Canvas (Four 4s Formula) and the Productionizable Viable Product (PVP) Approach, which collectively bridge the gap between business intent and production-grade implementation.

Mustafa is also actively engaged as an advisor with Your Partner Technologies ( https://www.yourpartnertechnologies.com ), supporting enterprise-scale Data & AI initiatives. In addition, he hosts the podcast series “Let’s Talk About Data!” ( https://www.youtube.com/@letstalkaboutdata ), where global Data & AI leaders from over 35 countries share insights, experiences, and perspectives on the evolving data landscape.

Why Him!

Most businesses are already making decisions, but they don’t fully trust the data behind them. The issue isn’t a lack of data; it’s that different reports don’t match, definitions vary, and teams end up debating numbers instead of acting.

He help companies make faster, more confident decisions by ensuring they can trust their data. He typically come in as a Data and AI Professional/Advisor, leading focused engagements to identify where trust breaks across dashboards, reports, definitions, and data sources, and fixing those gaps end-to-end.

The engagement starts with the Directors and C-Level executives where decisions are frequent and the cost of being wrong, or slow is high. He works closely across all levels to uncover where numbers don’t align, where gaps become visible quickly. From there, IHealign business and IT around a single, reliable view of the data.

The result is simple: when a number shows up in a meeting, nobody questions it, they act on it.

(HOW TO KEEP CHATGPT FROM EATING YOU OUT OF HOUSE AND HOME)

By W H Inmon

Dave Rapien

The world has been taken by the power of generative AI. ChatGPT and others have come forward with technology that opens the door to “common” people having access to awesome computing abilities with a natural interface. Much of the old ways of analytical processing have died with the advent of generative AI.

At the foundation of generative AI is technology known as LLM – large language models. Simply put, for generative AI technology to work, raw text is read, reduced to tokens and placed in the LLM. The LLM then uses the tokens in order to conduct an analysis of the text that is presented to it for analysis.

DIFFERENT TYPES OF LLM’S

There are two different kinds of LLM’s – one LLM is serving the universe. That kind of LLM is what is found in ChatGPT. It applies to the large body of users that can ask any sort of question.

The second kind of LLM is the one that serves the corporation. It is a corporate LLM or a private LLM.

The universal LLM can contain a very wide variety of data. The corporate LLM applies only to private corporate data, such as proposals, call center conversations, legal contracts, corporate projections of sales, and so forth.

There are many good reasons why private corporate data should never find its way into a universal LLM.

The conversations in this paper apply to the corporate LLM, not the universal LLM.

THE RANDOMNESS OF RAW TEXT

Raw text that is fed to the LLM can come from anywhere – voice recordings, newspapers, spreadsheets, the Internet, emails, and more. Furthermore, there is no order to raw text. A person can say or write whatever they want.

Raw text is simply random.

As stated, at the heart of generative AI is the LLM – the large language model. In many ways the LLM is analogous to the brain of the human body. Information is stored, organized and used there in order to control and manage the other functions of the technology.

INGESTING RAW TEXT

The LLM is created from the ingestion and structuring of raw text. The contents of the LLM and the volume of the LLM all depend on the raw text that is used to form the contents of the LLM’s vector database.

What is not understood by most is that the massive volume of text shaping the LLM has a profound effect on the cost and speed of the analytical processing that will occur after the LLM is built. But it is a fact, the volume of text fed to the LLM for ingestion does have a profound effect on performance and resources consumed once the analytical processing of the LLM starts to occur.

SEPARATING RAW TEXT

Let’s step back and examine what the contents of the raw text entering the LLM might look like. In particular let’s look at how much of the raw text is devoted to the actual business of the organization versus text in the document that is extraneous to the business of the organization.

In the diagram that is shown, ingested text that is relevant to the business of the organization is shown in black and ingested text that is irrelevant to the running of the corporation is shown in red.

It is seen that the LLM can be likewise classified. The irrelevant raw text turns into the part of the LLM that contains irrelevant contents. And by the same token, the relevant raw text turns into a portion of the LLM that is relevant to the business of the corporation.

There are many ways to characterize this division of relevancy. One way to characterize it is to say that the black portion of the diagram is the muscle of the LLM and the red portion of the LLM is the flab.

Business analytical processing can proceed nicely with no reference to or mention of the flab. The flab just gets in the way and makes analytical processing slower and more expensive.

FLAB

Another way of thinking of flab is that the flab is unnecessary and a costly drag on operations.

It is absolutely normal for the organization to gather flab in its interior when raw text is just fed to the LLM. Some of the raw text is useful, some of the raw text is not.

SO WHY SHOULD I CARE?

So why should an organization pay attention to the flab in the raw text feeding the LLM? Is it really important? Does it really matter

A CASE STUDY

In order to answer this question in the context of reality, a case study was done. A banking procedures document was read into the systems in order to build an LLM using Retrieval Augmented Generation or RAG, which is a technique to improving LLM’s accuracy and reducing hallucinations. The banking procedures document was divided into business relevant text and business irrelevant text.

In the diagram shown, the highlighted text is text that is relevant to banking business and the text that is not highlighted is text that is extraneous to the conduct of business.

A CASE STUDY

In order to answer this question in the context of reality, a case study was done. A banking procedures document was read into the systems in order to build an LLM using Retrieval Augmented Generation or RAG, which is a technique to improving LLM’s accuracy and reducing hallucinations. The banking procedures document was divided into business relevant text and business irrelevant text.

In the diagram shown, the highlighted text is text that is relevant to banking business and the text that is not highlighted is text that is extraneous to the conduct of business.

When a word that is relevant to business is encountered, the text surrounding the word – a chunk or a snippet – is collected. The chunk or snippet is then collected into a data base where the word, the classification or context of the word is determined, and the source of the word is stored.

The data base then is used to preprocess raw text before the text is included into the LLM

.SO, WHAT ARE THE COSTS?

An interesting question then arises. The question is this – how much does it actually cost you to include extraneous text along with business relevant text for inclusion into the LLM?

You are going to be shocked at what was found in the case study that was conducted.

In order to be clear about the calculation that will be presented, we need to be crystal clear about what is being compared. The first part of the calculation refers to the inclusion of only the business relevant text. The second part of the calculation refers to the inclusion of all the text – both flab and business-related text – into the building of the LLM.

In the case study there were 440 words that were found that were deemed to be business relevant. The document in its entirety encompassed 1600 words. 1160 of those words were flab, or words that were not deemed to be directly relevant to the business of the corporation

.CORELATIVE ANALYSIS

In order to make the calculation you need to know that when a query is done, that a corelative analysis against all tokens in the LLM is done. The more tokens you have, the more comparisons that have to be made. And each comparison costs a tiny bit of energy in the computer.

The simple concept for the comparisons of tokens that need to be made is this –

Lower number of tokens x number of tokens = Lower total energy required to do each analysis.

Using the relevant text, with the Textual ETL supported context, the LLM reduced it token usage, and thus energy costs, and thus real costs by 92%! The amount of hallucinations were reduced by almost 20%! So the results were better and the costs were drastically reduced.

THE TRUE COST OF FLAB

By adding flab to the mix, the organization has raised the amount of energy in the computer by a factor of almost 10. In other words, the computer had to expend almost 10 times the amount of energy to process the entire document as opposed to the amount required to simply process the business relevant data.

The organization has consumed over an order of magnitude of energy by adding irrelevant data to the LLM. Furthermore, there’s no business payback for the irrelevant text that has been added. And even more – the addition of the irrelevant text increased the hallucinations produced by the LLM in it analytical processing.

If there were any business value that had been added by including extraneous text, then it might be useful to add the irrelevant text. But there is no business value that has been added. Instead, the organization has placed an anchor on its speedboat

.

ON A QUERY-BY-QUERY BASIS

Not only is the energy factor interesting, the waste of computer energy becomes really interesting when it is understood that EACH new query triggers this waste of energy. Every time anyone does any query, a huge amount of computer energy is wasted. Furthermore, the answers that include the FLAB were misleading or just plain wrong more than 20% of the time. Result quality increased and costs decreased by using Textual ETL as part of the preprocessing of your LLM

.A SECRET (AND KEEP IT A SECRET!)

So, there is a secret. (Don’t tell this to anyone.)

If you want to save a LOT of money and streamline your processing, just load business relevant text into your LLM.

A LOT OF MONEY!!

LLM MANAGEMENT

So how can you do this? Just give LLM Management organization a call and we will show you just how easy it is to separate business relevant text from extraneous text.

Give Carol Renne a call at – crenne54@gmail.com

Why Data Architecture Needs Unit Economics in the Agentic Era

By Karl Ivo Sokolov and Mario Meir-Huber

Abstract

Data architecture has always mattered but the nature of its consequences has changed. In the agentic era, architectural decisions carry asymmetric financial weight that most enterprises are poorly equipped to evaluate. We introduce the concept of the Finance Layer, a discipline that connects platform choices to P&L outcomes. Our argument is that the current “agentic transformation” is, at its core, a data architecture transformation with less familiar economics. The February 2026 SaaS-software selloff offered a preview: data infrastructure held value while application-layer software repriced dramatically. Organizations treating agentic AI as an application-layer initiative, bolting agents onto existing infrastructure, are investing heavily while remaining structurally unprepared to capture value. We explore the addressability problem that “clean data for AI” fails to solve, how architectural efficiency functions as a financial variable, and why platform decisions now carry asymmetric consequences that demand new governance approaches including strategic decommissioning.

The Addressability Problem

Many consultants have arrived with a familiar pitch: “You need clean data for AI.” And while the statement sounds reasonable it may be inadequate in ways that could cost organizations years of misdirected effort.

“Clean data” is a quality attribute that asymptotes quickly. You can spend two years improving data quality from 85% to 92%, and the marginal value of each percentage point diminishes as you approach it. The deeper problem is that “clean” remains undefined in the agentic context. Clean for what purpose? Clean according to whose schema? Clean data sitting in a warehouse that agents cannot programmatically address is clean but useless.

When we began working with enterprises preparing for agentic deployment, we found that the organizations struggling most were the ones that had invested heavily in data quality without investing in the infrastructure required for machines, rather than humans, to consume that data. The shift required is from cleanliness to addressability. Addressability means an agent can locate the data it needs, understand what the data represents through proper semantic modeling, act on the data within defined guardrails, be traced for what it did, and have its costs allocated to the appropriate business unit.

The “clean data for AI” framing solves yesterday’s problem by preparing data for human analysts who can interpret ambiguity, navigate incomplete metadata, and mentally translate between systems. Agents do this poorly and expensively, burning tokens on context reconstruction that a properly architected semantic layer would have made unnecessary.

The practical difference in investment is significant. A data quality initiative might cost $2M and take 18 months. A data architecture transformation for agent readiness, including semantic layer construction, API exposure with governance, and proper cost attribution mechanisms, might cost $15M and take three years. The first feels safer and shows progress in dashboards. The second is what actually unlocks the value that everyone is projecting in their AI business cases.

The Frey Formula: Architecture as a Financial Variable

At the Vienna Engineering Management Conference in February 2026, Mathias Frey, Chief Engineering Technology Officer of Erste Digital at Erste Group, Austria’s largest bank, presented a formulation that captures what most data leaders intuitively know but rarely express in terms that Finance departments can evaluate:

V ∼ D/p = C = logₐ(n)

The variables: D represents demand (the work the organization needs done), p is a product capability multiplier reflecting how effectively that demand is managed and filtered, C is capacity (what can actually be delivered), n is the number of people, and a is the architecture efficiency coefficient, the base of the logarithm.

The formula’s power lies in what it makes visible. Capacity scales logarithmically with headcount, and the steepness of that curve is determined by architecture. When a = 10, you need ten times the people to produce one additional unit of output. When a = 2, you only need twice the people. The gap between those scenarios represents the difference between an organization that scales and one that drowns in coordination costs.

Frey’s observation that matters most: AI impacts n (the people variable), but does not fundamentally change p (product capability) or a (architecture efficiency). You can deploy agents against poorly architected data, and the agents will fail faster, or worse, confidently produce wrong outputs at scale. The architecture coefficient remains the binding constraint.

As organizations compress headcount through AI augmentation, the architectural efficiency variable becomes more consequential, not less. In a world where n is being actively compressed, a becomes the primary lever for value creation. And a is determined by decisions made in data architecture, decisions that were often made years ago, for different reasons, with different constraints. Architecture belongs in investment discussions, not as a technical appendix but as a parameter that determines what productivity gains are achievable and at what cost.

What “Agentic” Actually Means for Data Architecture

Enterprise software vendors are pushing what they call the “agentic layer” into sales conversations with remarkable intensity, Microsoft with Copilot, Salesforce with Agentforce, SAP with Joule, and others. The marketing suggests revolution. The underlying architecture suggests evolution.

Strip away the branding, and the agentic layer is a combination of API enablement, semantic modeling, orchestration infrastructure, and governance planes. Data architects have been building these components, under different names, for years. What has changed is the consumer of this infrastructure. Previously, the consumers were BI tools, data scientists, and application developers. Now, the consumers are AI agents operating at machine speed with machine economics.

This shift in consumer changes the design requirements: latency tolerance narrows, context window limitations impose retrieval precision requirements, and cost attribution per inference becomes necessary for financial governance. The agentic layer is the next iteration of data architecture, built for machine consumers with different constraints, which is precisely why agentic transformation budgets should be evaluated as data architecture investments rather than AI experiments. The success criteria, the governance intensity, and the payback period all follow platform economics.

What We Learned Operating Under Private Equity-Style Economics

We have had the experience of operating technology functions in environments where private equity economics applied, where every initiative was scrutinized for its contribution to value creation, where carrying costs were reviewed quarterly, and where the question was never “is this interesting?” but “does this move the number?”

Very few technology expenditures successfully pass through that filter.

What does is technology that takes cost out of the business while maintaining or improving output, technology that unlocks capacity without proportional headcount growth, technology that reduces cycle times on revenue-generating processes. What does not pass includes capability-building programs with vague “future flexibility” rationales, data quality initiatives that cannot trace impact to a P&L line, and AI experiments that produce cool demos but not deployable workflows.

The PE operating model would typically look like this: 20% annual increases in technology spend, not decreases, combined with 40%+ reductions in operational costs enabled by that technology. Technology becomes a lever to extract operational gains that exceed the investment.

We raise this because the economic pressure that PE creates explicitly is now arriving implicitly for most enterprises. Competitive pressure from AI-native entrants, margin compression, and investor scrutiny of AI spending without demonstrated returns are creating PE-like conditions across industries. Data architecture investment either passes the test of demonstrable value creation, or it becomes a target for the next round of cost reduction.

The Virtue Cycle That Architecture Should Enable

Business departments spinning up agents should not be pursuing one-time improvements. The potential, unrealized in most organizations, is a self-improving virtue cycle where structured process data, unstructured documents, and workflow telemetry combine to continuously improve agent performance.

Consider what happens when an agent processes procurement requests. Each request generates structured data (vendor, amount, category, approval chain) alongside unstructured context (email threads, specifications, negotiation notes). If the architecture supports it, this combined signal can flow back into the semantic layer, improving retrieval for similar future requests, refining the agent’s understanding of which approvals require escalation, and identifying patterns that suggest process inefficiency or fraud risk. The agent that processed a thousand requests should be meaningfully better than the agent that processed ten, but only if the architecture captures the learning.

This requires semantic layers that can incorporate feedback loops, observability that captures agent behavior in ways that support learning, and governance that permits this feedback while maintaining controls. Most enterprise data architectures were designed for batch reporting, where data flowed in one direction, from operational systems to warehouses to dashboards to humans who made decisions. The agentic architecture requires bidirectional flow. Organizations that build for this will compound their advantage. Organizations that deploy agents against static architecture will achieve one-time productivity gains that plateau quickly.

Where Value Is Concentrating

The February 2026 “Software-mageddon” offered a natural experiment. While application-layer software lost approximately $2 trillion in market capitalization, the data infrastructure layer held. Databricks raised $5 billion at a $134 billion valuation, oversubscribed, during the selloff. Snowflake’s AI Services consumption is reported to be growing while traditional SaaS seats and their pricing are being questioned across the industry.

The signal is clear: investors who allocate capital professionally have concluded that the application layer is being commoditized by models, while the data layer is becoming more valuable.

Foundation models compress the domain logic that previously required custom software development. The barrier to replicating application functionality has collapsed. Value is migrating to two places: the model layer (captured by foundation model providers) and the data layer (captured by whoever owns proprietary, well-architected enterprise data).

And since you are most probably not building foundation models, your competitive position depends on the data layer. The ROI calculation for data architecture has shifted accordingly, from the efficiency gains of better BI to the option value of participating in AI value creation.

How Platform Costs Map to Financial Statements

Data platform leaders rarely possess fluency in how their investments appear on financial statements, and finance leaders reviewing data platform spend rarely understand the technical drivers of those costs. This mutual illegibility creates problems that compound over time.

Traditional on-premise data infrastructure was CAPEX: servers purchased, depreciated over useful life, creating assets on the balance sheet. Cloud infrastructure is predominantly OPEX: consumed as services, expensed immediately, hitting the P&L in the period incurred. This shift has made data platform costs more visible to CFOs on a quarterly basis, which sounds like improved governance but often creates perverse incentives, pressure to reduce cloud spend in the current quarter at the expense of architectural investments that would reduce total cost of ownership over a multi-year horizon.

Where costs land on the P&L determines scrutiny intensity. AI costs embedded in COGS receive intense board attention because they directly affect gross margin. AI costs in R&D receive different scrutiny: they are expected to be investment-like, with payoff in future periods. AI costs buried in G&A often receive the least strategic attention, treated as overhead rather than investment, which may be exactly wrong if those costs are enabling productivity gains across the organization.

The depreciation question is increasingly problematic. When you capitalize software development under ASC 350-40 (US GAAP) or IAS 38 (IFRS), you assert that the development creates an asset with multi-year useful life. But this seems to be less true for AI systems. E.g. AI systems would drift; models degrade as distributions shift and prompts require maintenance as underlying models update. We have seen organizations capitalize AI development on five-year schedules for systems that require substantial rework within eighteen months, creating a mismatch between accounting treatment and economic reality.

Data Platform Economics

When a CIO signs a three-year platform commitment, they are making a capital allocation decision that will shape cost trajectories, constrain architectural choices, and determine what the organization can afford to do with agents. Yet most platform selection processes evaluate features and benchmarks. Almost none model the financial behavior of the platform under the workload patterns that agentic deployment will create.

The two dominant platforms are not interchangeable products with different logos. They are different financial instruments. Snowflake’s separated compute model charges by warehouse runtime and rewards discipline and predictability. It punishes concurrency: agents issuing hundreds of small, high-frequency queries against the semantic layer will trigger scaling costs that BI-only projections just never really anticipated. Databricks charges through DBUs, a normalized processing measure, and rewards strong engineering discipline (that comes with a cost of its own, typically requiring more rigorous engineering capabilities). It punishes immaturity: unoptimized Spark jobs, poorly partitioned tables, and over-scanning retrieval pipelines burn DBUs without producing proportional value. We are just experiencing large Databricks projects migrated poorly that even run into performance issues, next to cost issues precisely for this reason. We have also seen organizations with comparable workloads where annual platform costs easily diverged by 100% from $300K to $600K purely based on the match between the platform’s financial model and the organization’s operational maturity.

In terms of Mathias Frey’s Formula, platform choice thus directly affects the architecture efficiency coefficient a, because misaligned economics raise the cost of every unit of capacity (i.e. independent of the vendor, subpar architecture induces subpar economics).

Similarly, the convergence around Apache Iceberg may be the most financially significant data infrastructure development since the shift to cloud, precisely for what it does to the reversal premium of platform decisions. I.e. this optionality is valuable on its own merit and not on technical gains alone. When data sits in an open format on object storage, the compute layer becomes interchangeable. A decision that once required an 18-month migration to reverse can now be reversed in weeks. Organizations adopting Iceberg are buying optionality: converting a high-reversal-premium commitment into something closer to a rolling option. Databricks’ moves with Lakebase and Snowflake’s deepening Iceberg integration both signal that the vendors recognize this shift in buyer power.

Serverless compute is marketed as the efficient default, and for bursty, unpredictable workloads it genuinely is. But we are finding that agentic workloads are not that bursty. Agents in production generate sustained, high-frequency load that seems to resemble application workloads more than it does analyst queries workloads.

More tests are needed for this, but observations and a rule of thumb, show that similar to application workloads, with anything over 60% sustained utilization, provisioned capacity costs 30% less than serverless for equivalent throughput.

Organizations that default to serverless during experimentation and fail to re-evaluate as workloads mature accumulate an overspend that compounds monthly.

This raises a governance question about commitment structures. Vendors offer one- and three-year discounts of 15-30%, but a multi-year commitment is a bet that workload patterns will remain stable, and agentic adoption will change them. The better approach we believe is to have structured optionality: commit on the predictable base, retain on-demand capacity for agentic workloads, and migrate into committed tiers as consumption patterns stabilize. The 10-15% premium over a full commitment can be seen as the price of “the right to adapt”. For Finance departments (and esp. Purchasing deparments) evaluating platform renewals, the question is not what is the deepest discount on compute possible, but what is the cost of being wrong about the consumption forecast in an environment where that forecast has never been less certain.

Architecture Patterns for Agent Readiness

Agent readiness can be seen as a stack with three distinct layers, each requiring different architectural investment. Layer one is retrieval: can an agent find the data it needs? This is where most organizations focus today, building vector stores, tuning RAG pipelines, optimizing embedding models. Layer two is semantics: can the agent understand what the data represents, what business logic applies, and what constraints govern its use? Layer three is action: can the agent execute a governed operation, with cost attribution, audit trails and guardrails? Most enterprise investment concentrates on layer one while layers two and three remain immature (but indeed also unfunded). This is why current agent deployments can retrieve information but cannot reliably act on it. We predict that this will also reverse the overall ROIs and efficiency expectations down, as enterprises find out that fully bringing these things into production requires also an additional investment in CAPEX, plus additional OPEX spend to keep them maintained well into production.

The architectural mistake compounding this imbalance is the generalist agent. Organizations default to building broad-purpose agents that attempt to handle diverse tasks across multiple domains. This feels efficient because it reduces the number of components to manage. It is, in practice, we believe the opposite. A generalist agent must carry the context for every domain it might encounter and it needs access to every data source, every schema, every set of business rules. This means larger context windows, more tokens consumed per inference, more retrieval surface to search, and weaker governance because permissions must be broad enough to accommodate the widest possible task range. The result is an agent that is expensive to run, difficult to govern, and mediocre at most of its tasks.

Specialized agents could invert this equation. For instance, a procurement agent needs access to vendor data, contract terms, and approval workflows, but it does not need access to HR records, marketing analytics, or financial planning models and so on. By constraining the agent’s scope, you constrain its context requirements, its retrieval surface, and its permission boundaries. Fewer tokens per inference, more precise retrieval and indeed tighter governance. The unit economics improve at every level of the stack here. In our example above, specialized agent processing procurement requests at a cost of $0.03 per transaction will outperform a generalist agent doing the same work at $0.12, precisely because of this architecture choice. Specialization also improves output quality, while in many cases permitting the use of smaller, less expensive models without sacrificing performance or cost.

This is where agent routing becomes an architectural requirement. A routing layer sits above the specialized agents and directs incoming requests to the agent best equipped to handle them, based on intent classification, domain mapping, and required capability. Routing is lightweight: it consumes minimal tokens to classify and forward. The expensive inference happens in the specialized agent, which operates within a constrained, well-governed context. The pattern mirrors how well-run organizations work. You do not send every question to the CEO. You route it to the person with the relevant expertise and authority. Agent routing is the organizational chart of an agentic architecture, and like a good org chart, it makes the system faster, cheaper, and more accountable.

Standards like the Model Context Protocol (MCP) reinforce this pattern by providing a standardized interface through which agents connect to data sources and tools. MCP means each specialized agent does not need custom integration code for every system it touches. It reduces the engineering overhead of specialization, which was historically the argument against it. We could assume that with MCP and similar protocols, the cost of building ten specialized agents approaches the cost of building two or three generalist ones, while the operational efficiency and governance properties remain superior. The architecture that supports this, a semantic layer designed for machine consumption, a routing layer for intent-based dispatch, and specialized agents with constrained scope, is what would enable organizations to deploy agents profitably.

The Edge and On-Premise Arbitrage

Cloud was the obvious choice when inference required the same GPU clusters used for training, but that coupling too is dissolving. Apple Silicon, optimized smaller models, and dedicated inference hardware mean that for predictable, high-volume inference workloads, the cost crossover where owning compute beats cloud consumption is arriving faster than most CFOs (and many CIOs) realize. An enterprise running specialized agents at sustained volume, the procurement agent project example from the previous section, processing thousands of daily transactions, can reach a breakeven against cloud inference pricing within 12-18 months of a modest hardware investment.

This connects directly to the specialized agent argument. Generalist agents with unpredictable workload patterns remain cloud-native by necessity: you cannot size hardware for demand you cannot forecast. Specialized agents with constrained scope and predictable throughput are the ideal candidates for on-premise or edge deployment. The architecture implication is a hybrid model: centralized data platform in the cloud for governance and complex analytics, distributed inference at the edge for workloads where cost, latency, or data residency requirements make cloud consumption the more expensive option. Organizations that architected for specialization will have the option while organizations that built generalist agents will not.

The Asymmetry of Architectural Consequences

Not all architectural choices carry equal weight. Some decisions are reversible: you can change a data model, swap an orchestration tool, refactor a pipeline. The cost of being wrong scales roughly with the effort invested, which is of course uncomfortable, but remains linear.

Other decisions are more structural: from data platform selection, to multi-year cloud commitment, data residency choices, core schema design. These decisions carry asymmetric consequences. The goal is to get them right, so that you have a foundation that enables years of capability building at decreasing marginal cost. But if you get them wrong, or pursue an inconsistent and duplicate data architecture, and you pay what we have come to call the Dual-Run Tax, i.e. the multiplicative cost of running old and new systems in parallel during migration.

The Dual-Run Tax is not simply 2x, but in our experience across various migration projects, it runs 2.2x to 2.5x, because parallel systems require data synchronization between environments (often the hardest engineering problem), staff capacity split across paradigms (reducing effectiveness in both), governance overhead for both systems, and extended timelines as teams context-switch between incompatible mental models.

This asymmetry also creates an obligation that most organizations neglect but would be well advised to embrace and that is: strategic decommissioning. Legacy components that remain in production impose carrying costs, maintenance, security patching, staff knowledge, governance overhead. In a PE-style efficiency regime, every system that does not contribute to current value creation is a candidate for retirement. We have found that organizations serious about agentic transformation budget decommissioning as explicitly as they budget new capability. The capacity freed by removing legacy load is often more valuable than the capacity added by new hires.

Before committing to a platform or architecture, the question Finance should ask is: what is the reversal premium? What will it cost to unwind this decision if circumstances change? Decisions with high reversal premiums, multi-year cloud commits, proprietary format adoption, deep platform dependencies, require different governance intensity than decisions that can be unwound at the cost of engineering time alone.

What This Means for the Data Architect’s Role

Data architecture is no longer a technical specialization that produces artifacts for other teams to consume. In our view it has become also an economic discipline that determines what an organization can and cannot achieve with AI, and at what cost.

Data architects who cannot speak the language of unit economics, TCO modeling, OPEX/CAPEX tradeoffs, and investment payback periods will find their influence shrinking even as the technical importance of their work grows. Data leaders are advised to study the CFO’s perspective and treat it as a constraint that improves architecture by forcing clarity on value creation and cost allocation; it is the fabled “fit-for-purpose”-ness that good data architecture requires.

We believe the organizations that manage to capture value in the agentic era the quickest will be those that treat data architecture as a first-class strategic investment, governed with the same rigor applied to M&A activity or major capital allocation decisions.

References

Sokolov, K.I. & Meir-Huber, M. (2026). Finance-Grade Data & AI Products: The VANE Loop Framework. Vane Loop Research. vaneloop.com/book

Frey, M. (2026). “Scaling Beyond Code: The Architecture of Leadership.” Vienna Engineering Management Meetup.

Inmon, W.H. & Linstedt, D. (2014). Data Architecture: A Primer for the Data Scientist. Morgan Kaufmann.

Taleb, N.N. (2012). Antifragile: Things That Gain from Disorder. Random House. [On asymmetric consequences and optionality]

Gartner (2025). “40% of Enterprise Applications Will Feature Task-Specific AI Agents by 2026.”

MuleSoft (2026). “2026 Connectivity Benchmark Report.” [On agent proliferation and governance gaps]

The Finance Layer: A PE-Inspired Operating Model for Data Architecture

Why Data Architecture Needs Unit Economics in the Agentic Era

By Karl Ivo Sokolov and Mario Meir-Huber

Abstract

Data architecture has always mattered but the nature of its consequences has changed. In the agentic era, architectural decisions carry asymmetric financial weight that most enterprises are poorly equipped to evaluate. We introduce the concept of the Finance Layer, a discipline that connects platform choices to P&L outcomes. Our argument is that the current “agentic transformation” is, at its core, a data architecture transformation with less familiar economics. The February 2026 SaaS-software selloff offered a preview: data infrastructure held value while application-layer software repriced dramatically. Organizations treating agentic AI as an application-layer initiative, bolting agents onto existing infrastructure, are investing heavily while remaining structurally unprepared to capture value. We explore the addressability problem that “clean data for AI” fails to solve, how architectural efficiency functions as a financial variable, and why platform decisions now carry asymmetric consequences that demand new governance approaches including strategic decommissioning.

The Addressability Problem

Many consultants have arrived with a familiar pitch: “You need clean data for AI.” And while the statement sounds reasonable it may be inadequate in ways that could cost organizations years of misdirected effort.

“Clean data” is a quality attribute that asymptotes quickly. You can spend two years improving data quality from 85% to 92%, and the marginal value of each percentage point diminishes as you approach it. The deeper problem is that “clean” remains undefined in the agentic context. Clean for what purpose? Clean according to whose schema? Clean data sitting in a warehouse that agents cannot programmatically address is clean but useless.

When we began working with enterprises preparing for agentic deployment, we found that the organizations struggling most were the ones that had invested heavily in data quality without investing in the infrastructure required for machines, rather than humans, to consume that data. The shift required is from cleanliness to addressability. Addressability means an agent can locate the data it needs, understand what the data represents through proper semantic modeling, act on the data within defined guardrails, be traced for what it did, and have its costs allocated to the appropriate business unit.

The “clean data for AI” framing solves yesterday’s problem by preparing data for human analysts who can interpret ambiguity, navigate incomplete metadata, and mentally translate between systems. Agents do this poorly and expensively, burning tokens on context reconstruction that a properly architected semantic layer would have made unnecessary.

The practical difference in investment is significant. A data quality initiative might cost €2M and take 18 months. A data architecture transformation for agent readiness, including semantic layer construction, API exposure with governance, and proper cost attribution mechanisms, might cost €15M and take three years. The first feels safer and shows progress in dashboards. The second is what actually unlocks the value that everyone is projecting in their AI business cases.

The Frey Formula: Architecture as a Financial Variable

At the Vienna Engineering Management Conference in February 2026, Mathias Frey, Chief Engineering Technology Officer of Erste Digital at Erste Group, Austria’s largest bank, presented a formulation that captures what most data leaders intuitively know but rarely express in terms that Finance departments can evaluate:

V ∼ D/p = C = logₐ(n)

The variables: D represents demand (the work the organization needs done), p is a product capability multiplier reflecting how effectively that demand is managed and filtered, C is capacity (what can actually be delivered), n is the number of people, and a is the architecture efficiency coefficient, the base of the logarithm.

The formula’s power lies in what it makes visible. Capacity scales logarithmically with headcount, and the steepness of that curve is determined by architecture. When a = 10, you need ten times the people to produce one additional unit of output. When a = 2, you only need twice the people. The gap between those scenarios represents the difference between an organization that scales and one that drowns in coordination costs.

Frey’s observation that matters most: AI impacts n (the people variable), but does not fundamentally change p (product capability) or a (architecture efficiency). You can deploy agents against poorly architected data, and the agents will fail faster, or worse, confidently produce wrong outputs at scale. The architecture coefficient remains the binding constraint.

As organizations compress headcount through AI augmentation, the architectural efficiency variable becomes more consequential, not less. In a world where n is being actively compressed, a becomes the primary lever for value creation. And a is determined by decisions made in data architecture, decisions that were often made years ago, for different reasons, with different constraints. Architecture belongs in investment discussions, not as a technical appendix but as a parameter that determines what productivity gains are achievable and at what cost.

What “Agentic” Actually Means for Data Architecture

Enterprise software vendors are pushing what they call the “agentic layer” into sales conversations with remarkable intensity, Microsoft with Copilot, Salesforce with Agentforce, SAP with Joule, and others. The marketing suggests revolution. The underlying architecture suggests evolution.

Strip away the branding, and the agentic layer is a combination of API enablement, semantic modeling, orchestration infrastructure, and governance planes. Data architects have been building these components, under different names, for years. What has changed is the consumer of this infrastructure. Previously, the consumers were BI tools, data scientists, and application developers. Now, the consumers are AI agents operating at machine speed with machine economics.

This shift in consumer changes the design requirements: latency tolerance narrows, context window limitations impose retrieval precision requirements, and cost attribution per inference becomes necessary for financial governance. The agentic layer is the next iteration of data architecture, built for machine consumers with different constraints, which is precisely why agentic transformation budgets should be evaluated as data architecture investments rather than AI experiments. The success criteria, the governance intensity, and the payback period all follow platform economics.

What We Learned Operating Under Private Equity-Style Economics

We have had the experience of operating technology functions in environments where private equity economics applied, where every initiative was scrutinized for its contribution to value creation, where carrying costs were reviewed quarterly, and where the question was never “is this interesting?” but “does this move the number?”

Very few technology expenditures successfully pass through that filter.

What does is technology that takes cost out of the business while maintaining or improving output, technology that unlocks capacity without proportional headcount growth, technology that reduces cycle times on revenue-generating processes. What does not pass includes capability-building programs with vague “future flexibility” rationales, data quality initiatives that cannot trace impact to a P&L line, and AI experiments that produce cool demos but not deployable workflows.

The PE operating model would typically look like this: 20% annual increases in technology spend, not decreases, combined with 40%+ reductions in operational costs enabled by that technology. Technology becomes a lever to extract operational gains that exceed the investment.

We raise this because the economic pressure that PE creates explicitly is now arriving implicitly for most enterprises. Competitive pressure from AI-native entrants, margin compression, and investor scrutiny of AI spending without demonstrated returns are creating PE-like conditions across industries. Data architecture investment either passes the test of demonstrable value creation, or it becomes a target for the next round of cost reduction.

The Virtue Cycle That Architecture Should Enable

Business departments spinning up agents should not be pursuing one-time improvements. The potential, unrealized in most organizations, is a self-improving virtue cycle where structured process data, unstructured documents, and workflow telemetry combine to continuously improve agent performance.

Consider what happens when an agent processes procurement requests. Each request generates structured data (vendor, amount, category, approval chain) alongside unstructured context (email threads, specifications, negotiation notes). If the architecture supports it, this combined signal can flow back into the semantic layer, improving retrieval for similar future requests, refining the agent’s understanding of which approvals require escalation, and identifying patterns that suggest process inefficiency or fraud risk. The agent that processed a thousand requests should be meaningfully better than the agent that processed ten, but only if the architecture captures the learning.

This requires semantic layers that can incorporate feedback loops, observability that captures agent behavior in ways that support learning, and governance that permits this feedback while maintaining controls. Most enterprise data architectures were designed for batch reporting, where data flowed in one direction, from operational systems to warehouses to dashboards to humans who made decisions. The agentic architecture requires bidirectional flow. Organizations that build for this will compound their advantage. Organizations that deploy agents against static architecture will achieve one-time productivity gains that plateau quickly.

Where Value Is Concentrating

The February 2026 “Software-mageddon” offered a natural experiment. While application-layer software lost approximately $2 trillion in market capitalization, the data infrastructure layer held. Databricks raised $5 billion at a $134 billion valuation, oversubscribed, during the selloff. Snowflake’s AI Services consumption is reported to be growing while traditional SaaS seats and their pricing are being questioned across the industry.

The signal is clear: investors who allocate capital professionally have concluded that the application layer is being commoditized by models, while the data layer is becoming more valuable.

Foundation models compress the domain logic that previously required custom software development. The barrier to replicating application functionality has collapsed. Value is migrating to two places: the model layer (captured by foundation model providers) and the data layer (captured by whoever owns proprietary, well-architected enterprise data).

And since you are most probably not building foundation models, your competitive position depends on the data layer. The ROI calculation for data architecture has shifted accordingly, from the efficiency gains of better BI to the option value of participating in AI value creation.

How Platform Costs Map to Financial Statements

Data platform leaders rarely possess fluency in how their investments appear on financial statements, and finance leaders reviewing data platform spend rarely understand the technical drivers of those costs. This mutual illegibility creates problems that compound over time.

Traditional on-premise data infrastructure was CAPEX: servers purchased, depreciated over useful life, creating assets on the balance sheet. Cloud infrastructure is predominantly OPEX: consumed as services, expensed immediately, hitting the P&L in the period incurred. This shift has made data platform costs more visible to CFOs on a quarterly basis, which sounds like improved governance but often creates perverse incentives, pressure to reduce cloud spend in the current quarter at the expense of architectural investments that would reduce total cost of ownership over a multi-year horizon.

Where costs land on the P&L determines scrutiny intensity. AI costs embedded in COGS receive intense board attention because they directly affect gross margin. AI costs in R&D receive different scrutiny: they are expected to be investment-like, with payoff in future periods. AI costs buried in G&A often receive the least strategic attention, treated as overhead rather than investment, which may be exactly wrong if those costs are enabling productivity gains across the organization.

The depreciation question is increasingly problematic. When you capitalize software development under ASC 350-40 (US GAAP) or IAS 38 (IFRS), you assert that the development creates an asset with multi-year useful life. But this seems to be less true for AI systems. E.g. AI systems would drift; models degrade as distributions shift and prompts require maintenance as underlying models update. We have seen organizations capitalize AI development on five-year schedules for systems that require substantial rework within eighteen months, creating a mismatch between accounting treatment and economic reality.

Data Platform Economics

When a CIO signs a three-year platform commitment, they are making a capital allocation decision that will shape cost trajectories, constrain architectural choices, and determine what the organization can afford to do with agents. Yet most platform selection processes evaluate features and benchmarks. Almost none model the financial behavior of the platform under the workload patterns that agentic deployment will create.

The two dominant platforms are not interchangeable products with different logos. They are different financial instruments. Snowflake’s separated compute model charges by warehouse runtime and rewards discipline and predictability. It punishes concurrency: agents issuing hundreds of small, high-frequency queries against the semantic layer will trigger scaling costs that BI-only projections just never really anticipated. Databricks charges through DBUs, a normalized processing measure, and rewards strong engineering discipline (that comes with a cost of its own, typically requiring more rigorous engineering capabilities). It punishes immaturity: unoptimized Spark jobs, poorly partitioned tables, and over-scanning retrieval pipelines burn DBUs without producing proportional value. We are just experiencing large Databricks projects migrated poorly that even run into performance issues, next to cost issues precisely for this reason. We have also seen organizations with comparable workloads where annual platform costs easily diverged by 100% from $300K to $600K purely based on the match between the platform’s financial model and the organization’s operational maturity.

In terms of Mathias Frey’s Formula, platform choice thus directly affects the architecture efficiency coefficient a, because misaligned economics raise the cost of every unit of capacity (i.e. independent of the vendor, subpar architecture induces subpar economics).

Similarly, the convergence around Apache Iceberg may be the most financially significant data infrastructure development since the shift to cloud, precisely for what it does to the reversal premium of platform decisions. I.e. this optionality is valuable on its own merit and not on technical gains alone. When data sits in an open format on object storage, the compute layer becomes interchangeable. A decision that once required an 18-month migration to reverse can now be reversed in weeks. Organizations adopting Iceberg are buying optionality: converting a high-reversal-premium commitment into something closer to a rolling option. Databricks’ moves with Lakebase and Snowflake’s deepening Iceberg integration both signal that the vendors recognize this shift in buyer power.

Serverless compute is marketed as the efficient default, and for bursty, unpredictable workloads it genuinely is. But we are finding that agentic workloads are not that bursty. Agents in production generate sustained, high-frequency load that seems to resemble application workloads more than it does analyst queries workloads.

More tests are needed for this, but observations and a rule of thumb, show that similar to application workloads, with anything over 60% sustained utilization, provisioned capacity costs 30% less than serverless for equivalent throughput.

Organizations that default to serverless during experimentation and fail to re-evaluate as workloads mature accumulate an overspend that compounds monthly.

This raises a governance question about commitment structures. Vendors offer one- and three-year discounts of 15-30%, but a multi-year commitment is a bet that workload patterns will remain stable, and agentic adoption will change them. The better approach we believe is to have structured optionality: commit on the predictable base, retain on-demand capacity for agentic workloads, and migrate into committed tiers as consumption patterns stabilize. The 10-15% premium over a full commitment can be seen as the price of “the right to adapt”. For Finance departments (and esp. Purchasing deparments) evaluating platform renewals, the question is not what is the deepest discount on compute possible, but what is the cost of being wrong about the consumption forecast in an environment where that forecast has never been less certain.

Architecture Patterns for Agent Readiness

Agent readiness can be seen as a stack with three distinct layers, each requiring different architectural investment. Layer one is retrieval: can an agent find the data it needs? This is where most organizations focus today, building vector stores, tuning RAG pipelines, optimizing embedding models. Layer two is semantics: can the agent understand what the data represents, what business logic applies, and what constraints govern its use? Layer three is action: can the agent execute a governed operation, with cost attribution, audit trails and guardrails? Most enterprise investment concentrates on layer one while layers two and three remain immature (but indeed also unfunded). This is why current agent deployments can retrieve information but cannot reliably act on it. We predict that this will also reverse the overall ROIs and efficiency expectations down, as enterprises find out that fully bringing these things into production requires also an additional investment in CAPEX, plus additional OPEX spend to keep them maintained well into production.

The architectural mistake compounding this imbalance is the generalist agent. Organizations default to building broad-purpose agents that attempt to handle diverse tasks across multiple domains. This feels efficient because it reduces the number of components to manage. It is, in practice, we believe the opposite. A generalist agent must carry the context for every domain it might encounter and it needs access to every data source, every schema, every set of business rules. This means larger context windows, more tokens consumed per inference, more retrieval surface to search, and weaker governance because permissions must be broad enough to accommodate the widest possible task range. The result is an agent that is expensive to run, difficult to govern, and mediocre at most of its tasks.

Specialized agents could invert this equation. For instance, a procurement agent needs access to vendor data, contract terms, and approval workflows, but it does not need access to HR records, marketing analytics, or financial planning models and so on. By constraining the agent’s scope, you constrain its context requirements, its retrieval surface, and its permission boundaries. Fewer tokens per inference, more precise retrieval and indeed tighter governance. The unit economics improve at every level of the stack here. In our example above, specialized agent processing procurement requests at a cost of $0.03 per transaction will outperform a generalist agent doing the same work at $0.12, precisely because of this architecture choice. Specialization also improves output quality, while in many cases permitting the use of smaller, less expensive models without sacrificing performance or cost.

This is where agent routing becomes an architectural requirement. A routing layer sits above the specialized agents and directs incoming requests to the agent best equipped to handle them, based on intent classification, domain mapping, and required capability. Routing is lightweight: it consumes minimal tokens to classify and forward. The expensive inference happens in the specialized agent, which operates within a constrained, well-governed context. The pattern mirrors how well-run organizations work. You do not send every question to the CEO. You route it to the person with the relevant expertise and authority. Agent routing is the organizational chart of an agentic architecture, and like a good org chart, it makes the system faster, cheaper, and more accountable.

Standards like the Model Context Protocol (MCP) reinforce this pattern by providing a standardized interface through which agents connect to data sources and tools. MCP means each specialized agent does not need custom integration code for every system it touches. It reduces the engineering overhead of specialization, which was historically the argument against it. We could assume that with MCP and similar protocols, the cost of building ten specialized agents approaches the cost of building two or three generalist ones, while the operational efficiency and governance properties remain superior. The architecture that supports this, a semantic layer designed for machine consumption, a routing layer for intent-based dispatch, and specialized agents with constrained scope, is what would enable organizations to deploy agents profitably.

The Edge and On-Premise Arbitrage

Cloud was the obvious choice when inference required the same GPU clusters used for training, but that coupling too is dissolving. Apple Silicon, optimized smaller models, and dedicated inference hardware mean that for predictable, high-volume inference workloads, the cost crossover where owning compute beats cloud consumption is arriving faster than most CFOs (and many CIOs) realize. An enterprise running specialized agents at sustained volume, the procurement agent project example from the previous section, processing thousands of daily transactions, can reach a breakeven against cloud inference pricing within 12-18 months of a modest hardware investment.

This connects directly to the specialized agent argument. Generalist agents with unpredictable workload patterns remain cloud-native by necessity: you cannot size hardware for demand you cannot forecast. Specialized agents with constrained scope and predictable throughput are the ideal candidates for on-premise or edge deployment. The architecture implication is a hybrid model: centralized data platform in the cloud for governance and complex analytics, distributed inference at the edge for workloads where cost, latency, or data residency requirements make cloud consumption the more expensive option. Organizations that architected for specialization will have the option while organizations that built generalist agents will not.

The Asymmetry of Architectural Consequences

Not all architectural choices carry equal weight. Some decisions are reversible: you can change a data model, swap an orchestration tool, refactor a pipeline. The cost of being wrong scales roughly with the effort invested, which is of course uncomfortable, but remains linear.

Other decisions are more structural: from data platform selection, to multi-year cloud commitment, data residency choices, core schema design. These decisions carry asymmetric consequences. The goal is to get them right, so that you have a foundation that enables years of capability building at decreasing marginal cost. But if you get them wrong, or pursue an inconsistent and duplicate data architecture, and you pay what we have come to call the Dual-Run Tax, i.e. the multiplicative cost of running old and new systems in parallel during migration.

The Dual-Run Tax is not simply 2x, but in our experience across various migration projects, it runs 2.2x to 2.5x, because parallel systems require data synchronization between environments (often the hardest engineering problem), staff capacity split across paradigms (reducing effectiveness in both), governance overhead for both systems, and extended timelines as teams context-switch between incompatible mental models.

This asymmetry also creates an obligation that most organizations neglect but would be well advised to embrace and that is: strategic decommissioning. Legacy components that remain in production impose carrying costs, maintenance, security patching, staff knowledge, governance overhead. In a PE-style efficiency regime, every system that does not contribute to current value creation is a candidate for retirement. We have found that organizations serious about agentic transformation budget decommissioning as explicitly as they budget new capability. The capacity freed by removing legacy load is often more valuable than the capacity added by new hires.

Before committing to a platform or architecture, the question Finance should ask is: what is the reversal premium? What will it cost to unwind this decision if circumstances change? Decisions with high reversal premiums, multi-year cloud commits, proprietary format adoption, deep platform dependencies, require different governance intensity than decisions that can be unwound at the cost of engineering time alone.

What This Means for the Data Architect’s Role

Data architecture is no longer a technical specialization that produces artifacts for other teams to consume. In our view it has become also an economic discipline that determines what an organization can and cannot achieve with AI, and at what cost.

Data architects who cannot speak the language of unit economics, TCO modeling, OPEX/CAPEX tradeoffs, and investment payback periods will find their influence shrinking even as the technical importance of their work grows. Data leaders are advised to study the CFO’s perspective and treat it as a constraint that improves architecture by forcing clarity on value creation and cost allocation; it is the fabled “fit-for-purpose”-ness that good data architecture requires.

We believe the organizations that manage to capture value in the agentic era the quickest will be those that treat data architecture as a first-class strategic investment, governed with the same rigor applied to M&A activity or major capital allocation decisions.

References

Sokolov, K.I. & Meir-Huber, M. (2026). Finance-Grade Data & AI Products: The VANE Loop Framework. Vane Loop Research. vaneloop.com/book

Frey, M. (2026). “Scaling Beyond Code: The Architecture of Leadership.” Vienna Engineering Management Meetup.

Inmon, W.H. & Linstedt, D. (2014). Data Architecture: A Primer for the Data Scientist. Morgan Kaufmann.

Taleb, N.N. (2012). Antifragile: Things That Gain from Disorder. Random House. [On asymmetric consequences and optionality]

Gartner (2025). “40% of Enterprise Applications Will Feature Task-Specific AI Agents by 2026.”

MuleSoft (2026). “2026 Connectivity Benchmark Report.” [On agent proliferation and governance gaps.

Kark Ivo Sokolov

Mario Meir-Huber

Karl Ivo Sokolov:

Karl Ivo Sokolov is Managing Director Data & AI at Specific-Group Austria, where he leads international teams across eight European countries. His work focuses on building robust data products and guiding enterprises through the modernization of complex data environments spanning both legacy and modern platforms. Beyond his role at Specific-Group, Karl Ivo also serves on the Global Board of Directors of the U.S. Institute of Management Accountants (IMA), contributing to the advancement of data-driven decision-making in finance and management accounting worldwide.

Mario Meir-Huber:

Mario Meir-Huber helps organizations turn scattered data initiatives into governed, business-driven Data Products. A former Head of Data and ex-Microsoftie, he has built Data Products across European companies. He created the GAP (Governance–Architecture–People) model and the DRIVE lifecycle framework and applies them in real engagements. Mario lectures at WU Vienna and TU Wien, teaches on LinkedIn Learning and speaks at events like WeAreDevelopers, Data Modelling Zone, London Tech Week, Data Science Conference and GITEX. He co-authored Handbook Data Science & AI and is writing The Data Products Series.

Karl Ivo and Mario Meir-Huber are co-authors of “Finance-Grade Data & AI Products: The VANE Loop Framework”, Jan 2026, available at Amazon and vaneloop.com/book

By

Derek Strauss (Gavroshe USA, Inc.) and Alan Rodriguez (Data Freedom Foundation)

Executive Summary

In the modern business landscape, organizations are inundated with vast amounts of data generated from customer interactions, operational processes, and external partnerships. Despite this abundance, many struggle to convert this data into actionable insights and tangible financial gains. The challenges they face include data overload—the exponential growth of data makes it difficult to manage and extract meaningful information. Siloed systems trap data in disparate systems across departments, hindering a unified view. Unrealized value arises when investments in data technologies don’t always translate into financial returns due to human factors, technical complexities, and mounting ‘data debt’. Governance risks add layers of complexity and risk with compliance to evolving data privacy and security regulations.

To address these challenges, AI-driven Knowledge Governance emerges as a transformative framework. It unites emerging AI governance and traditional data governance to transform raw data into actionable knowledge, driving significant financial returns. Central to this is the Data Liquidity Protocol (DLP), which revolutionizes data governance by embedding cryptographic proofs directly into data products, ensuring they are verifiable, secure, compliant, and monetization-ready. DLP binds five essential cryptographic proofs to every data product: Proof of Integrity for provenance and quality, Proof of Ownership for enforceable control, Proof of Security for layered protection, Proof of Compliance for privacy-preserving adherence to regulations, and Proof of Value for measuring ROI from every data use case.

By combining Knowledge Governance with DLP, organizations can automate compliance, manage risk, enhance data quality, unleash innovation, and unlock new revenue streams. This synergy allows transformation of raw data into valuable, marketable assets without exposing sensitive information or violating regulatory standards.

The process unfolds in three key phases:

• Contextualization: Utilize AI tools to refine and structure raw data, adding context and meaning. This involves AI-driven modeling to organize data into meaningful categories, data mapping to link data through taxonomies and ontologies with every siloed instance of every critical data element, and continuous refinement to ensure data quality and relevance through profiling and lineage tracking.

• Productization: Convert structured knowledge into reusable and scalable data products. Key steps include data cleansing to finalize quality, accuracy, and readiness; product design leveraging AI to create user-friendly and valuable data products; connectivity to ensure interoperability across systems and platforms; and deployment to integrate data products into workflows for real-time use.

• Monetization: Deploy data products to generate financial value through internal monetization, enhancing operational efficiency and reducing costs by utilizing data products within the organization, and external monetization, generating new revenue streams by selling or licensing data products to customers, partners, or through data marketplaces.

Implementing Knowledge Governance provides a significant competitive advantage: accelerated time to value through AI automation, new revenue streams by unlocking underutilized data, and improved decision-making with real-time, predictive insights. Key benefits include financial asset creation by systematically transforming data into monetizable knowledge assets, enhanced governance and compliance with real-time adherence to regulations, faster insights for immediate decision-making, and measurable ROI to demonstrate financial returns on data investments.

Combined with AI tools like knowledge graphs, agentic systems, and large language models, this enables proactive governance, automated compliance, enhanced trust, and new revenue streams—positioning data as a core profit center rather than a cost.

Transforming Data Governance with AI

Traditional data governance often struggles with inefficient practices due to manual processes, siloed data systems, and reactive approaches to risk and compliance. These limitations hinder the ability to derive timely insights and fully capitalize on data assets. Siloed systems scatter data across departments and platforms, leading to fragmented insights and hampering a unified organizational view. Manual processes rely on human intervention, making governance slow, error-prone, and unable to keep pace with real-time data flows. Delayed insights from time-consuming data integration and analysis result in missed opportunities and slow decision-making. Limited scalability makes it difficult to expand governance practices across growing data volumes and diverse data types. A reactive approach addresses governance issues after they occur rather than preventing them proactively, increasing risks.AI changes this by enabling automated, proactive, and scalable practices. Standardized metadata becomes the foundation for consistency, interoperability, and future-proofing across complex ecosystems. It supports semantic integration via ontologies and knowledge graphs and unified handling of structured and unstructured data. By adopting industry-standard metadata frameworks, governance ensures consistency across systems with uniform data descriptions, reducing ambiguity and enabling seamless integration. Interoperability with external ecosystems aligns with global protocols, facilitating collaboration with partners and compliance with regulatory frameworks. Scalability for future growth supports the integration of new data types, AI models, and governance requirements, ensuring long-term adaptability.

As data architectures evolve, the roles of data models and ontologies are foundational. Data models serve as internal-facing blueprints—much like a compass and map—for structuring and storing repetitive, key-dominated structured data. They define entities, attributes, relationships, and keys (e.g., through entity-relationship diagrams, data item sets, and DDL), providing vertical organization tailored to the enterprise’s operational needs.

In contrast, ontologies address the vast majority of corporate data—unstructured text, which often comprises 95% or more and remains underutilized due to its free-form, non-repetitive nature. An ontology is a formal specification of a shared conceptualization: the ultimate controlled vocabulary that captures concepts, entities, properties, relationships, and contexts within a domain. Built from one or more taxonomies (focused vocabularies pairing words with contextual meanings), ontologies provide horizontal, dynamic structure. They resolve key challenges in text, such as polysemy (e.g., “fire” as combustion, weapon discharge, or termination), negation (e.g., “did not have cancer”), and multilingual variations, by deriving context from surrounding content.

While data models are static and internal (enterprise-defined), ontologies are external-facing, drawing from common language usage to enable semantic disambiguation and believability. In many ways, the ontology is to text what the data model is to structured data—both unify chaos and provide essential governance. Without ontologies, text lacks context and becomes “pointless” or even “dangerous”; with them, unstructured data gains structure, lineage, and usability, integrating seamlessly with structured sources.

The Cognitive Engine revolutionizes data governance by leveraging artificial intelligence to automate, scale, and enhance governance processes. It integrates advanced AI technologies to create a dynamic and responsive framework. For automated and proactive governance, it offers real-time enforcement where AI continuously monitors data usage, dynamically adjusting data contracts, and ensures compliance without manual oversight. Policy automation embeds data contracts within data products shared with AI solutions, automating tasks like data quality checks, access controls, and compliance enforcement. Issue detection and resolution identifies potential governance issues proactively, such as data anomalies or security risks, and initiates corrective actions.

For scalable governance across complex environments, it provides a unified governance framework that manages both structured and unstructured data uniformly, eliminating blind spots and ensuring consistent policy application. Global scalability adapts to various regulatory environments, enabling organizations to scale governance practices globally without added complexity. Dynamic adaptation adjusts governance strategies in real-time based on data flow changes, new data sources, or evolving regulations.

Enhanced data lineage and provenance allow every data product to meticulously record its origin, movements, transformations, and interactions, providing multiparty observability and traceability. Immutable records maintain tamper-proof logs of multiparty data activities, essential for audits and compliance verification. Accountability ensures stakeholders know how data has been used and by whom.

Improved compliance and regulatory adherence include automated compliance monitoring to ensure adherence to regulations like GDPR, HIPAA, and CCPA by enforcing relevant policies in real-time. Consent management dynamically manages user consents and preferences, ensuring data usage aligns with individual rights.

Advanced AI technologies power this shift:

• AI co-pilots assist human decision-making.

• Large language models process unstructured content, grounded by ontologies to move beyond probabilistic patterns toward rule-based reasoning and inference.

• Knowledge graphs enable reasoning over interconnected relationships.

• AI agents autonomously handle monitoring, anomaly detection, and policy adaptation.

Metadata is especially critical in agentic AI systems, providing semantic context, dynamic rules, and interoperability for autonomous governance tasks. Ontologies enhance this by incorporating formal logic, property inheritance, and constraints, preventing nonsensical outputs and avoiding siloed semantic issues.

Benefits include efficiency gains by automating manual tasks, reducing time and resources for governance; faster insights with real-time data readiness for decision-making; risk reduction through proactive issue detection and resolution; cost savings from streamlined operations and reduced compliance penalties; and enhanced stakeholder trust with transparent, accountable data practices—critical for scaling operations and maintaining competitive edge.

Contextualization: From Raw Data to Actionable Knowledge

Raw data holds limited value until refined. Contextualization adds relevance, structure, and meaning, preparing it for analysis and decision-making. The role of contextualization is to transform raw data into knowledge by adding layers of context, making it actionable for business insights. It bridges the gap between data collection and utilization, enabling organizations to derive value from their data assets.

Active metadata management drives continuous discovery, behavioral insights, automated enrichment, and semantic integration—bridging silos and creating a living knowledge framework. It involves active metadata management in contextualization for ongoing data discovery and profiling, behavioral metadata to capture data usage patterns, automated metadata enrichment with AI-generated tags, and semantic metadata for linking concepts through ontologies.

The Data Liquidity Protocol embeds cryptographic integrity early, ensuring provenance, compliance, and trustworthiness from the start. In contextualization, DLP applies proof of integrity to verify data provenance and quality during refinement, proof of compliance to embed privacy proofs from the outset, and proof of value to track early ROI indicators.AI-powered contextualization with the Cognitive Engine automates profiling and classification, pattern recognition and anomaly detection, taxonomy and ontology building, quality monitoring and remediation, and lineage tracking. This ensures data is continuously refined and enriched.

In this phase, data models and ontologies play complementary, essential roles. For structured elements, data models enforce precise relationships and storage schemas. For unstructured sources—like emails, documents, reports, or sensor narratives—ontologies (rooted in taxonomies) provide the critical semantic layer. Processes akin to Textual ETL scan raw text against ontology taxonomies to extract meaningful chunks (e.g., concepts or word-level elements with context), disambiguate terms, and place them in structured, horizontal databases—complete with lineage for traceability.

For example, in a medical context, an ontology might map “envarsus” to “medication” or “blood pressure” to “vital sign,” enabling accurate extraction even from free-form notes. Iterative refinement allows taxonomy adjustments (adding/deleting contexts) and quick reruns, turning vast unstructured volumes into analyzable insights rapidly. This not only resolves ambiguities but supports inference, governance, and integration with structured data.

The strategic importance for data leaders lies in unlocking data value by making raw data actionable, supporting compliance through built-in proofs, fostering innovation with enriched knowledge, and laying the foundation for productization. It serves as the foundation for productization by creating structured knowledge ready for packaging into data products.

By combining data models for structured precision and ontologies for unstructured context, contextualization unlocks holistic knowledge, supports compliance, fosters innovation, and lays the groundwork for reusable products. Examples include fraud detection in finance, where contextualized transaction data reveals patterns; personalized diagnostics in healthcare, where patient records are enriched with ontologies for better insights; and predictive maintenance in manufacturing, where sensor data is structured for proactive alerts.

Productization: Turning Knowledge into Reusable Data Products

Productization bridges knowledge to tangible assets—reusable, scalable data products for internal workflows or external offerings. The role of productization is to package contextualized knowledge into standardized, reusable formats that can be easily integrated and scaled, turning knowledge into operational tools.

Active metadata management in productization maintains catalogs for discoverability, enforces policies by design, enables user-centric discovery, and tracks lifecycle and value realization. It involves metadata catalogs for product discovery, policy-embedded metadata for governance, user-centric metadata for accessibility, and value-tracking metadata for ROI measurement.

The Data Liquidity Protocol in productization certifies cleansed data with proofs of integrity, ownership, security, compliance, and value; supports interoperable design; and enables versioned deployment. It applies proof of ownership to enforce control, proof of security for protection, and proof of value for monetization readiness.AI-powered productization with the Cognitive Engine streamlines cleansing for quality assurance, product design for customizable and user-friendly formats, connectivity via open standards and APIs, and real-time updates for dynamic products.

The strategic importance for data leaders includes faster time-to-market with automated processes, consistent quality through AI oversight, flexibility for adapting products, and enhanced value for both internal and external use. This phase leads to the path to financial advantage by preparing products for monetization.

Outcomes include faster time-to-market, consistent quality, flexibility, and enhanced value—enabling operational dashboards, risk models, and marketplace offerings. This maximizes utilization, drives innovation, diversifies revenue, and supports agility.

Monetization: Generating Financial Value from Data Products

Monetization realizes bottom-line impact through internal optimization and external revenue. The role of monetization is to convert data products into financial assets, creating new revenue streams and cost savings.

Active metadata management in monetization enables usage-based valuation, regulatory enforcement, marketplace readiness, and ROI analytics. It involves valuation metadata for pricing, compliance metadata for enforcement, marketplace metadata for discoverability, and analytics metadata for performance tracking.

The Data Liquidity Protocol in monetization supports secure, trustless transactions with dynamic pricing, compliant marketplaces, and subscription models—without exposing sensitive data. It applies all five proofs to ensure trusted monetization.

Metadata-driven trust in data marketplaces uses metadata to build confidence through verifiable proofs, enabling secure exchanges.AI-powered monetization with the Cognitive Engine facilitates internal gains like process automation, risk mitigation, and decision enhancement; and external models like marketplace sales, subscriptions, and licensing.

Key metrics for measuring ROI and financial impact include revenue from data products, cost savings from efficiency, time-to-insight reductions, customer retention improvements, market expansion opportunities, and metadata quality/completeness scores.

The strategic importance for data leaders is in turning data into a profit center, maximizing ROI, and driving growth through monetized assets.

Real-world examples include subscription-based risk models in finance, personalized recommendations in retail, predictive analytics licensing in healthcare, and consumer insights marketplaces.

Case Studies: Metadata-Driven Governance in Action

To illustrate the practical application of Knowledge Governance, consider these real-world examples across industries, demonstrating how the framework delivers trust, scalability, and financial returns in complex ecosystems.

Financial Services: Real-Time Risk Management with Agentic AI

In a large financial institution, traditional governance struggled with siloed transaction data and manual compliance checks, leading to delayed risk assessments and high operational costs. Implementing Knowledge Governance, the organization used the Cognitive Engine for contextualization, applying ontologies to unstructured fraud reports and data models to structured transactions. Active metadata tracked lineage, while DLP embedded proofs of integrity and compliance.

During productization, AI cleansed and packaged data into reusable risk models, with metadata catalogs enabling discovery. Monetization involved internal use for real-time fraud detection, reducing losses by 25%, and external licensing to partners, generating $8M in annual revenue. Agentic AI agents autonomously monitored anomalies, accelerating assessments by 40% and cutting costs by 30%. This not only enhanced compliance with regulations like CCPA but also built trust through immutable provenance, turning data debt into a competitive asset.

Retail: Metadata-Enhanced Consumer Insights Marketplace

A global retail chain faced data overload from customer interactions across online and in-store channels, with siloed systems hindering unified views. Knowledge Governance transformed this through contextualization, using AI to map customer data via taxonomies and ontologies, enriching profiles with behavioral metadata. DLP ensured proof of ownership and security, preventing data leaks.

Productization created scalable insight products, like personalized recommendation engines, with AI-designed interfaces and API connectivity. Metadata enforced policies and tracked value. For monetization, the chain launched an internal dashboard for inventory optimization, saving $5M in costs, and an external marketplace for anonymized insights, generating $10M annually. Engagement boosted 25% via metadata-driven discoverability, fostering collaborations with suppliers and demonstrating measurable ROI through key metrics like customer retention.

Healthcare: Metadata-Driven Patient Governance and Analytics Licensing

In a healthcare network, compliance risks from regulations like HIPAA and fragmented patient records limited innovation. Knowledge Governance addressed this with contextualization, leveraging ontologies for unstructured medical notes (e.g., disambiguating “fire” in contexts) and data models for structured vital signs. The Cognitive Engine automated lineage tracking, while DLP provided proofs of compliance and value.

Productization packaged knowledge into patient governance tools, with AI ensuring quality and interoperability. Metadata supported dynamic contracts. Monetization included internal use for personalized diagnostics, improving outcomes by 15% and cutting violations by 50%, plus licensing anonymized analytics to research firms for $5M in revenue. This enabled secure data sharing in multiparty ecosystems, enhancing trust and positioning the network as a leader in data-driven care.

Supply Chain: Optimizing Logistics with Decentralized Insights

In a multinational supply chain network, stakeholders struggled with data silos and lack of trust in shared information. Knowledge Governance facilitated contextualization by AI mapping logistics data through ontologies for unstructured reports and data models for inventory. DLP embedded proofs for secure sharing of insights, not raw data.

Productization created reusable optimization products, with metadata for interoperability. Monetization involved internal efficiency gains, reducing costs by 20%, and external licensing via decentralized marketplaces, adding $7M in revenue. This decentralized approach drove collaboration, innovation, and agility, compounding data value in the knowledge economy.

These cases highlight how Knowledge Governance turns challenges into opportunities, delivering financial advantage through AI, metadata, and DLP.

The Imperative of Governance, Security, and Trust

Robust governance and security are non-negotiable amid breaches, privacy risks, and regulations like GDPR, HIPAA, and CCPA. The imperative stems from the need to protect data integrity while enabling innovation. Knowledge Governance builds trust through proactive controls, immutable provenance via blockchain-integrated metadata, and automated compliance, ensuring data remains compliant, secure, and trustworthy throughout its lifecycle.

Embracing this for future growth distinguishes leaders from laggards, driving innovation, operational excellence, and sustainable financial growth. It offers a comprehensive framework that transforms raw data into strategic knowledge assets and significant financial advantage.

Future Outlook

As the data-driven economy evolves, metadata will play an increasingly pivotal role in Knowledge Governance. Emerging technologies will elevate metadata from a supportive layer to a transformative force, enabling decentralized, secure, and interoperable data ecosystems.

Decentralized metadata frameworks, enabled by Web3, will allow distributed repositories for global interoperability, aligning with DLP. Blockchain-based provenance tracking offers tamper-proof records, with smart contracts automating policies for real-time enforcement.AI-driven metadata evolution, via generative AI and LLMs, will autonomously enrich metadata, capturing nuanced relationships for enhanced knowledge graphs and agentic systems, accelerating contextualization and productization.

Metadata as a universal language will enable seamless integration across platforms, industries, and jurisdictions, with standardized protocols ensuring data products remain accessible, compliant, and valuable.

The Data Liquidity Protocol will redefine governance and monetization, shifting from centralized control to decentralized ecosystems. DLP’s impact includes decentralized data marketplaces for trading verifiable insights, fostering trust and new business models; AI governance by design, ensuring models use trusted data for transparency; and data as a service, empowering owners to license on their terms with enforced rights.

For example, in supply chains, DLP allows sharing insights while maintaining control, enhancing collaboration and efficiency. Organizations adopting DLP will transition from defending data to compounding its value, preparing for a data-driven future where data drives strategic advantage and growth.

Call to Action

Senior data leaders must champion Knowledge Governance to turn data into a profit center, maximize ROI, ensure compliance, and drive innovation. This framework integrates AI, metadata, and emerging protocols to transform raw data into monetizable assets—positioning organizations for a data-driven future.

As data continues to be a pivotal driver of business success, the time to act is now. By adopting Knowledge Governance, leaders can capitalize on financial opportunities, safeguard assets with automated governance, and demonstrate clear returns. It represents a paradigm shift, moving beyond traditional management to a holistic approach integrating AI, governance, and monetization. For adopters, the potential is clear: a future where data drives strategic, operational, and financial excellence, unlocking its full spectrum as a dynamic engine of growth and profitability.

For a deeper dive into this topic and related material, check out our 4 whitepapers:

· The Knowledge Governance White Paper defines the strategic imperative: turning data into knowledge and knowledge into financial advantage.

· The Data Liquidity Protocol (DLP) White Paper details the technical foundation: how governed Data products with standardized security and integrity envelopes provide cryptographic proof of ownership, integrity, provenance, security, compliance, usage, consent, and value.

· The Digital Rule of Law (DRoL) White Paper outlines the legal foundation: enforceable digital property rights that bind every data product and data transaction to its rightful owner.

· The Data Profit Center (DPC) White Paper brings these frameworks together in practice. It translates trust into measurable return. The DPC as the first operational step in the Knowledge Governance framework.

About the Contributors
Derek Strauss (Gavroshe USA, Inc.) and Alan Rodriguez (Data Freedom Foundation) collaborate on frameworks uniting AI and traditional governance. The Data Liquidity Protocol, stewarded neutrally by the Foundation, provides cryptographic foundations for secure, compliant data sharing and monetization.

Derek Strauss

Founder and Chairman of Gavroshe, a consulting company of 30+ years, which specializes in planning and implementation of multi-year Enterprise Transformation Programs, utilizing AI, and Data & Analytics.

Former Chief Data Officer at TD Ameritrade for approximately 5 years; responsible for building the Office of the CDO from scratch; this entailed establishing enterprise capabilities in Advanced Analytics and Data Science, Data Governance, Data Architecture and Enterprise Data Asset Development.

Alan Rodriguez

Founder of Data Freedom Foundation, an open-source, non-profit standards body creating revolutionary Consent and Privacy Technologies.

Invented and coded first-generation payment platforms. Created first-generation global supply chain platforms. Invented and created first-generation marketing, preference centers, and customer data management platforms. Invented the Data Liquidity Protocol to secure and control the uses of our data by others after being shared.

By W H Inmon

Jessica Talisman

“The answer is 7”

What does this statement tell you? It says that something has been counted and there were 7 of them. But it does not tell you what there are seven of –

Seas of the ocean

Days of the week

Dwarfs that attended Snow White

Dollars

And so forth.

Strictly speaking – taken by itself – the statement is almost meaningless. In fact, ANY number without context is practically meaningless.

In order to convey meaning, a number or any other piece of data has to have context attached to or otherwise associated with it.

STRUCTURED CONTEXT

Structured data always has context attached to it. In structured data there are data bases, keys, indexes attributes, and so forth that allow context to be attached to data. And these structured facilities are determined before the first data flows into it. So finding the context of structured data is so simple as to be taken for granted. It is simply always there.

Consider the simple structured table shown below.

In this simple structured table, it is seen that there are columns that contain information about –

Name

Sex

Age

State

Education

Marital status

Number of children

In every column of data, context defines the type of data that resides in the column. It would not be normal for doghouse to be found in the education column or mayonnaise to be found in the column for sex or age. The column name is context that defines the content of the column.

The context of structured data is determined at the moment structured data is formed. And the format and context is rigidly enforced. As such, context of structured data is straightforward and relatively simple.

Context in the world of unstructured data – text – is very, very different from context in the structured world.

UNSTRUCTURED CONTEXT

Whereas context in the structured environment is open, obvious and well defined, context of text is anything but open and obvious. There is a need for context of text for both words and acronyms. In unstructured context, the context of the word or acronym is external to the text, not predefined and is internal to the text itself as is the case with structured data. In other words, the analyst needing to determine the context of text or acronyms must ferret out – infer - the context using several techniques.

There are many ways that context of text can be determined. One method for determining the context of text is through what can be called immediate adjacency.

IMMEDIATE ADJACENCY

Consider the phrase – Dallas Cowboys. When a person hears the term, they think of a once great professional football team in Texas. But if the word Dallas is found on one page in isolation, the person thinks of a city in north Texas and if the world cowboy is found three pages later the person thinks of a man on a horse with a hat, a rope, and a pistol on his hip. It is only when the words are immediately adjacent to each other that the term takes on the meaning of a professional football team.

Immediate adjacency then has lent a new meaning altogether to words that mean something else independently placed. In many ways immediate adjacency of words is the simplest form of context inference.

SENTENTIAL CONTEXT

Another form of context of text is context that exists in the same sentence but is not immediately adjacent. As a simple example of sentential context, consider the simple sentence shown below -

When a person reads the sentence, it is understood that the tiger being referred to is an animal, not Tiger Woods. Words occurring in the same sentence is a very powerful form of inferring context.

REFERENTIAL CONTEXT

But there is context of text that can be inferred that lies beyond words appearing in the same sentence. Consider a broader method by which context is inferred. Consider the following sentence.

In this sentence, the word fire is inferred to mean the pulling of the trigger of a gun. This is because gun is mentioned in the same group of words, but not in the sentence that the word fire appears in. In this case context is inferred. The physical juxtaposition of the words outside of a sentence can be used to infer context as well.

The challenge is selecting which words infer context. It is entirely possible that there may be conflicting types of context that may have been inferred by looking over a large amount of words.

The further context moves away from the word being examined, the greater the chance that context will not be inferred properly.

GENERAL CONTEXT

But there is an even broader means of inferring context. An entire document may have context. For example, consider the IRS tax code. All text found in the IRS tax code will have some relationship – either direct or indirect – with federal taxation.

And as with other forms of context, the broader the contextual reference from the word being examined, the greater the chance that the word will be interpreted incorrectly.

CONTEXT RESOLUTION

One approach to context inference is to apply the rules used for inference very time the word appears. This approach is very costly in terms of the resources required for processing. Another approach is to determine the contextual inference the first time the word appears and to apply that context to all further occurrences of the word in a document.

While not perfect, this approach is efficient and works well for most bodies of text.

Using the implied inference for all subsequent appearances of a word can be called context resolution.

When the text in a document is being parsed and meaning is attached to the words being processed, how is the context of the word resolved?

The simplest resolution occurs when there is immediate context. Although it is possible that a mistaken resolution can occur, immediate adjacency of words has a very high degree of proper and accurate resolution.

The second way words can have the inference of their context resolved is to simply say that all occurrences of a word have the same context. In many cases this is true. But it is not universally true.

For example, suppose a document has the word plant that is found in the document. In almost every case the word plant refers to some sort of botanical structure – a tree, the grass, a shrub, etc. However, in one place in the document the word plant refers to a factory location where manufacturing is being done.

In this case the context of plant has been inferred and resolved incorrectly.

A third way a resolution of text can be done is for each occurrence of a word to have its own contextual resolution. The context of each word is enumerated.

A simple example might be –

Water – water the lawn

Water – ocean

Water – have a sip

Water – the baby is coming any minute now

And so forth

RESOLUTION TRADEOFFS

The challenge with the enumerated approach is that it is very expensive to do. For a large document it is prohibitively expensive to consider.

CONTEXT RIGIDITY

The context of structured data is highly rigid. That is the nature of structured data.

But the context of unstructured data – text – is highly variable. The context can be anything. Typical forms of context for unstructured data include –

Time – when was the word said or what time does the word refer to

Location – where was the word said or where does the word refer to

Cost – how expensive was the word being analyzed

Speaker – who spoke the word that is being analyzed

INTONED CONTEXT

But the complexity of context management in the world of text goes further than the appearance of words found together. Intonation of words also conveys meaning.

Take the simple expression – I love you.

If the emphasis is on – I – the implication is that it is me who loves you, not John, Terry, or Martha. If the emphasis is on – love – the implication is that my feelings for you are love, not like, cherish or detest. If the emphasis is on – you – the implication is on that fact that it is you that I love, not Mary, Susie, or Claudia.

Even though the words are exactly the same, the intonation and speed with which the words are spoken have a great deal of influence on how the words are understood.

CONTEXT COMPLEXITY

And even in the best of circumstances there can be misinterpretations. For example, how would you interpret the meaning of “ha” in the notes of a cardiologist?

In almost every case the cardiologist is referring to a heart attack when he/she uses the expression. But there is absolutely no reason why the cardiologist might be referring to a headache, not a heart attack.

CONTEXT DEPENDS ON CONTEXT

Context complexity reveals a spectrum of proximity, where one type of context cannot exist without context dependencies. Each layer of context requires its own contextual scaffolding to function. Immediate adjacency means nothing without knowledge of the language. Sentential context collapses without shared semantic conventions. Referential context demands a model of the world. General context presupposes domain knowledge accumulated over years of exposure. Context, in other words, does not interpret itself. Context requires context.

This is where the engineering of meaning is hard. The challenge is less about identifying context and demands knowing which context governs in any particular instance. A cardiologist’s “ha” is ambiguous at multiple levels—there’s the abbreviation presenting as an acronym, the clinical register in which it is utilized, the documentation conventions of the institution and the habits of medical practitioners using ha. Resolving the right meaning requires traversing all of these layers in the correct order. Collapse any single layer, and the wrong meaning propagates onwards.

INSTITUTIONAL CONTEXT

Among the most consequential and least visible forms of context is institutional context — the rules, roles, conventions, and assumptions that govern how language is used within an organization. A term that means something inside one institution can be meaningless or misleading outside it. “Admitted” means something specific in a hospital. “Closed” means something specific in a law firm. “Released” means something specific in a prison. These words carry meaning as stipulated by the organizational framework in which they appear. Institutional context is the water that specialized language swims in. It is almost always invisible to anyone who has not been inside the institution — and almost always unrepresented in the data.

PROVENANCE CONTEXT

Context is also shaped by who is speaking, under what conditions and with what authority. This is provenance context — the record of origin that determines how an utterance should be weighted and interpreted. The same clinical notation carries different interpretive force depending on whether it was written by an attending physician, a resident, nurse or a medical student. The same financial forecast means something unique depending on whether it comes from an internal analyst or an external auditor. Provenance changes everything about how the words are understood. In the absence of provenance, interpretation defaults to assumptions that may be entirely wrong or misinterpreted.

The structured world handles interpretive ambiguity through enforced schemas. Column names constrain values while data types constrain columns and relational constraints control table logic. Structure is, at its core, pre-committed context. Database schemas work because the interpretive framework is declared before the data arrives. The brittleness of structured data — its inability to absorb the unexpected — is the price paid for exacting contextual clarity.

Unstructured text pays the opposite price. Nothing is pre-committed. Context must be inferred, assembled, and continuously re-evaluated. And so strategies for resolving entities described above require immediate inferences, uniform assumptions and full enumerations. But the strategies for handling unstructured text relies upon how much contextual infrastructure you are willing to build and maintain. For example, an enumerated approach is expensive because it treats each word as its own contextual problem. The expense becomes the true cost of meaning.

Data management cannot make context tractable because it only handles structure, storage, and retrieval. Data management on its face does not handle meaning. Meaning is the province of knowledge management and information management — disciplines that are fundamentally social, not technical. Knowledge does not exist in databases, as knowledge involves people and people exist in communities of practice. Within communities of practice, shared conventions allow one professional to understand another without spelling everything out. Information management is the work of making that social knowledge legible, transferable, and reliable across people and over time. It requires governance, curation, editorial judgment, and institutional memory. None of these are properties of data. All of them are properties of human organizations.

This is where AI systems are most exposed. A language model trained on vast amounts of text has absorbed enormous amounts of contextual signals. But signal is not the same as understanding and pattern is not the same as meaning. The model has no institutional memory and carries no provenance. It cannot distinguish between a notation written by a senior clinician with thirty years of practice and one written by a first-year resident on an overnight shift. It has processed the words without inheriting the social fabric that gives the words their weight. Context exists in the negative spaces surrounding text, behind the text and beneath the text. Machines trained only on text inherit none of that surrounding structure unless humans deliberately put it there.

The irreducible complexity of context points to something deeper than an engineering problem. Human beings cannot resolve contextual ambiguity through enumeration or statistical inference alone. Humans are an integral part of context interpretation and derive understanding from lived experiences, cultural knowledge, embodied knowledge and social awareness—all facets that evade capture in formal systems.

The cardiologist who writes “ha” has internalized a clinical world — its institutional rhythms, its provenance chains and its unwritten conventions. When the cardiologist’s notation is flagged as ambiguous, the residuals of interpretation are revealed, which formal systems don’t capture. The residuals are the hallmarks of meaning that lives in human practice.

Machines can improve how they document, record and represent context but must rely upon humans to understand and model context. Context is a human-led social agreement that must be maintained. Humans decide what counts as context and what constitutes knowledge. These are not decisions that can be automated away or delegated to a model. They require governance, judgment, and the kind of institutional accountability that only human organizations can provide. The work is sociotechnical by nature— the deliberate alignment of human knowledge practices with the systems built to extend them.

Context will always depend on context.

ABOUT THE AUTHORS

W.H. Inmon is widely recognized as the father of the data warehouse and has spent decades helping organizations structure and leverage their data assets for analytics and decision-making. Bill has sold over 1,500,000 books in his life

Jessica Talisman is a semantic infrastructure consultant and knowledge organization expert specializing in ontology development, controlled vocabularies, and enterprise semantic architectures that bridge library science principles with modern AI and knowledge management systems. Jessica created the Ontology Pipeline, a framework for building semantic knowledge infrastructures and ontologies.

Some of Bill’s latest books include –

DATA ARCHITECTURE – BUILDING THE FOUNDATION, with Dave Rapien, Technics Publications.

STONE TO SILICON – A HISTORY OF TECHNOLOGY AND COMPUTERS, with Dr Roger Whatley, Technics Publications.

Articles and presentations by Jessica can be found on Linkedin, Substack and Youtube.

By Andreas Schurch

In the early days of business intelligence, the problem appeared to be one of data. Data was locked in applications. Data arrived late. Data conflicted across reports. So the industry did what it had to do. It built data warehouses, data marts, ETL systems, dashboards, semantic layers, and all the machinery needed to move and organize data.

Much good came from that work.

But there was another problem hiding in plain sight. That problem was meaning.

A word such as revenue looks simple. A word such as customer looks simple. A word such as churn looks simple. But each of these words is actually a bundle of decisions. What is included? What is excluded? What time period applies? What source system is official? At what level of detail is the number measured? Which business exceptions matter, and which do not?

For many years, organizations were able to live with this ambiguity. The ambiguity was expensive, but it was often hidden. Reports disagreed quietly. Teams argued in meetings. Analysts reconciled numbers by hand. Executives learned which dashboard to trust and which dashboard to ignore.

AI has changed the visibility of the problem. An AI assistant is expected to answer immediately. If the organization has multiple definitions for the same metric, the assistant will still produce an answer. It may be a plausible answer. It may even be a numerically correct answer for one definition. But if it is the wrong definition, the damage is immediate. Metric chaos that once lived in back rooms and spreadsheet tabs now surfaces in front of the CFO.

This paper argues that the metric definition is the smallest unit of business meaning. It is the point where language, data, and decision making meet. If that unit is vague, the rest of the stack becomes fragile. If that unit is explicit and maintained, the stack becomes trustworthy.

Looked at more broadly, the structure around these definitions can be thought of as a Metrics Ontology. The ontology describes the concepts and relationships required to give business numbers stable meaning in the AI era.

THE HIDDEN PROBLEM OF METRIC MEANING

The modern data stack is very good at moving data. It is good at storing data. It is good at transforming data. It is getting better and better at serving data through semantic layers and AI interfaces.

What it is not naturally good at is preserving business meaning.

Business meaning is not simply a matter of arithmetic. The formula for gross margin may be short. The debate around gross margin is usually not short. The formula for churn may fit on one line. The decisions about grace periods, reactivations, trial conversions, and contract boundaries do not fit on one line.

This is why so many organizations suffer from a strange contradiction. They are rich in data and poor in agreement.

The missing layer is the layer where the business says, with precision, what a metric means, who owns that meaning, what variants exist, how the metric may be sliced, where it appears, and how changes are approved over time. Without that layer, every downstream implementation is built on a shaky foundation.

Figure 1 shows the real issue. One metric name can easily produce several numbers. Each number may be defensible. But only one can be official for a given context.

Figure 1. One metric name, multiple numbers.

EARLY APPROACH 1 - THE METRIC IN PEOPLE’S HEADS

The first place metric meaning lives is in the heads of experienced people.

Finance knows what revenue means.
Operations knows what fill rate means.
Marketing knows what pipeline means.
Customer success knows what churn means.

This works for a while. In a small organization there are only a few reports and only a few people making decisions. Meaning is carried socially. One person asks another person what a metric means, and the answer is passed along.

But this approach breaks as soon as the organization grows. People leave. New teams arrive. New systems are added. The same word travels into new contexts, and its original meaning becomes blurred. When knowledge lives only in people, the organization creates a single point of failure. When people move on, meaning drifts.

EARLY APPROACH 2 - THE METRIC IN THE REPORT

The second place metric meaning lives is in the report itself.

This is a step forward. When a number appears on a dashboard or in a board deck, at least it has some visible context. The report title, the surrounding charts, and the audience all provide hints as to what the number means.

But a report is not a durable home for meaning. Reports consume metric definitions. Ownership belongs upstream, where definitions are versioned and governed.

Once a metric appears in ten dashboards, five spreadsheets, three slide decks, and two planning models, the report-centered approach collapses. Every copy of the metric starts to drift. One dashboard applies a filter. Another changes a date rule. A third pulls from a different source system. The report becomes a hiding place for logic rather than a window onto logic.

EARLY APPROACH 3 - THE METRIC IN THE WIKI

The third place metric meaning lives is in a wiki, a document, a ticket, or a data catalog.

This is another step forward. At least now there is an attempt to write the definition down. It becomes searchable. It can be reviewed. It can be shared across teams.

But a wiki page or ticket rarely carries clear authority. There is often no clear owner, no approval status, and no reliable signal that the definition is current.

A paragraph may say that churn is the percentage of customers lost in a month. That sounds fine until someone asks: Are pauses counted as churn? What about partial terminations? What if a customer leaves one product and keeps another? Which customer hierarchy is official? Which clock defines the month? Are we counting logos, contracts, or revenue?

Writing the definition down is progress. But wikis, tickets, and documents rarely carry the official stamp. There is often no clear owner, no approval status, no current version, and no reliable signal that the definition is still valid. In many companies, there are multiple wikis. You find five pages for the same metric, each with a different definition. The definition becomes archaeology.

Even when the prose is well written, it rarely serves as an operational contract. It usually omits the mechanics that determine the number: inclusion and exclusion rules, grain, allowed slices, time windowing, attribution rules, and the declared source of truth.

EARLY APPROACH 4 - THE METRIC IN CODE

The fourth place metric meaning lives is in code.

This is where many modern organizations are today. The metric is defined in SQL, in a semantic model, in a BI calculation, in a notebook, or in a transformation framework. This is much better than having no formal definition at all. Code is precise. Code is executable. Code can be versioned.

But code has a different problem. Code is implementation, not business language.

Most business users do not want to read SQL in order to understand churn. They do not want to inspect joins and window functions to understand whether revenue is recognized or billed. Code tells an engineer how the metric is computed. It does not necessarily tell the business what the metric means, which variants are official, which slices are allowed, or why the definition changed last quarter.

The fool’s game is to assume that the formula alone defines the metric. It does not. The formula is one component of meaning. It is not the whole of meaning.

THE METRIC DEFINITION MODEL

If the early approaches are incomplete, what is needed?

What is needed is a metric definition model in which the metric is treated as a first class business object.

The metric definition model does not replace data warehouses, semantic layers, dashboards, or code. It sits above them. It supplies the official business meaning that those downstream assets must honor.

In one sense, the metric definition model is a companion to the Business Language Model - BLM. The BLM identifies the words that matter, the context of those words, and how those words are to be interpreted. The metric definition model does something parallel for business numbers. It identifies the metrics that matter, the context that makes a number valid, the variants that are official, and the relationships that connect the number to reports, slices, source systems, and other metrics. In both cases the issue is not merely storage. The issue is controlled meaning.

This parallel matters because the modern corporation lives in two worlds at once. One world is text, where contracts, emails, policies, and conversations must be understood in context. The other world is metrics, where revenue, churn, margin, and growth must be defined before they are calculated. If language is loose in either world, trust suffers. The BLM turns raw language into governed meaning. The metric definition model turns raw arithmetic into governed business meaning. Both serve as a bridge between what people say and what the enterprise can safely analyze.

Figure 2 shows the progression. Metric meaning moved from heads, to reports, to documents, to code. The next step is not merely another storage location. The next step is a governed object with structure, ownership, and relationships.

At its heart, the metric definition model says something very simple:

A metric is not just a number.
A metric is not just a formula.
A metric is a governed statement of business meaning.

Figure 2. Evolution of where metric meaning lived.

THE ANATOMY OF A METRIC DEFINITION

What then does a metric definition look like?

At first glance, the physical structure is simple. A metric definition can be expressed as a small set of fields. But, as with many simple structures, the simplicity of the container hides the complexity of the content.

A well formed metric definition normally contains the following components:

A canonical name. This is the official name of the metric.

A business purpose. Why does the organization care about this metric?

A formula or source declaration. Is the metric calculated? Or is it sourced from a system of record?

A grain. Is the metric measured by customer, by order, by policy, by invoice, or at some other level?

A time rule. What is the measurement period? When is an event considered to belong to that period?

Inclusion and exclusion rules. Which records count? Which records do not count?

Units and conventions. Currency, percentages, decimal behavior, and similar conventions matter.

Synonyms and acronyms. The organization may say gross sales, bookings, top line, or revenue, depending on who is speaking.

Allowed slices. By region, product, channel, customer segment, and so forth.

Ownership and status. Who is responsible for the definition? Is it draft, reviewed, approved, or retired?

Version history. Meaning changes over time and those changes need to be visible.

Source systems. Where does the underlying data originate?

Dependent metrics. What other metrics are required in order to compute this one?

Reports and uses. Where does this metric appear?

Figure 3 shows these components. The point is not to create a giant bureaucracy around every number. The point is to make explicit the decisions that are already being made implicitly.

Figure 3. The anatomy of a metric definition.

A SIMPLE EXAMPLE - NET REVENUE

Consider a metric called net revenue.

At first it appears obvious. But once the questions begin, the hidden complexity appears immediately.

Is net revenue based on orders, invoices, or recognized accounting entries?
Are returns subtracted in the current period or the original sales period?
Are taxes included?
Are discounts included?
Are credits and rebates included?
Which currency is official?
At what exchange rate is the amount translated?
At what grain is the metric valid: line item, invoice, customer, or month?
Which source system is authoritative when the CRM, billing system, and general ledger disagree?

Now the point is clear. A metric that looked like one line of arithmetic is actually a governed business object. The formula matters, but the formula by itself is not enough.

Figure 4 shows a simple example of how one metric sits inside a larger web of slices, reports, sources, and dependent metrics.

Figure 4. One metric in context.

SLICES, REPORTS, AND SOURCES

A metric definition does not live alone.

In practice, three other objects are needed if the business is to make consistent use of a metric.

The first object is the slice. A slice is a business friendly way of expressing how a metric can be broken down. In technical terms, the slice is a dimension. Region, product, channel, policy type, customer segment, and month are all examples of slices. Slices may also have hierarchies. Country rolls into region. Product rolls into product family. City rolls into state, and state rolls into country. Once these slices are defined, dimensional modeling follows naturally.

The second object is the report. A report tells us where a metric is actually used. This matters because meaning is often discovered at the point of use. A metric that appears on an executive dashboard, a regulatory filing, and a weekly operating review is not a casual metric. It is a metric that deserves extra scrutiny.

The third object is the source. Some metrics are computed from other metrics. Others are taken from a system of record. In many companies the source question is not trivial. Does revenue come from the CRM system, the billing system, or the accounting system? If multiple source systems exist, they must be made explicit.

Figure 5 shows a simple way to organize the work: metrics, slices, reports, and sources. This four part structure is simple enough for business users to understand and precise enough for engineers to implement.

This four part structure can also be thought of as a simple Metrics Ontology. The ontology describes the concepts and relationships required to interpret business numbers consistently across the enterprise. Metrics define the measure. Slices define how the measure may be broken down. Reports define the analytical context of use. Sources define where the underlying data originates.

The Metrics Ontology is the conceptual structure. It defines the entities and relationships required to interpret business numbers consistently. The metric definition model is the operational form used to capture that structure. It is where definitions are recorded, approved, versioned, and maintained. This is intentionally a small ontology. Most organizations first need agreement on a few core metrics before they attempt a larger enterprise ontology. Build what you need, then extend as needed.

Figure 5. Metrics, slices, reports, and sources.

HIERARCHY AND RELATIONSHIPS

Some relationships in a metric model are hierarchical.

A slice such as geography may contain region, country, state, and city.
A slice such as product may contain category, family, and SKU.

These hierarchies matter because they allow normal rollups and drilldowns. If a number is known by city, then it can often be rolled up to state or country. If a number is known by product family, then it may be compared across categories.

But not all useful relationships are hierarchical. A Metrics Ontology must also capture non-hierarchical relationships between metrics.

A metric may depend on another metric.
A metric may be sourced from a system.
A metric may appear in a report.
A metric may influence or be influenced by other metrics.

For example:

Net revenue depends on gross revenue and discounts.
Gross margin depends on revenue and cost.
Customer churn may affect lifetime value.
Net claims ratio may depend on claims paid and premiums earned.

These relationships do real work. A hierarchy tells us what kind of thing something is. A dependency tells us how a number is computed.

VALUE DRIVER TREES

Calculation dependencies explain how a number is produced.
Value driver relationships explain how the business believes results are created.

When value driver relationships are made explicit, they form a value driver tree.

Outcome metrics sit at the top.
Driver metrics sit below.
Operational metrics often sit further down.

The tree shows which levers move an outcome. It supports root cause analysis, planning, and prioritization. It also exposes missing drivers and redundant drivers.

A calculation graph tells us what must be computed first.
A value driver tree tells us what must improve first.

Both are part of the Metrics Ontology.

Figure 6. Example value driver tree showing outcome metrics and their drivers.

Figure 7. A slice hierarchy.

Figure 8. Relationships around a metric.

Figure 7 and Figure 8 show the difference between hierarchy and relationship. Both are needed.

SCOPE AND GRANULARITY

Every metric program faces two basic questions.

How much should be modeled?
How much detail should be modeled?

The first question is a question of scope. It is tempting to try to define every metric in the company. That is usually a mistake. The most effective programs start with the metrics that matter for a concrete business need: board reporting, finance signoff, regulatory reporting, AI assistant readiness, or a major transformation initiative.

The second question is a question of granularity. Some metrics can be defined at a broad level and still be useful. Others need great precision.

Take customer churn. A broad definition may be sufficient if the only goal is to assign ownership and route questions to the right team. But if the task is to compare churn across products, countries, and contract types, then much finer detail is required.

There is always the temptation toward over modeling. One can define endless subtypes, endless exceptions, endless relationship types, and endless annotations. The result is often a model that impresses architects and exhausts everyone else.

The right level of granularity is the level that supports the decisions that need to be made.

Figure 9 shows the tradeoff between scope and granularity: models that stay too coarse become vague, while models that span too many domains at too fine a granularity become slow and hard to maintain.

Figure 9. Tradeoff between scope and level of detail in a metric model.

THE BUSINESS VALUE OF EXPLICIT METRIC MEANING

Why go to this trouble?

The first benefit is trust. If a metric is defined explicitly, owned explicitly, and versioned explicitly, then disagreements can be resolved against a visible standard rather than by personality and memory.

The second benefit is speed. Once a metric definition exists, every new dashboard, semantic layer, BI model, and AI assistant can draw on the same business meaning. Teams stop reinventing the same metric in slightly different ways.

The third benefit is lineage. When a metric depends on other metrics, or on source systems, it becomes possible to understand impact. If the definition of gross revenue changes, which downstream metrics and reports are affected? If a source system is replaced, which definitions must be reviewed?

The fourth benefit is business design. When value driver relationships are explicit, they form a value driver tree. The organization can see gaps and redundancies. Some important outcomes have no supporting driver metrics. Some metrics exist only because they were inherited from old reports and no longer matter.

Explicit meaning is not just an administrative convenience. It is an asset.

AI AND THE NEED FOR RETRIEVABLE MEANING

AI has made all of this urgent.

An AI assistant speaks the language of the user. But the business answer it gives must still rest on an official definition.

Without a governed metric definition, the assistant is forced to infer meaning from whatever context it can find. It may search report titles. It may search code. It may search documentation. It may retrieve multiple conflicting fragments and still produce a smooth answer.

That is dangerous.

The correct pattern is different. First retrieve the official metric definition. Then retrieve the allowed implementation or source. Then answer the question, with the relevant context attached.

Figure 10 shows the contrast. On one side is the unguided answer path, where the model produces a plausible number. On the other side is the governed answer path, where the definition is retrieved first and the answer is grounded in approved business meaning.

AI did not create the meaning problem. AI merely removed the ability to ignore it.

Figure 10. Unguided and governed AI answer paths.

ONE DEFINITION, MANY IMPLEMENTATIONS

A final point needs to be stressed.

The metric definition model is not an argument against semantic layers or implementation frameworks. It is an argument for stabilizing them.

Modern organizations are multi platform. The same business may use Snowflake, Databricks, dbt, several BI tools, spreadsheets, and multiple AI interfaces. Each of these environments may need its own implementation artifact. But there should not be a different business meaning for each implementation artifact.

One governed metric definition may project into many downstream forms.

That is the practical value of the model. Define once in business language. Then implement many times in technical language, with drift controlled by governance and versioning.

That is also why a Metrics Ontology matters. The ontology provides the stable conceptual meaning that sits above each implementation. Without that layer, each platform slowly invents its own version of the truth.

Figure 11 shows this pattern. The official definition sits at the center. Semantic models, dashboards, and AI agents consume that definition rather than inventing their own.

Figure 11. One definition, many implementations.

MAINTENANCE OF THE METRIC DEFINITION MODEL

Like any useful model, the metric definition model is never finished.

Businesses change. Products are launched. Pricing changes. New channels appear. A merger introduces a new customer hierarchy. A regulator changes a reporting rule. Finance decides to restate a number. In each of these cases the definitions need to be reviewed.

The good news is that most business change is incremental. Once the first iteration of the metric model is built, later maintenance is usually marginal. The difficult work is not the hundredth update. The difficult work is the first decision to make meaning explicit.

The model therefore needs stewardship.

Someone must own each metric.
Someone must approve changes.
Someone must track versions and releases.
Someone must retire obsolete definitions when they are no longer useful.

Figure 12 shows why this matters over time. Without stewardship, drift returns.

Figure 12. Lifecycle and stewardship.

CONCLUSION

For years the data industry has treated structure as the central challenge. And structure is important. But structure alone does not solve the problem of meaning.

Meaning lives at the level of the metric definition.

That is where the business states what revenue is, what churn is, what margin is, what a customer is, what sources are official, which slices are allowed, and how changes are controlled.

If the metric definition is weak, the dashboards are weak, the semantic layer is weak, and the AI answer is weak.

If the metric definition is strong, the rest of the stack has a chance to be strong.

That is why the metric definition is the smallest unit of business meaning that can still be governed consistently. It is rich enough to carry context. And it is important enough to determine whether the modern data stack produces clarity or confusion.

The Metrics Ontology defines the conceptual structure. The metric definition model captures that structure in daily work. The ontology clarifies which entities and relationships matter. The model gives those entities fields, owners, versions, and lifecycle. Together they keep business numbers trustworthy in the age of AI.

ABOUT THE AUTHOR

Andreas Schurch is the Founder and CEO of Duodata, a company focused on business approved metric definitions and semantic consistency. He has led partnerships and go to market work in the data ecosystem, including roles at Matillion, Trifacta (Alteryx), and VaultSpeed. His work centers on reducing metric chaos by making business meaning explicit, versioned, and operational across modern data stacks. The four part conceptual model used at Duodata - Metrics, Slices, Reports, and Sources - was created by Luciano Franceschina, Duodata’s Co-founder and CTO.

Connect
LinkedIn: https://www.linkedin.com/in/andreas-schurch/
https://duodata.ai

By Dia Adams

Introduction: The Hidden Barrier to Data Success

In the rush to fully capitalize on artificial intelligence, companies are dumping billions into data infrastructure, hiring legions of engineers, and deploying cutting-edge algorithms[1]. Yet, despite these efforts, many organizations find their data initiatives stalling, their AI projects underperforming, and their promised transformations falling short[2].

The culprit, more often than not, is not the technology. It is culture. While executives and technologists focus on the mechanics of data (how to collect it, store it, and analyze it) they frequently overlook the human element: how people interact with data, how they trust it, and how they integrate it into their daily work. This oversight is costly, as organizations with strong data cultures are more likely to outperform their peers[3]. Yet, for every success story, there are countless tales of data projects that faltered not because of technical flaws, but because of cultural resistance.

The age of AI has intensified this challenge, as it demands a fundamental shift in how organizations think about and use data. In the AI era, data must become a cultural cornerstone that is embedded in decision-making, embraced by employees at every level, and governed by shared values and practices. For data strategists, engineers, and architects, this means that the most critical work may not involve writing code or designing pipelines, but shaping the norms, behaviors, and mindsets that determine whether data initiatives thrive or fail.

Why Culture Is the Linchpin of Data & AI Strategy

Artificial intelligence is a catalyst that accelerates and amplifies everything that is already working—or broken—in an organization’s data culture. If teams are siloed, AI will deepen those divisions. If data quality is poor, AI will magnify the errors. If leaders do not trust data-driven insights, AI will only give them more reasons to not to.

At the same time, AI also presents an unprecedented opportunity to reimagine how data flows through an organization. It forces leaders to confront essential questions: Who is responsible for data integrity? How do we ensure our models reflect our values, not our biases? How can we foster a culture where data is not just accessible, but actively used?

From Data as Asset to Data as Culture

For decades, businesses have treated data as an asset to be collected, stored, and monetized. But in the AI age, data must be more than an asset. It must become a cultural artifact, shaping how people work, collaborate, and make decisions.

Consider the example of Netflix. The company’s success is not just a result of its recommendation algorithms, but of a culture that treats data as central to every function, from content creation to customer service. Data is not the domain of a specialized team; it is the language of the organization.

Without updating culture, even the most advanced AI tools will fail to deliver on their promise.

Core Cultural Components of a Modern Data Strategy

The most effective data strategies are built on the shared values, behaviors, and practices that determine how data is used, trusted, and leveraged across an organization. For data professionals, understanding and shaping these cultural components is as important as designing pipelines or deploying models. Here, we examine the three pillars that define a data-driven culture in the age of AI.

1.Leadership and Vision: Setting the Tone from the Top

Data culture begins with leadership. Executive’s and senior managers must do more than endorse data initiatives; they must model data-driven decision-making in their own work. When leaders rely on intuition or anecdote rather than evidence, they send a clear signal: data is optional.

2. The Role of the Chief Data Officer

The rise of the Chief Data Officer (CDO) reflects the growing recognition that data requires dedicated leadership, and that this leadership requires more than technical expertise. The best CDOs are not just data experts, they are change agents. They have to bridge the gap between technical teams and the business, and that means speaking the language of both.

3. Alignment and Accountability

A data strategy should not exist in isolation; it should align with the broader business strategy and be communicated clearly and consistently. This requires defining roles and responsibilities: Who owns data quality? Who is accountable for governance? Who ensures that data is used ethically? Answering these questions with clear answers, provides the foundation necessary for initiatives to thrive.

[1]Zhang, R., Liao, R., Wu, T., & Shoham, Y. (2023). Artificial Intelligence Index Report 2023. Stanford Institute for Human-Centered AI (HAI).

[2]Bughin, J., Hazan, E., Ramaswamy, S., Chui, M., Allas, T., Dahlström, P., Henke, N., & Trench, M. (2023). The state of AI in 2023: Generative AI’s breakout year. McKinsey & Company. Link: McKinsey: The State of AI in 2023

[3]Mikalet, R., & Kotusev, S. (2020). The role of data governance in enabling data-driven decision-making: A systematic literature review. Journal of Data and Information Quality, 12(2), 1–22.

Collaboration and Silo-Busting: Data as a Team Sport

Data silos are the enemy of innovation, because when information is hoarded by departments or trapped in legacy systems, it limits the organization’s ability to derive insights and act on them. A cultural shift towards collaboration is necessary to break silos down.

The Rise of Federated Governance

Many leading organizations are adopting federated data governance models, which balance centralized standards with decentralized execution. This approach empowers teams to manage their own data while ensuring consistency and quality across the enterprise. Federation is about extending trust and giving teams the autonomy to use data while maintaining the guardrails that keep it reliable and secure.

Engineers, Architects, and Strategists: A Unified Approach

Data engineers, architects, and strategists often work in parallel, with little interaction. But in a truly data-driven culture, these roles are deeply interconnected. Engineers build the infrastructure, architects design the systems, and strategists ensure that data aligns with business goals. Strong organizations foster collaboration amongst these groups, creating cross-functional teams that can move quickly and adapt to change.

Technology can facilitate collaboration, but it cannot replace trust and communication. Tools like data catalogs, collaborative analytics platforms, and shared dashboards are only effective when teams are incentivized to use them.

Trust and Ethics: The Foundation of Data Culture

Trust is the bedrock of any successful data culture. If employees do not trust the data, they will not use it. If customers do not trust how their data is being used, they will take their business elsewhere. Building trust requires transparency, accountability, and a commitment to ethical practices.

Data Quality and Lineage

Trust begins with data quality. Organizations must invest in processes to ensure that data is accurate, consistent, and up-to-date. Data lineage (tracking the origin and movement of data) is equally important. If you can’t explain where your data came from, others can’t trust it.

Ethical AI and Bias Mitigation

As AI systems become more pervasive, the risk of bias and unintended consequences grows[1]. A data-driven culture must include mechanisms for identifying and mitigating bias, as well as clear guidelines for ethical AI use. This is not just a technical challenge; it is a cultural one. Ethics cannot be an afterthought, It has to be baked into every stage of the data lifecycle.

Transparency and Explainability

Transparency is key to building trust. Employees and customers alike need to understand how data is collected, used, and protected. Explainable AI, models that can be understood and interrogated by non-experts, is becoming a standard expectation[2]. Black box models might be powerful, but they are also risky. If you can’t explain how a decision was made, you’re taking on unnecessary reputational risk.

[1]Kordzadeh, N., & Ghasemaghaei, M. (2022). Algorithmic bias: Review, synthesis, and future research directions. European Journal of Information Systems, 31(3), 388–409. Link: https://www.tandfonline.com/doi/full/10.1080/0960085X.2021.1927212

[2]McKinsey & Company. (2021, November 30). Building AI trust: The key role of explainability. McKinsey QuantumBlack. https://www.mckinsey.com/capabilities/quantumblack/our-insights/building-ai-trust-the-key-role-of-explainability

Skills and Mindset: Cultivating a Data-Literate Workforce

Data literacy is now an absolute necessity. Every employee, from the C-suite to the front lines, must understand how to interpret data, ask the right questions, and make informed decisions.

Upskilling for the AI Age

Organizations must invest in training programs that go beyond technical skills. Data literacy includes understanding statistical concepts, recognizing bias, and communicating insights effectively. The aim of data literacy is not to turn everyone into a data scientist, but to give people the tools they need to work with data confidently. Data literate cultures thrive on curiosity and continuous learning. Employees should be encouraged to experiment, ask questions, and learn from failure.

Change Management: Overcoming Resistance

Cultural change is never easy. Resistance can come from any level of the organization, often rooted in fear or misunderstanding. Successful data leaders address these concerns head-on, communicating the benefits of data-driven decision-making and providing support for those who need it. Change is a process, not a one time event. Good leaders meet people where they are and bring them along.

AI-Specific Cultural Challenges and Opportunities

As organizations integrate AI into their operations, they face a new set of cultural challenges, including redefining roles and responsibilities and balancing innovation with ethical risks. However, these challenges also present opportunities to rethink how data and AI can drive value, foster collaboration, and create competitive advantage.

AI as a Cultural Catalyst

AI projects do not exist in a vacuum, and as such they expose and amplify the strengths and weaknesses of an organization’s data culture. If teams are siloed, AI will deepen those divisions. If data quality is inconsistent, AI will magnify the errors. If leaders are skeptical of data-driven insights, AI will only reinforce their doubts.

But AI also has the power to catalyze positive change. It forces organizations to confront long-standing issues: Who owns the data? Who is responsible for its quality? How do we ensure our models are fair and transparent? These are not just technical questions; they are cultural ones.

The Role of AI in Breaking Down Silos

AI projects often require cross-functional collaboration, bringing together data scientists, engineers, business analysts, and domain experts. This collaboration can break down silos and foster a more integrated approach to data. AI is a team sport; it requires diverse perspectives and a shared commitment to the outcome.

New Roles and Responsibilities

The rise of AI has given birth to new roles—AI ethicists, MLOps engineers, prompt engineers—and reshaped existing ones. These new roles require new ways of thinking, new forms of collaboration, and new approaches to governance.

The AI Ethicist

As AI systems become more powerful, the need for ethical oversight grows. AI ethicists are responsible for ensuring that models are fair, transparent, and aligned with organizational values. Ethics is not a checkbox; it’s a continuous process of questioning, evaluating, and improving.

The MLOps Engineer

MLOps engineers bridge the gap between data science and IT, ensuring that models are not just accurate, but also scalable, reliable, and integrated into business processes. This role requires a deep understanding of both technology and organizational dynamics.

The Prompt Engineer

With the rise of generative AI, prompt engineers have emerged as a critical link between humans and machines. Their work is not just about crafting effective prompts, but about understanding how language and context shape AI outputs—and how those outputs are used in real-world decisions.

Cultural Approaches to Responsible AI

Bias and Fairness

AI systems are only as good as the data they are trained on. If that data is biased, the AI will be too. Addressing bias is a technical challenge as well as cultural imperative. Organizations must foster a culture of critical thinking, where employees are encouraged to question data sources, challenge assumptions, and advocate for fairness.

Transparency and Accountability

Organizations must be transparent about how AI systems are developed and deployed. This includes documenting data sources, model training processes, and decision-making criteria. Accountability mechanisms—such as audits, impact assessments, and feedback loops—are essential for maintaining trust.

Innovation vs. Risk: Balancing Speed and Responsibility

AI moves fast. The pressure to innovate can lead organizations to cut corners, overlook risks, and prioritize speed over responsibility. A strong data culture balances these competing demands, fostering innovation while maintaining ethical standards.

The Role of Governance

Effective AI governance is about spurring innovation, while enabling it responsibly. This includes clear policies for data use, model development, and deployment, as well as mechanisms for monitoring and enforcement. Contrary to some misconceptions governance is not the enemy of innovation, it’s what makes innovation sustainable.

The Importance of Experimentation

A culture of experimentation encourages teams to test new ideas, learn from failures, and iterate quickly. This requires psychological safety—an environment where employees feel comfortable taking risks and sharing lessons learned. This cultural element is important, because innovation happens when people are not afraid to fail.[1]

Measuring and Nurturing Data Culture

Cultural change is not something that must be thoughtfully cultivated. Organizations need to measure the health of their data culture, identify areas for improvement, and take deliberate steps to nurture it.

Metrics That Matter

What gets measured gets managed. But measuring data culture is not as straightforward as tracking technical metrics. It requires a combination of quantitative and qualitative approaches.

Data Literacy Rates: How many employees understand the basics of data analysis, visualization, and interpretation? Organizations can assess data literacy through surveys, training participation, and certification programs.

Cross-Team Collaboration: Are teams sharing data and insights, or are they working in silos? Metrics such as the number of cross-functional projects, shared datasets, and collaborative tools usage can provide insight into the health of data culture.

Trust in Data: Do employees trust the data they use? Surveys and feedback mechanisms can help gauge confidence in data quality, accuracy, and relevance.

Feedback Loops

Data culture is far from static as it evolves over time. Organizations need feedback loops to assess progress, identify challenges, and adapt their approach.

Regular surveys and retrospectives can help organizations understand how employees perceive data culture. What’s working? What’s not? What barriers exist? This feedback is essential for continuous improvement.

Ironically, one of the best ways to measure data culture is to look at how data is used in decision-making. Are leaders relying on data, or are they defaulting to intuition? Are teams using data to drive innovation, or are they stuck in old ways of working?

Continuous Evolution

Data culture is not a one-time initiative; it’s an ongoing journey. As technology evolves, as business needs change, and as new challenges emerge, organizations must be prepared to adapt.

The Role of Leadership

Leaders must champion data culture, in words and in actions. This means modeling data-driven decision-making, investing in training and development, and holding teams accountable for cultural change.

The Importance of Storytelling

Data is powerful, but it’s not always intuitive. Storytelling—using data to tell compelling narratives—can help bridge the gap between technical teams and business stakeholders. Data tells a story. It is data leaderships job to make sure that story is heard.

The Future of Data Strategy Is Cultural

The age of AI has made one thing clear, data strategy is not just about technology. It’s about people and culture. Organizations that embed data into their culture—fostering collaboration, trust, and innovation—will thrive in the new AI era. For data strategists, engineers, and architects, this means that the most critical work may not involve writing code or designing pipelines. It may involve shaping the norms, behaviors, and mindsets that determine whether data initiatives succeed or fail.

[1]Edmondson, A. C. (1999). Psychological Safety and Learning Behavior in Work Teams. Administrative Science Quarterly, 44(2), 350–383.

Dia Adams is a seasoned data and AI leader who recently served as Enterprise Data Strategist at the White House. In addition, Adams has advised global companies on digital transformation and authored the best-selling executive guide “Winning With AI: A Blueprint for Corporate Leaders.” She currently serves as Chief Data & AI Officer at Datafolx.ai, a consultancy driving enterprise-wide data strategy and responsible AI adoption for corporations and government. She is presiding Board Chair at the Washington, DC-based think tank The AI Table—which is dedicated to documenting and disseminating best AI policies and practices.

By Ross Leher

Text is notoriously complex. There are many exceptions, twists and turns, double meanings in text that have to be manipulated and managed if you want to untangle text. The technology/discipline that is suited for grappling with the vagaries of text is an ontology and its associated taxonomies.

Taxonomies and ontologies are data models for enterprise knowledge management solutions that include the follow domains of knowledge:

• Search -- Document tagging

• B.I. Analytics

• Sentiment Analytics

• Predictive Analytics

• Spend Analytics

• Call Center Analytics

• Big Data

• Records Management

• Vendor Classification

• CRM-Client Classification

• Expertise Classification

• Product Information Management

• On Line Catalog Creation

• Many others

The process that follows below would seem to be very intuitive, yet many organizations fail to adhere to what we suggest is a recipe for success for information management projects.

What do I want to know?

What is my Role?

For a component

a. From Engineer Perspective

b. From Ordering Perspective

c. From Quality Perspective

And If I can know it, then value will be created for the enterprise.

A. What do I want to know?

This is the critical step one. We must first define the goal for our knowledge management project. If we don’t know where we are going, how can we possibly get there? Many organizations initiate and fund knowledge management, innovation management, search or big data projects without have a clear definition of what the goal for the project is. After the goal(s) are established, we must then define what value will be created for the enterprise. Who will use this knowledge and why will it matter—what value will be created for the enterprise.

B. What source data exists as a knowledge store for what I want to know? How reliable is the data?

We must define the source data/documents. Further, we must confirm or understand the reliability of the data/documents that we are analyzing. We have all heard the buzz surrounding “Data Lakes”. If the data lakes are full of polluted, unreliable data, then our results will be suspect at best and useless at worst. Incomplete data is another factor that impacts reliability. An example of this problem is where we have a patient’s medical record that is incomplete. If we perform analytics on such records that perhaps do not contain all of the medications that a patient is taking, error factors in the analytics will be introduced.

C. How do I process this data—what taxonomies/ontologies are models that apply to the data? What applications are to be used?

Various computer applications have defined purposes. Based upon “What we want to know”, we must select the appropriate applications that will be used. There are 100+ enterprise applications that can import taxonomies/ontologies for various enterprise knowledge purposes. When we have a bolt that needs to be tightened, we don’t use a screwdriver—rather we use the appropriate wrench. The correct enterprise application must be selected for the job at hand.

Further, it would be inappropriate in index human resources documents with a sales and marketing taxonomy/ontology. We must select the appropriate taxonomy/ontology that addresses the domain(s) of information that we are analyzing.

D. How do I present this data?

This is a key element. If we have correctly addressed A, B and C, we now have a result set that has converted our text to knowledge. How do we now present the results such that they are meaningful to the “consumers” of the knowledge? This can be in the form of knowledge graphs, search interfaces, visualization tools and other user interfaces.

Back to Basics

Taxonomies/ontologies are the data models for unstructured text.

I would suggest that there is no such thing as unstructured text—Unstructured text does not exist! And if I put a company brochure on the table or a newspaper article on the desk, any reasonable person would say that these are only two of millions examples of unstructured text. I would respond back and reply that we are able to ready the company brochure and understand it perfectly. We are able to read the newspaper article—it is not unstructured text for the human. So, we must make a modification to the statement above:

Taxonomies/ontologies are the data models for unstructured text that the computer application(s) can use to understand and interpret the text much the same way the human brain would.

Further, implicit taxonomies/ontologies are how the human brain organizes information, and these have been developed ever since the day we were born. When a child is born, the first concepts that they know are “Wet and Dry”, “Hungry and My Stomach is Full”. The newborn does not put words to these concepts, but from the day of birth, he/she starts to organize a knowledge base. At four years of age, the child knows that there is a top level term of food, a narrower term of fruit, baked goods, drinks, etc. and can name the types of fruit, baked good and drinks that fall below these broader terms. We organize the domain of knowledge geography with taxonomies:

Continents>>

Countries>>

States>>>

Cities>>>

Districts, etc.

The human brain organizes all domains of knowledge using implicit taxonomies/ontologies.

Explicit taxonomies/ontologies are simply representations of how the human brain and enterprises organize information.

And, explicit taxonomies/ontologies are used with enterprise applications for text to knowledge purposes.

Taxonomies/ontologies are the data models for enterprise information management solutions.

Derived Ontologies for Analytics and Search

Pre-Built ontologies are used to analyze many different corpuses of documents. In particular, there are many medical ontologies that have been developed over many years to analyze patient health care records. These ontologies are used by analytic software applications in conjunction with statistical algorithms that are used in these applications to index documents for search and/or B.I. or other purposes. These pre-built ontologies presuppose that the ontological relationships fully address the domain of knowledge for the corpus of documents that are to be analyzed. But what if some concepts or relationships don’t exist in these pre-built ontologies? Then, we have a data model that is incomplete.

Man and Machine Come to the Rescue—The Iterative Process

After the first indexing of documents that is applied to the corpus, domain experts can analyze terms and phrases that have been algorithmically generated from the indexing that do not appear in the formal ontologies that are being use for the indexing process. The terms and phrases then become candidate terms to be added to the formal ontologies for a second indexing of the corpus. This iterative process may take place multiple times to improve the indexing results.

In most cases, formal ontologies have not been developed for the many thousands of knowledge domains that a corpus of documents requires. In this case, pre-built curated taxonomies can be used to index the documents and the appropriate software application can create relationships across the taxonomies to develop a derived ontology.

For illustration, we are going to use a corpus of documents that relate to the ignition switch that created so many problems for General Motors. Various taxonomies that are relevant will be used to index this corpus of documents. A possible list would include

:A. Taxonomy of component parts

B. Taxonomy of manufacturers of component parts

C. Taxonomy of subcontractors to the manufacturers

D. Taxonomy of quality control documents

E. Taxonomy of engineering documents—(example below)

F. Taxonomy of manufacturing process documents---(example below

Examples:

• Engineering Design Process (104)

• Prototype Testing

• Stress Testing

• Test Requirements

• Testing Schedules

• White-Box Testing

• Detailed Design (28)

• Feasibility Assessment (3)

• Identification of Need

• Planned Obsolescence (5)

• Preliminary Design (3)

• Production Planning (10)

• Redesign

• Component Testing

• Critical Part Testing

• Destructive Testing (4)

• Development Testing

• Functional Testing

• Non-Destructive Testing

• Design Specifications (3)

• Design Testing (16)

• Black-Box Testing

• Design Research (2)

• Design Evaluation (4)

• Design Requirements (8)

• Manufacturing Production Processes

• Casting (14)

• Cleaning (2)

• Final Assembly (2)

• Finishing (30)

• Forming (45)

• Hardening (3)

• Inspection

• Insulating

• Joining (57)

• Lubricating

• Design Conceptualization (5) Machining (52)

• Mixing (3)

• Molding (19)

• Packaging (1)

• Testing (8)

Design Analysis (2)

The indexing of the document in this corpus using the pre-built taxonomies will generate relationships that are found in the documents among the selected taxonomies to create a derived ontology. This “grassroots” approach derives the ontology from the corpus of documents versus providing a pre-defined ontology which may or may not be accurate or extensible to other document sets that are related.

FOUNDATION ONTOLOGY

Man and Machine Come to the Rescue Again—The Iterative Process

The principle of man and machine described above is still valid and needs to be executed. After the first indexing of documents that is applied to the corpus, domain experts can analyze terms and phrases that have been algorithmically generated from the indexing that do not appear in the formal ontologies that are being use for the indexing process. The terms and phrases then become candidate terms to be added to the formal ontologies for a second indexing of the corpus. This iterative process may take place multiple times to improve the indexing results. Further, we may select additional pre-built taxonomies that we did not contemplate would be necessary in the initial phase.

My Favorite BI Program from the 1800’s in Dodge City Kansas

As discussed previously, implicit taxonomies/ontologies are how the human brain organizes information.

In 1850, on Main Street in Dodge City, there was a dress shop and the proprietress of the dress shop had customers that had patronized her shop for the past 20 years. The proprietress had served each customer who entered her store many times of the years and in her brain used a number of taxonomies to classify each person.

A partial list of the taxonomies that the proprietress had in her head:

· Type

o 01 One Piece

o 02 Two Piece

· 02 Dress Type

o 01 A-Line

o 02 Babydoll

o 03 Blouse and Skirt

o 04 Cardigan and Dress

o 05 Coatdress

o 06 Cocktail

o 07 Jacket and Skirt

o 08 Mock Turtleneck

o 09 Shirt

o 10 Swing

o 11 T-Shirt

o 12 Turtleneck

o 13 Other

· 03 Material

o 01 Cotton

o 02 Cotton Corduroy

o 03 Cotton/Polyester

o 04 Leather

o 05 Polyester/Boucle

o 06 Polyester/Spandex

o 07 Rayon

o 08 Rayon/Acetate

o 09 Rayon Crepe

o 10 Rayon Faille

o 11 Rayon/Polyester

o 12 Silk

o 13 Other

· 04 Construction

o 01 Crocheted

o 02 Knitted

o 03 Other

· 06 Sleeve Length

o 01 Long

o 02 Short

o 03 Sleeveless

· 07 Length from Waist

o 01 Less Than 19 Inches

o 02 19 to 19.50 Inches

o 03 20 to 20.50 Inches

o 04 21 to 21.50 Inches

For each of her customers, she would go to the back room and select three dresses to show them that matched their individual tastes.

She would be able to project what to purchase based upon the aggregate tastes of all of her customers.

And next door at the saloon, the bar tender was asking the cowboy who just walked in “Hey Charlie, will you have the usual”

The human brain stores and organizes and stores knowledge using implicit taxonomies and ontologies that describe domains of information. From the beginning of time, these are the data models that organize knowledge. And the formalization of these models for use in computer applications enhances the enterprise applications that we all use in the world of business.

Ross Leher has a background that includes international commerce, trade, and retailing, and has a proven record of growing innovative businesses in highly competitive markets. For 25 years, he served as Chairman and Chief Executive Officer of Robert Waxman Inc., the fifth largest camera retailer in the United States with over $50 million in annual sales. Robert Waxman, Inc., in business for over 38 years, was recognized as a leader in its business segment and for its success in innovating and applying new information technologies to facilitate sales and commerce. He was responsible for expanding Robert Waxman, Inc. beyond a successful retail camera operation into new areas that provided the underpinning for WAND. During his executive tenure at Robert Waxman Inc., Ross and his staff began the development of WAND products and technologies in 1982. Ross is a graduate of the University of California, Berkeley with a degree in Economics.

Johnson Kuan

For all the data warehouse enthusiasts out there, this analogy is for you.

AI is in its OLTP era. The world needs AI OLAP.

• OLTP = online transaction processing (think Postgres)

• OLAP = online analytical processing (think ClickHouse)

Most AI inference out there is transactional processing: one prompt request, one model response. Think chatbots, copilots, agents.

But what if we want to run AI as a big data processor, scanning large volumes of raw, unstructured data and generating structured, database-ready tables for downstream analytics? And by the way, we also want it to be fast, accurate, and cost-effective. Transactional AI inference was not built for this. What we need for this workload is Analytical AI inference. AI OLAP.

A Quick History Lesson

In the early days of databases, everything ran on a single system. Your point-of-sale transactions, your inventory lookups, and your end-of-quarter reporting all hit the same database. It worked until it didn’t.

OLTP systems were optimized for what they were designed to do: process individual transactions quickly and reliably. Insert a row. Update a record. Look up an account. Low latency, high concurrency, one operation at a time.

But when a business analyst needed to ask, “what were our top-selling products across all regions last quarter?” that query would scan across millions of rows, lock tables, and bring the transactional system to its knees. The workloads were fundamentally different, and the architecture needed to reflect that.

That’s why OLAP was born. Data warehouses. Star schemas. Pre-aggregated cubes. Systems purpose-built not for one record at a time, but for scanning, aggregating, and analyzing massive datasets. Eventually, columnar storage largely replaced the cubes, but the core insight never changed: the same data needed a completely different engine to serve a completely different job.

Sound Familiar?

AI inference today looks a lot like databases did before the OLTP/OLAP split.

Right now, the entire AI infrastructure stack—the models, the serving frameworks, the APIs, the tooling—is largely built around a single pattern: one request in, one response out. A user asks a question, the model answers. An agent gets a task, the model reasons through it. It’s transactional. And for that use case, it works beautifully.

But enterprises are sitting on oceans of unstructured data: customer support tickets, compliance communications, user-generated content, call transcripts, contracts, claims. By most estimates, 80% or more of enterprise data is unstructured. And the question they’re asking isn’t “can AI answer one question?” It’s “can AI process all of this?”

That’s an analytical workload. And trying to force it through transactional infrastructure is the modern equivalent of running your quarterly reporting queries on your production OLTP database. You can technically do it. But it’ll be slow, expensive, brittle, and operationally painful.

The 80% Problem

There is no intelligence without knowledge. And for most enterprises, 80% of their knowledge is trapped in unstructured data and invisible to analytics.

Structured data (the rows and columns living in databases and spreadsheets) accounts for roughly 20% of what an enterprise generates. The other 80% is unstructured: support tickets, compliance communications, call transcripts, contracts, claims, legal filings, clinical notes, user reviews, internal memos. It’s the data that actually tells you what’s happening in your business.

And for decades, it’s been essentially dark matter. You know it’s there. You know it matters. But you can’t query it, aggregate it, or build dashboards on it. Not because the data lacks value, but because there was no scalable way to convert it into something an analytical system could work with.

It’s not for lack of trying. Enterprises have thrown everything at this problem. Keyword search and regex rules catch the obvious patterns but miss context, nuance, and anything you didn’t think to look for in advance. Manual review is accurate but doesn’t scale. You end up sampling 5% and hoping it’s representative. Offshore labeling teams improve coverage but introduce latency, inconsistency, and management overhead. Traditional NLP promised automation but required months of feature engineering per use case and still broke on anything outside its narrow training distribution.

The result? Most enterprises have simply accepted the blind spot. Compliance teams sample. Customer experience teams triage by keyword. Content moderation teams react to flags. Nobody is actually analyzing the full picture, because the full picture was never analytically accessible.

Large language models change the equation. For the first time, we have models that can read unstructured text and extract structured meaning with genuine reading comprehension, not simple pattern matching, not keyword heuristics, but actual semantic understanding. Sentiment, intent, topics, entities, risk indicators, obligation clauses, clinical findings. The kind of extraction that used to require a human reader.

But here’s what most people miss: the breakthrough isn’t the model. It’s what becomes possible when you pair that capability with the right infrastructure. A model that can understand one document is interesting. An engine that can process every document in your data lake and produce structured, analytics-ready output is transformational.

That’s the shift. The 80% doesn’t have to stay dark. But unlocking it requires treating AI processing as what it actually is: an analytical workload.

What Makes the Workloads Different

The distinction isn’t just about volume. It’s about the entire shape of the work.

Transactional AI inference is interactive. A human (or an agent) is in the loop, waiting for a response. Latency matters because someone is on the other end. The input is typically a single prompt or a short conversation. The output is freeform, natural language meant to be read by a person.

Analytical AI inference is batch oriented. No one is sitting there waiting for row 47,312 to finish processing. What matters is throughput, cost-efficiency, and consistency across the entire dataset. The inputs are thousands or millions of records, often with the same analytical question applied uniformly. And the output isn’t a paragraph. It’s structured data. Labels, scores, extracted fields, classifications, summarizations. Data that flows into dashboards, databases, and downstream systems.

When you frame it this way, the architectural requirements diverge sharply:

The Gap in the Market

Today, if an enterprise wants to run AI on a million documents, here are their options:

Option 1: Loop over a chat API. Take your dataset, write a script, and call GPT/Claude/Gemini for each record. This technically works, but even with concurrent calls you’re hitting rate limits, paying conversational model prices for batch processing, managing retries and backoff logic, getting inconsistent outputs across records, and praying your structured output parsing doesn’t break at row 500,000. And you’re paying for general-purpose models that were built to do everything—creative writing, open-ended reasoning, world knowledge—when all you need is accurate, structured extraction from your specific data. Every API call is burning compute and money on billions of extra model parameters that we don’t actually need for the task.

Even with batch APIs from the major providers, we still need to construct a prompt for every record, format JSONL files, manage job submissions, parse outputs, and validate schemas ourselves. You’re building a DIY analytical pipeline every time. You’ve essentially duct-taped OLTP infrastructure into an OLAP job.

Option 2: Build it yourself. Stand up your own inference cluster, fine-tune a model, build the orchestration layer, handle schema validation, manage distributed processing, and maintain it all. This is what well-resourced ML teams at big tech companies do. For everyone else, it’s a 6 to 12-month project before you process your first production record.

Option 3: Don’t use AI at all. Stick with keyword matching, regex rules, and manual review. This is what most enterprises are still doing, not because they don’t want AI, but because the infrastructure to use it at analytical scale doesn’t exist as a product.

None of these options are ideal. We can do better. The market is missing the analytical layer.

What AI OLAP Actually Looks Like

Just as data warehouses weren’t just “bigger databases”, AI OLAP isn’t just “more API calls”. It’s a fundamentally different architecture optimized for a fundamentally different workload.

An analytical AI engine should be built from the ground up for batch, structured processing of unstructured data. That means purpose-built models that trade conversational fluency for structured output accuracy. Smaller, faster, and more cost-effective because they’re not carrying the overhead of capabilities the workload doesn’t need. You don’t need a model that can write poetry and recall ancient philosophy to classify a support ticket.

It means distributed inference that can process 50,000 records in seconds, not 50,000 API calls over hours. It means schema-aware processing that guarantees your output conforms to the exact structure your downstream systems expect, without the performance degradation that comes from forcing structured output at decoding time. And it means a delivery pattern that fits into existing data infrastructure. Data in, structured results out. Not a chat window.

In fact, if we’re being precise, analytical AI inference sits at the intersection of ETL and OLAP. It transforms unstructured data into structured output. That’s the extraction and transformation. But it does so at analytical scale, on demand, across millions of records. That’s the OLAP. It’s the part of the AI data stack that turns raw text into queryable, structured datasets.

The value of OLAP was never the query engine alone. It was that it unlocked an entirely new category of work. Before data warehouses, business intelligence was a niche discipline limited to IT specialists running custom reports. Once you could reliably scan and aggregate millions of records, BI became accessible to the entire organization, and an entire ecosystem of dashboards, reports, and data-driven decision-making emerged.

AI OLAP unlocks the same kind of shift. When you can reliably process millions of unstructured records with AI, you don’t just automate existing workflows. You enable entirely new ones. Compliance teams can monitor every communication, not a 5% sample. Customer experience teams can analyze every ticket, not just the ones that get escalated. Trust and safety teams can classify every piece of content, not just the ones flagged by keyword rules.

For Analytics, All Roads Lead to SQL

Here’s the punchline: at the end of the day, to get insights from data, we need to run SQL on tables.

Every BI dashboard, every executive report, every analytical query in every enterprise runs on structured tables. That’s how the entire analytics ecosystem works. Tableau, Looker, Power BI, dbt, Snowflake, BigQuery, Redshift, ClickHouse. All of it expects rows and columns.

AI OLAP enables every company to turn all of their raw text into structured outputs that can now live inside a database table. Support tickets become rows with sentiment, topic, and escalation columns. Compliance communications become rows with violation type, risk level, and confidence columns. Contracts become rows with obligation, liability cap, and governing law columns.

The unstructured data that was invisible to analytics is now queryable. The same SQL that powers every other business decision can now run on data that used to require a human to read.

Enterprise Use Cases for AI OLAP

The pattern is the same everywhere: large volumes of unstructured text, a business question as a query we run across every single text record, and structured outputs that give us database-ready tables for analytics.

Regulatory compliance. Financial institutions under FINRA oversight are required to monitor broker-dealer communications for potential violations. Today, many firms review a small sample, often 5% or less, of emails, chats, and correspondence. An analytical AI engine can process 100% of communications, classifying each one by violation type, risk level, and recommended action, with field-level confidence scores. The shift from sampling to census changes the compliance posture from reactive to proactive. The same pattern applies across regulated industries: banking (BSA/AML), healthcare (HIPAA), insurance (claims review), and government (FOIA processing).

Content moderation and trust & safety. Platforms generating millions of pieces of user content per day face an impossible scaling challenge. Keyword filters catch the obvious violations but miss context-dependent toxicity, subtle policy violations, and emerging abuse patterns. Analytical AI processes every piece of content against a structured policy schema (violation type, severity, confidence, recommended action), turning an unscalable human review bottleneck into a continuous, exhaustive data pipeline.

Customer service operations. Enterprises handle millions of support interactions annually. The strategic value isn’t in any single ticket. It’s in the aggregate patterns across all of them. What are the emerging product issues? Which agent responses lead to resolution? Where are the systemic breakdowns? Analytical AI processes every interaction, extracting sentiment, topic, resolution status, and escalation triggers into structured tables that flow directly into BI dashboards. The insight that used to require a team of analysts reading a sample now comes from querying a table.

Legal research and contract analysis. Law firms and corporate legal departments manage vast repositories of case law, contracts, and regulatory filings. Finding relevant precedents, extracting obligation clauses, identifying risk provisions. This work is done manually by associates spending hundreds of hours. Analytical AI can process entire contract databases, extracting structured fields (parties, obligations, termination clauses, liability caps, governing law) into queryable datasets. What took a team of paralegals weeks and months becomes an afternoon.

Healthcare and clinical records. Improving health outcomes requires understanding what’s happening across entire patient populations, not individual charts. Clinical notes, discharge summaries, and physician narratives contain critical information (diagnoses, treatment responses, adverse events, social determinants) that never makes it into structured EHR fields. Analytical AI can process millions of clinical documents, extracting structured medical data that enables population health analytics, quality measurement, and outcomes research at a scale that manual chart abstraction never could.

The common thread isn’t the industry. It’s the workload shape. Millions of records. A structured analytical question. And a business that needs the answer across every record, not a sample.

Where Analytical AI Fits in the Data Stack

For anyone who thinks in terms of data architecture (and if you’re reading this, you probably do), the natural question is: where does this sit?

Analytical AI isn’t a replacement for anything in the existing stack. It’s a new layer that fills a gap that’s existed since enterprises started accumulating unstructured data.

Here’s the picture. You have unstructured data sources: support platforms, communication archives, document repositories, content management systems, claims databases. This data flows into a data lake. It’s stored, but it’s inert. You can retrieve individual documents, but you can’t query across them analytically.

The analytical AI engine sits between the data lake and the data warehouse. It reads raw, unstructured data from storage, applies a prompt uniformly across every record, and writes structured, tabular results back out. The output is clean, schema-conforming data: columns of labels, scores, and extracted fields, that look exactly like any other structured dataset.

From there, the structured output flows into the systems the organization already uses. Data warehouses. BI dashboards. Reporting tools. Downstream applications. No new visualization layer required. No special tooling. The analytical AI engine produces data that speaks the language your existing infrastructure already understands.

Think of it as a new type of ETL. Traditional ETL moves and reshapes structured data. Analytical AI ETL transforms unstructured data into structured data using AI inference. The “extraction” is semantic, pulling meaning from text. The “transformation” is converting that meaning into schema-conforming rows and columns. And the “load” is writing it to wherever your structured data lives.

The key architectural point: the analytical AI engine is a processing layer, not a storage layer, and not an application layer. It doesn’t store your data. It doesn’t serve end users. It processes data in bulk and produces structured output for downstream consumption. It’s infrastructure that makes the rest of your data stack more valuable by feeding it data that was previously inaccessible.

Right-Sized Models for the Task

There’s a widespread assumption in enterprise AI that bigger models are better models. The reasoning seems intuitive: a 100 billion parameter model must be more capable than a 4 billion parameter one. But capability for what? And at what cost?

General-purpose LLMs are built to do everything. They can write poetry, reason about philosophy, generate code, translate between languages, recall historical facts, and carry on freeform conversations. These are remarkable capabilities. They are also completely irrelevant to most enterprise analytical workloads.

When your task is classifying the sentiment of a support ticket, extracting compliance violations from broker communications, or identifying obligation clauses in a contract, you don’t need a model that has internalized the complete works of Shakespeare or can debate the finer points of epistemology. You need a model that can read text, understand its meaning in context, and produce an accurate, structured output that conforms to a specific schema.

Every additional parameter in a general-purpose model represents computational overhead. You’re paying for inference across billions of weights that encode capabilities your workload will never touch. At transactional scale (one request at a time), this overhead is a cost nuisance. At analytical scale (millions of records), it’s a cost explosion.

A right-sized model purpose-built for structured analytical tasks makes different tradeoffs. It doesn’t carry the parameters for creative writing, open-ended reasoning, or world knowledge recall. Instead, it invests its capacity in the capabilities that matter for the workload: accurate text reading comprehension, reliable structured extraction, and consistent schema-conforming output.

The result is a model that’s not just cheaper to run, it’s better at the job. A smaller model trained specifically for structured extraction tasks can outperform a model nearly 20 times its size on those tasks, precisely because it’s not distracted by capabilities the task doesn’t require. It’s the same principle the database world learned: a columnar analytical database outperforms a general-purpose row-store on analytical queries.

There’s also a consistency argument. General-purpose models, designed for diverse and creative outputs, naturally exhibit more variance in their responses. That’s a feature for conversation and a liability for analytics. When you’re processing a million records with the same schema, you need uniform, predictable output. A model built for analytical processing, trained on structured extraction patterns, produces the kind of consistency that makes the output trustworthy at scale.

Right-sizing isn’t about settling for less. It’s about recognizing that the best tool for a job is the one purpose-built for that job. Nobody installs a full Oracle database to serve a static website. The same logic applies to AI: match the model to the workload.

The Split Is Inevitable

The database world learned this lesson decades ago. You don’t run analytical workloads on transactional infrastructure. Different jobs need different tools.

AI is learning the same lesson right now. The industry has spent the last few years building increasingly powerful transactional inference: better chatbots, better copilots, better agents. That work is valuable and will continue.

But the next frontier isn’t making chatbots smarter. It’s making AI infrastructure that can process the vast majority of enterprise data that never enters a chat window. The unstructured data sitting in data lakes, flowing through pipelines, accumulating in storage buckets. Waiting to be understood. And while text is where the need is most acute, the same workload pattern applies to any unstructured data: images, audio, video. The shape is identical.

The transactional era gave us AI we can talk to. The analytical era will give us AI that works for us. Quietly, at scale, on the data that actually runs the business.

The OLTP/OLAP split is coming to AI. And just like in the database world, the companies that get the analytical layer right will define the next era of the industry.

About the Author

Johnson Kuan is the Founder and CEO of Queryboost, a startup building the analytical AI engine for unstructured data. He holds a Master of Science in Computer Science from Georgia Tech with a specialization in Machine Learning and has over 10 years of enterprise AI experience at AT&T and DIRECTV. Johnson won the Most Innovative Award at Andrew Ng's Data-Centric AI Competition for his Data Boost algorithm and is an open-source contributor to Ray, the leading distributed AI compute framework.

By W H Inmon

Dan Linstedt

Cindi Meyersohn

Technology began in the simplest manner. In the beginning there was a computer, some input and some output. The input was usually in the form of punched cards and the output was placed on a magnetic tape file.

Very quickly (as far as evolutions go) reports were written and input from a terminal became a possibility. With these basic components the IT organization began to blossom. Applications started to spring up like dandelions in the springtime.

With the proliferation of applications came the innocent extract program. The extract program read data in one place and transferred the data to another place. There was a need to share data from one application to another and its was thought that sharing of data was as simple ae moving data from one application to the next. Thus borne was the extract program.

The volume of data began to cascade. The magnetic tape file was an improvement over the punched card. But the magnetic tape file required that large amounts of data be read sequentially. The magnetic tape file brought with it an overhead of processing that was both onerous and unnecessary. Soon there was the disk. With disk storage data could be accessed directly.

The ability of disk storage to directly access data led to transaction processing. The network accessing the disc storage data base was managed by a teleprocessing monitor. The disc storage data base was managed by a high performance data base management system.

The advent of transaction processing opened the door to the usage of the computer into the day to day business of the corporation. With transaction processing came ATM machines, airlines reservation processing, bank teller processing, and so forth.

Once the door to transaction processing was opened, the computer exploded into the world of business. The computer went from being a useful commodity to being at the center of business activity. If the transaction processor went down, business immediately suffered.

Soon data was being extracted from one source to another, creating a bewildering avalanche of data.

To add to the complexity, the personal computer and the spreadsheet made its appearance. The personal computer and the spreadsheet changed the dynamics of the control of data. Once computer processing was controlled by the IT organization. With the introduction of the personal computer and the spreadsheet, the end user could own and manage their own data without the interference of the IT organization.

To add to the complexity, the personal computer and the spreadsheet made its appearance. The personal computer and the spreadsheet changed the dynamics of the control of data. Once computer processing was controlled by the IT organization. With the introduction of the personal computer and the spreadsheet, the end user could own and manage their own data without the interference of the IT organization.

With enterprise data a reality inside a data warehouse, different functional departments wanted their own customized data in a data mart. Marketing, sales, finance and others wanted to take the enterprise data and tailor it to their own purposes.

OLAP and dashboards made their appearance.

Thus borne was the data stack. Any one vendor did not have a complete solution to all of the information needs of the organization. Instead, multiple vendors and their various technologies were needed in order to satisfy the many technical needs of the organization.

Placing those different technologies on top of each other was known as the “data stack”

.The data stack appeared to satisfy the myriad needs of the technicians in coping with the avalanche of data. But the data stack had one major drawback. Unless there was an overlying architecture that sat beneath the stack, the stack produced conflicting and incorrect information.

In order to make the data stack successful it was necessary to unify the stack with an architecture.

Analogically, the data stack was the human body and the architecture was the brain and nerve system of the human body. The human body may be strong and flexible, but without a brain and a nerve system, the human body was worthless.

The same is true of the data stack. The data stack may provide technology needed to store and manage data. But if the data residing in the data stack is not believable, then the best data stack in the world does nothing to solve the problems facing the corporation.

Organizations found that in order to be successful that it was not sufficient to have just a data stack. Instead a data stack unified by an architecture of data was a necessity.

There is a a simple but uncompromising point to be made: a data stack without architecture cannot be trusted. That observation has aged well, but the environment around it has changed in ways that make the consequences more immediate and more visible. What once surfaced slowly now appears quickly. What once remained internal now emerges under scrutiny. And what was once considered a technical shortcoming is increasingly interpreted as a leadership failure.

The need to continue this discussion is not academic. The modern narrative surrounding platforms has quietly shifted responsibility away from design and toward procurement. Architecture is no longer debated. It is assumed. That assumption now lives inside funding decisions, operating models, and executive expectations in ways that did not exist when the data stack first took shape.

This shift deserves closer examination.

When Platforms Are Treated as Architecture

Modern platforms are no longer positioned as components within a system. They are presented as the system itself. The promise is rarely stated outright, but it is clearly implied: scale replaces structure, automation replaces intent, and integration replaces accountability. Architecture becomes something the platform is assumed to contain rather than something leadership must deliberately establish.

The belief persists because it appears reasonable. Platforms do perform work that once required careful design. They connect sources quickly. They absorb change. They deliver results faster than any prior generation of technology. The conclusion feels earned. If the platform handles complexity, architecture must already be embedded.

But platforms do not define meaning. They move data. They do not establish authority. They process transactions. They do not reconcile conflicting definitions of revenue, risk, customer, or product across business lines.

That work belongs to architecture

.The problem is not that platforms fail. It is that they succeed in ways that obscure responsibility. They move data efficiently without establishing who owns meaning, which transformations are authoritative, or how disagreement is resolved when results conflict. Those questions do not disappear. They simply resurface later, under conditions where time, patience, and optionality no longer exist.

At that moment, the distinction becomes clear. A platform can coordinate activity. Only architecture can coordinate accountability.

Speed Has Changed the Cost of Being Wrong

Earlier generations of data systems failed slowly. Inconsistencies accumulated quietly. Reports disagreed, but the impact was often localized. Time absorbed architectural weakness.

That buffer no longer exists. Cloud scale and automated pipelines compress the distance between design decisions and business exposure. What is incorrect propagates as quickly as what is correct. By the time disagreement is noticed, it has already influenced planning, funding, external commitments, or regulatory posture.

Speed magnifies consequences.

In regulated industries, regulators increasingly expect traceability and demonstrable lineage. In public companies, boards expect confidence in reported metrics. In highly competitive markets, strategic decisions are made on analytics that assume internal consistency.

When architecture is weak, the stack becomes fragile under stress.

This acceleration changes the nature of executive risk. The question is no longer whether the system works today, but whether it can explain itself tomorrow. When answers are required quickly, architecture is either present or it is not. There is no retroactive substitute.

Consider a common enterprise review where two leadership teams arrive with materially different numbers drawn from the same platform. The platform performed exactly as designed. The architecture did not. At that point, accountability does not remain distributed. It moves upward.

Understanding this shift requires examining the role of the architect more closely.

The Architect’s Role and the Temporal Fracture

Architecture is evaluated retrospectively.
It is built prospectively.

This temporal fracture is subtle but powerful. Architecture decisions are made during delivery, under schedule pressure, funding constraints, and competing priorities. They are judged later, when the system is examined under audit, board review, regulatory inquiry, or strategic reassessment. The individuals building the system rarely face scrutiny in the moment. The scrutiny arrives after the system has already shaped business decisions.

The absence of architecture is invisible during delivery.
It becomes visible during review.

During delivery, velocity dominates. Milestones are met. Pipelines execute. Dashboards render. The stack appears successful because it is functioning. There is no immediate signal that meaning is fragmented, ownership is unclear, or lineage is incomplete. In fact, the absence of architectural rigor may even appear efficient. Less friction. Fewer constraints. Faster output.

The problem emerges when explanation is required.

Architecture exists not merely to move data, but to define authority, preserve history, reconcile disagreement, and sustain meaning over time. When those elements are not deliberately designed, the stack still operates. What is missing is the institutional memory that proves how and why it operates.

How Executives Recognize Architectural Drift

Executives rarely see architecture directly. They see its effects. Recognizing architectural weakness requires observing patterns rather than artifacts.

Architectural drift is often present when:

• Business definitions vary subtly across reports with no clear authority to reconcile them.

• Transformation logic resides inside pipelines without documented ownership.

• Platform selection discussions are framed as architectural decisions.

• Architects defend tooling choices more strongly than business meaning.

• Questions about lineage produce explanations of processing rather than explanations of intent.

Resistance to executive direction is rarely overt. It is often expressed as technical inevitability. “The platform requires this.” “The stack works this way.” “The tool enforces this structure.” When the physical stack is treated as destiny, architecture has quietly yielded authority to implementation.

Another signal is silence.

When architectural review forums focus on performance, scalability, or integration mechanics but do not address ownership of meaning, reconciliation boundaries, or definitional authority, architecture is being shaped by the stack rather than governing it.

When Architecture Is Defined by the Stack

Designing based on the selected stack is not inherently wrong. Every architecture must operate within technical constraints. The fracture occurs when the constraints define the structure rather than inform it.

When the stack dictates architecture:

• Data models mirror physical storage patterns rather than business semantics.

• Integration flows reflect tooling capability instead of enterprise coherence.

• Governance becomes documentation layered on top of decisions already embedded in code.

• Architectural standards follow platform features rather than executive intent.

At that point, architecture is reactive. It describes what has been built rather than directing what should be built.

True architecture begins with purpose. It defines what constitutes authoritative information, how it persists across change, and who is accountable when definitions evolve. The stack should implement those decisions, not determine them.

Controls Executives Can Put in Place

If architecture is judged retrospectively, executive control must be applied prospectively.

First, architectural authority must be clearly separated from platform selection authority. Tooling decisions must not substitute for definitional decisions. Executives should require explicit articulation of business meaning, ownership, reconciliation rules, and lineage expectations before stack-specific design begins.

Second, architectural reviews must examine intent, not only implementation. Reviews should ask:

• Who owns this definition?

• How is historical change preserved?

• Where are reconciliation boundaries defined?

• How is disagreement resolved?

• What evidence exists that this meaning can be reproduced under examination?

These are architectural questions. They are not tooling questions.

Third, architectural artifacts must exist independently of the stack. If the architecture cannot be described without referencing specific platform features, then the stack has already shaped the design. Architecture should survive tool replacement. If it cannot, it was never architecture.

Fourth, escalation pathways must be explicit. When definitional conflict arises, there must be a defined authority empowered to resolve it. Without that mechanism, architectural decisions default upward only after conflict becomes visible.

Finally, architectural compliance must be evaluated at governance gates, not after production release. Once pipelines are live and dashboards are adopted, reversal becomes politically and operationally expensive. Prospective enforcement is uncomfortable. Retrospective correction is destabilizing.

The architect’s role is therefore not merely technical. It is structural. Architects translate executive intent into durable information structures that persist beyond tooling cycles and delivery waves. When that translation is absent, the stack performs. When review arrives, explanation falters.

Even with controls in place, the ultimate burden of architectural weakness follows a predictable path.

Where Accountability Ultimately Lands

Architecture has always existed to answer questions that arise after decisions are made. It explains what data represents, how it was formed, and why it should be trusted over alternatives. When that explanatory chain is missing, accountability does not vanish. It relocates.

If ownership of meaning is not explicit, leadership inherits it. If reconciliation paths are undefined, governance absorbs the burden. If no one can demonstrate how results were produced, responsibility rests with those who approved the system that produced them.

This is not a failure of execution. It is the predictable outcome of assuming that modern platforms eliminate the need for architectural intent. The organization continues to operate. Reports continue to render. Decisions continue to be made. What changes is who must answer when those decisions are questioned.

The quiet cost of this assumption is not operational. It is institutional. Over time, trust in information erodes, not because individuals acted improperly, but because the system cannot prove itself under examination.

At that point, the absence of architecture is no longer theoretical. It becomes visible in escalations, audit inquiries, funding friction, and governance intervention. The stack remains. The illusion of control does not.

ABOUT THE AUTHORS

Cynthia (Cindi) Meyersohn is Chief Operating Officer of DataVaultAlliance Holdings and Founder of DataRebels LLC, where she leads strategy, operations, and global enablement for the Data Vault 2.1 ecosystem. With more than four decades in information technology spanning commercial and federal sectors, she is a Certified Data Vault 2.1 Instructor and enterprise architect who has designed and implemented large-scale, auditable data warehouse solutions in highly secure environments. She holds a Master of Science in Systems Engineering from The George Washington University and a Bachelor of Science in Information Systems from Strayer University. She currently resides in Richmond, VA with her family, including seven grandchildren.

Dan Linstedt is the inventor of Data Vault and a recognized authority in scalable data architecture and enterprise data integration. He has advised global organizations on building resilient, audit-ready data ecosystems that support analytics, compliance, and AI at scale. As an AI strategist and industry educator, Dan focuses on aligning modern data engineering practices with emerging intelligent systems. His work bridges disciplined data architecture with practical innovation in advanced analytics and automation.

A Monograph on Trust, Context, and Master Data Management in the Age of AI

By Malcolm Hawker, CDO, Profisee

Executive Summary

As organizations accelerate their adoption of artificial intelligence, meaning has become the dominant concern in data management. Ontologies, knowledge graphs, and semantic layers are rightly viewed as essential to providing the contextual understanding of data that Generative AI systems require. Structured data alone is no longer sufficient.

But meaning without truth is confusion.

Trustworthy AI systems require more than semantic clarity. They require truth - grounded, explainable, and contextually appropriate truth. And truth, unlike meaning, lives at the record level. It is established through consistent, auditable identity resolution and reconciliation across systems, sources, and increasingly, across both structured and unstructured data. And it is applied to individual records, not objects or schemas or tables.

This monograph argues that the historical struggles of master data management do not reflect its irrelevance, but rather the ways in which it has been approached. Big-bang and consultant-driven implementations, framework-first thinking, domain-centric scope, and control-oriented governance have all undermined MDM’s ability to deliver value.

In an AI-driven world, MDM must evolve.

Modern MDM programs must support multiple versions of truth, recognizing that truth is contextually bound and purpose driven. It must act as connective tissue between structured data, unstructured knowledge, and semantic representations. It must prioritize explainability, and not just accuracy. And it must be governed pragmatically, where governance policies are focused on enablement and business outcomes rather than control.

When approached with discipline, pragmatism, and an unwavering focus on value delivery, MDM remains a foundational capability for trustworthy analytics and AI. Meaning provides understanding. Truth provides trust. Together, they enable confident, responsible decision-making at scale.

Part I: Meaning Without Truth

There is much talk these days about the importance of understanding meaning in data. This focus is largely a function of the fact that the one thing Generative AI needs most - rich contextual understanding - is largely missing from our highly structured databases.

Rows and columns are excellent at storing facts. They are far less effective at conveying intent, nuance, and real-world relationships. If we expect GenAI to unlock the latent value buried inside decades of highly structured enterprise data, we must provide more than schemas and tables. We must provide context.

This realization has driven a resurgence of interest in ontologies, knowledge graphs, and the broader discipline of knowledge management. Many are already calling 2026 the “year of the ontology,” and given the strategic importance of context in AI systems, that prediction may prove accurate.

This focus is entirely justified.

But in our rush toward meaning, many in the data management community are forgetting something equally important.

Truth.

To establish trust in data - and in the AI systems increasingly built on top of it - we must provide both meaning and truth. One without the other is insufficient.

Over the years, if there is one lesson my career has reinforced repeatedly, it is this: simply centralizing data does not make it accurate. Moving data into a warehouse, lake, or lakehouse does not ensure that it reflects reality. Scale does not create truth. Integration does not guarantee correctness.

Meaning without truth is confusion.

You can build the most elegant ontology imaginable - one that beautifully models customers, products, accounts, and interactions - but if you cannot say with confidence whether Bob Smith and Robert Smith are the same person, your insights will be wrong. Worse, they will look right, and GenAI solutions will confidently, but incorrectly, amplify the error.

This problem has existed for decades, and for decades many organizations have quietly worked around it rather than confronting it directly. We have often accepted ambiguity as inevitable, even as decisions have become more automated and more consequential.

The importance of pairing meaning with truth is something my friend Scott Taylor, the “Data Whisperer,” has been emphasizing for years. Much of my thinking on this topic has been shaped by that perspective, and I remain convinced that it is even more critical in an AI-driven world.

As an industry, we have built impressive capabilities around consistency, standardization, and integration. But accuracy - whether the data represents the real-world entity it claims to describe - has often eluded us.

This is where master data management becomes unavoidable.

MDM creates a bridge between meaning and measurement. It anchors semantic understanding to verifiable identity. Without that anchor, the progression from data to information to knowledge collapses under its own weight.

AI is the forcing function that makes this bridge impossible to ignore, and why MDM remains a critical enabling capability of any modern data ecosystem.

Part II: Why MDM So Often Failed

If truth is so fundamental, why has MDM struggled for so long?

The answer is not that MDM is wrong. It is that it has too often been pursued in ways that were disconnected from business reality.

Historically, many MDM programs shared a common set of characteristics. They were broad in scope, domain-centric, framework-driven, and dependent on centralized control of data. They promised transformation but delayed value. They prioritized architecture over outcomes.

MDM became something organizations were told they needed, but rarely something they could articulate the value of. This is especially the case where the data and governance requirements of a given business domain (like marketing), do not fully align to those at higher, more cross-functional levels of the organization.

Governance was positioned as a prerequisite rather than an enabler. The business value of MDM was only rarely quantified – and if it was – it only rarely connected with meaningful business KPI’s. Business stakeholders were asked to wait while foundations were laid. Inevitably, they stopped waiting.

Instead, they built workarounds - spreadsheets, shadow pipelines, manual reconciliation processes - not because they rejected MDM or governance, but because they needed results.

There is no better example of stakeholders getting tired of waiting than the birth and meteoric rise of the software market for customer data platforms (CDPs). CDP’s are marketing-specific solutions that use all the same capabilities of MDM’s, but are built for the scale, flexibility, and ease-of-use required by marketers. They were created out of a desire of marketing teams to have their own, domain-specific form of MDM, which is a demand that centralized IT teams managing enterprise MDM platforms were unwilling, or unable, to provide.

Marketers stopped waiting for central IT teams to deliver a ‘customer 360’, and many went and built their own solutions. These solutions, among many others, became the CDP software market - one that now cumulatively dwarfs the market size for MDM solutions.

This broader pattern of MDM being too big, or too bulky, or incapable of meeting the data requirements of business functions outside the analytical realm did real damage. It eroded executive confidence not just in MDM, but in data leadership more broadly. And once trust is lost, no amount of tooling can restore it.

In the age of AI, repeating these mistakes is far more dangerous.

AI systems amplify both insight and error. Plausible but wrong answers are more harmful than obvious failures. When AI outputs influence pricing, credit, medical decisions, or compliance, ambiguity around truth becomes unacceptable.

MDM must therefore be re-approached - not as a monolithic infrastructure initiative, but as a disciplined, pragmatic capability for enabling trustworthy decisions at scale - regardless of if those decisions are made by humans or algorithms.

Part III: Master Data Management Best Practices for an AI-Driven World

Business Value as the North Star

MDM should never exist for its own sake.

Truth is not valuable in the abstract. It is valuable only when it changes a business outcome. Executives do not fund accuracy; they fund results.

The most successful MDM programs I have seen begin by aligning data management and quality KPIs directly to business KPIs. Not loosely. Explicitly.

This requires reframing the conversation. Instead of asking how to improve data quality, the question becomes:

• How does identity uncertainty increase operational cost?

• How does duplicate data inflate business risk?

• How does lack of trust slow decisions or force conservatism, and what are the financial consequences of these delays?

• Where does ambiguity require manual intervention, and how much does this intervention cost?

Answering these questions requires partnering with the right business leaders - often in finance, operations, risk, or revenue functions - who understand both the business process being improved and the economics behind it.

Together, these partners can build pragmatic value models. These models do not need to be perfect. They need to be credible. For far too long data leaders have shrugged at the idea of building business models to measure the impacts of their investments in data management, largely because many believe that full causality is required for any model to be accepted in the business. Put another way, there is a long-held myth of many CDOs that if you cannot prove with 100% certainty that an investment in data management is the sole reason a business outcome has been delivered, then there’s no point in building those metrics.

This is an incorrect assertion, and it’s one that data leaders must abandon - both for the sake of their careers, and their integrity within their organizations.

After all, CDOs are the data people, and hearing the senior-most data person suggest that it’s impossible to generate data about the financial benefits of a mission-critical dependency for our organizations success seems both implausible and completely ridiculous.

Yet, many CDOs remain steadfast in this perspective, while all their peers in the C-suite can generate attribution models which strongly suggest that investments in things like employee satisfaction, better marketing, or more efficient procurement methods can all lead to improvements in the bottom line.

While many CDOs cling to the notion of ‘anything less than causal metrics to prove data is valuable is a waste of time’, their peers are taking far more pragmatic (and successful) approaches.

Leaders in departments across the org are showing that having fully causal models to prove, without a shadow of a doubt, that investments in marketing, HR, or finance can improve the bottom line are simply not required. They show that capturing data which is directionally correct, via attribution models, is more than sufficient to establish priorities, guide iteration, and justify continued investment.

CDOs looking to invest in MDM and data governance must take the same approach. Over time, as MDM improves truth, the models themselves improve, and will justify further investment.

Figure 1 below is a useful framework for CDOs looking to build a model which links their various data quality KPIs (like uniqueness, accuracy, completeness, etc.) to business KPIs. This linkage needs to happen at multiple levels starting at a field level, working all the way ‘up’ the value pyramid to the high-level strategies of the company as a whole.

Figure 1 – The Data and Analytics Metrics and Value Framework

MVP Thinking and Fast Value Delivery

One of the most common causes of MDM failure is overreach.

Organizations attempt to solve too many problems at once, define too many entities, and reconcile too many systems. The result is complexity without momentum.

An MVP (minimum viable product) mindset changes this dynamic.

Minimum viable MDM means starting small, delivering value quickly, and learning continuously. It prioritizes proof over completeness. Early wins build trust, and trust creates scale. This approach mirrors how successful AI initiatives already operate. Hypotheses are tested. Models are refined. Value emerges iteratively.

MDM must adopt the same posture, and doing so is much easier than you might think. Asking ‘what is the minimum about of people, process, and technology, bounded within an MDM program, needed to solve a single business problem?’ This includes the minimum amount of governance, the minimum amount of data, and the minimum level of engagement from the business.

When you ask these questions, and when you put a rabid focus on the delivery of value to the customers of an MDM solution in as short a period of time as possible, you can change the dynamic of MDM being big, bulky, and too complex – to something far more nimble, far more attainable, and far more valuable than MDM programs of the past.

Figure 2 shows a high level diagram of the MVP approach to MDM in action through the lens of an MDM operating model. Each MVP release builds on prior releases, and with each release, you build a stronger data management foundation. In time, over multiple releases, you will get closer and close to having 100% of both a data governance and management foundation – but where business value has been delivered at every phase.

The outer ring of this framework highlights all the things that a data leader cannot change in the short term – such as the business strategy, the overall corporate culture, or the governance maturity. These things are important, but cannot be influenced through any one MVP of an MDM program.

As data leaders focus on the MDM operating model within the inner ring – and the iterative delivery of value through MDM over time – they will slowly start to influence the outer ring. Taking this approach allows data leaders to avoid getting mired in a focus on things they cannot immediately change (like corporate culture), where instead they focus only on those things which they can change.

With a well-defined business outcome as the goal (eg. ‘increase customer retention 5% by eliminating duplicate customer records in our CRM’), MDM program leads must then ask: ‘what is the minimum I must deliver, for each driver of this framework, for an MDM program to deliver value?’ What is the minimum technology? What is the minimum governance? What is the minimum level of business engagement and alignment needed to deliver on the goal?

When these questions are answered, the scope for an MVP deployment is defined, and the chances of short-term success are drastically increased.

Figure 2 – An MVP Approach to MDM

Finding the Right Internal Customer

MDM does not start with data. It starts with someone who has a problem they want solved.

The strongest MDM programs partner with business leaders who already feel the pain of unreliable identity. These leaders are open to analytical approaches and motivated by outcomes, not compliance.

This is where product management becomes invaluable.

Treating data capabilities as products forces data teams to deeply understand customer needs, success criteria, and tradeoffs. It shifts the focus from delivery to adoption.

A focus on product management within an MDM program helps data leaders ask better questions:

• Who is the customer?

• What problem are we solving?

• How will we know if it worked?

• What tradeoffs are acceptable?

MDM programs that adopt a product discipline are far more likely to deliver relevance - and relevance drives adoption.

Process Over Domain

One of the most persistent - and least productive - patterns in MDM initiatives is a fixation on data domains. Customer domain. Product domain. Supplier domain. The assumption is that if we “solve” a domain, value will naturally follow.

In practice, this almost never happens.

Domains are abstractions. Businesses do not operate in domains; they operate through processes. Orders are fulfilled. Claims are adjudicated. Customers are onboarded. Risks are assessed. Revenue is recognized.

When MDM is framed around domains, conversations quickly devolve into debates about ownership, stewardship, and scope. Who owns the customer domain? Which system is the system of record? Whose definition is correct? These debates are rarely resolvable in isolation, and they consume enormous organizational energy without producing outcomes. Its these conversations, and the tension they naturally create in any organization, which have helped to give MDM programs a negative perception in the past, and its these perceptions we must change.

A process-centric approach changes the conversation entirely.

Processes have clear owners. Processes have measurable performance metrics. Processes have failure modes that executives already understand. Most importantly, processes create moments where truth matters - moments where a decision must be made, a risk accepted, or an action taken.

By anchoring MDM to one or more business processes, data leaders can answer far more practical questions:

• Where does identity ambiguity slow or distort this process?

• Where does uncertainty force manual intervention?

• Where does lack of trust increase cost, risk, or cycle time?

This framing naturally constrains scope. It also makes success visible, and infinitely easier to measure. Improving truth within a single high-value process often delivers more impact than attempting to standardize an entire domain across the enterprise.

Equally important, a process-centric approach reduces governance friction. Governance debates become contextual rather than abstract. Decisions about data definitions, survivorship, and matching logic are grounded in how the data is used, not theoretical completeness. The difference between a process-bounded approach to data governance policies and a data-bounded approach may seem trivial, but it is not. It’s foundational, and more of a process focus is the primary way how CDOs and other data leaders will transition from being a cost center, to a business enabler.

In an AI-driven world - where insights increasingly trigger automated or semi-automated actions - this alignment between truth (and the governance polices needed to enable it) and process is critical. Truth matters most at the moment of action, and processes define where those moments occur.

A unique defining characteristic of MDM is its unique ability, at least in comparison to other data management tools (like Data Quality or Data Integration tools) is its ability to influence how data is not only used in analytical environments, but also in how data is both created and consumed in operational systems – like CRM, ERP, and SCM applications. This is yet another reason why MDM is a critical enabling capability of any modern IT infrastructure – especially in situations where application-bound copilots and agents are interacting directly with data that may never even be replicated into a downstream analytical environment.

Figure 3 – Limiting the Scope of an MDM or Governance Effort

Avoiding Dogma in Implementation

Few areas of data management are as burdened by dogma as MDM.

Over the years, the industry has produced a steady stream of prescriptive debates: registry versus consolidation, operational versus analytical, centralized versus federated. These debates often masquerade as best practices, but in reality they are artifacts of specific contexts, technologies, and tribes.

The problem is not that these distinctions are meaningless. The problem is that they are often treated as decisions that must be made up front, rather than hypotheses to be tested.

In my experience, MDM initiatives that stall early almost always do so because too much energy is spent choosing an implementation philosophy, and not enough energy is spent delivering value. Architecture becomes a proxy for progress.

A more effective approach is to treat MDM as an analytical discipline first, and an operational capability second. Both are equally important, but if your goal is to realize the transformational value of MDM it must first be launched into the organization – and the best way to increase the probability that will happen is through an early embrace of an analytical style of MDM.

Analytical MDM - focused on insight generation rather than enforcement - allows organizations to:

• Test identity resolution logic quickly

• Minimize the complexity of governance policies needed to drive value

• Measure the impact of improved truth on decisions

• Learn where precision matters and where it does not

• Build credibility before introducing operational dependencies

• Avoid complex business process changes

This approach is particularly well-suited to an AI-driven environment, where iteration and learning are already normalized. It mirrors how machine learning models are developed: start with hypotheses, test them against reality, refine based on outcomes.

Operational MDM still matters. But operational rigor should follow demonstrated value, not precede it. When organizations attempt to operationalize MDM too early - before they understand where truth matters - they often hard-code assumptions that later become constraints.

Avoiding dogma does not mean avoiding discipline. It means allowing architecture to emerge from use, rather than imposing it in advance.

The right MDM implementation style is not the one that conforms to a textbook definition. It is the one that produces trustworthy insights quickly while preserving optionality for the future.

Just as important, modern MDM must be enabled by the business, for the business. The era of tightly centralized, top-down control over all representations of master data is effectively over. This is not a failure of discipline, but a reflection of organizational reality. Federated architectures are here to stay. Business units operate with different incentives, different risk tolerances, and different definitions of “fit for purpose.” Expecting a single, centrally imposed view of master data to satisfy every one of those contexts is neither realistic nor desirable.

This does not mean abandoning rigor or accountability. It means recognizing that some degree of local control over highly contextualized views of master data is appropriate. Modern MDM leaders must distinguish between core identity truth - which requires consistency, explainability, and auditability - and contextual representations of that truth, which legitimately vary by use case and business domain. When MDM attempts to over-centralize the latter, it creates friction, drives workarounds, and ultimately undermines trust. When it enables federated consumption while anchoring identity through shared rules and transparent reasoning, it earns adoption. Avoiding dogma here means accepting that federation is not the enemy of truth - it is the environment in which truth must now operate.

Removing Unnecessary Dependencies

Perhaps the most damaging myth surrounding MDM is the belief that organizations must first “get their house in order” before they can begin.

A fully articulated data strategy.
A mature enterprise governance framework.
Clear operating models and role definitions.

These are all worthwhile goals. But when positioned as prerequisites, they become barriers.

In practice, many MDM initiatives die in the planning phase - not because leaders disagree on the importance of truth, but because the list of dependencies grows faster than momentum. Strategy documents are written. Frameworks are debated. Committees are formed. Value is deferred. Often, this is at the hands of consultants given too much free reign on program scope. But other times, its simply a result of well-intentioned program leaders who have been convinced by people insufficiently experienced in real-world data management scenarios that pragmatism equals risk

An approach where perceived procedural dependencies are prioritized ahead of the delivery of value is backwards. .

In successful organizations, governance, strategy, and operating models co-evolve with value delivery. Early MDM successes surface real governance needs. They clarify where rigor is required and where flexibility is acceptable. They turn abstract debates into concrete decisions.

Waiting for perfect alignment before starting MDM is not prudence - it is avoidance. And from the perspective of the current tenure of the program lead, it is often existential.

This is especially problematic in an AI context, where business leaders are accustomed to rapid experimentation and visible progress. Data teams that insist on long lead times and extensive prerequisites risk being bypassed entirely, or caught in the next round of corporate downsizing.

Value delivery must be the North Star. Everything else - governance, standards, strategy - should orbit around it, informed by real outcomes rather than theoretical ideals.

Part IV: Non-Negotiables for Modern MDM

Explainability and Contextual Truth

Any serious discussion of modern MDM must begin by dismantling the myth of a single, universal version of the truth.

Truth is contextual.

The “correct” representation of an entity depends on how - and why - it is being used. The version of customer truth required for regulatory reporting may differ from the version needed for marketing personalization. Both can be accurate. Both can be defensible. And both can be true within their respective contexts.

Historically, many MDM programs attempted to force convergence on a single canonical truth. In practice, this often resulted in compromises that satisfied no one, or in shadow versions of truth emerging downstream.

Modern MDM must explicitly support multiple versions of truth, each governed by context, purpose, and risk tolerance.

Figure 4 below shows that the concept of ‘truth’ exists in at least three different ‘levels’ in an organization. These levels are functional (or ‘local’), cross functional (or ‘regional’), and enterprise wide (or, ‘global’). Each context can have its own definition of truth, and all three can be correct – depending on the use case. A more modern approach to MDM recognizes that all three (or more) contextually-bound notions of truth can exist at the same time.

This diagram recognizes something that data practitioners have known for a long time – the needs of one individual department or function are often not the same as the needs that exist on an enterprise-wide level. This tension between local / decentralized and global / centralized has existed for as long as we’ve managed data, and in the real of analytics, the latter tends to rue the day. A more recent push towards more federated and decentralized architecture, fueled in part by an increasing focus on GenAI, is driving a need for a more pragmatic and flexible approach to both master data management, and data governance.

Figure 4 – Three Levels of Governance in all Organizations

In a world dominated by AI, what matters most for MDM is not uniformity, but clarity and explainability.

Explainability is what transforms truth from an assertion into an asset. In regulated and high-risk environments, it is not enough to know that two records were linked; organizations must be able to explain why. Black-box identity resolution may be statistically impressive, but it is operationally dangerous.

This is where MDM provides something probabilistic AI systems cannot: deterministic, inspectable reasoning about identity. Rather than competing with AI, MDM establishes the guardrails within which AI can operate safely.

As AI systems increasingly participate in decisions that affect customers, finances, and compliance, explainability becomes non-negotiable. I’ve learned this lesson the hard way.

MDM as Connective Tissue into Knowledge Management

Meaning increasingly lives outside structured systems.

Contracts, policies, emails, call transcripts, documents, and narratives contain enormous amounts of contextual information that structured data alone cannot express. Knowledge graphs and ontologies are essential for extracting and organizing that meaning.

But meaning without anchoring remains fragile.

MDM’s evolving role is to serve as connective tissue between:

• Structured representations of entities

• Unstructured expressions of meaning

• Semantic frameworks that bind them together

In this sense, MDM is no longer merely a system of record. It becomes a system of reference and reconciliation, ensuring that entities described across different modalities - tables, text, graphs - are consistently understood and represented.

This evolution does not diminish the importance of MDM; it expands it. Truth becomes the stabilizing force that allows meaning to scale across systems and contexts.

Without that stabilizing force, knowledge management efforts risk drifting into semantic sophistication without operational trust. This risk will be completely avoided by forward-leaning CDOs who take the steps to better integrate their MDM programs – and all their data management initiatives – to any existing focus on knowledge management within their organizations.

Figure 5 shows the critical role that MDM plays in establishing a foundational layer of ‘truth’ that spans across all of data, information, and knowledge management.

Figure 5 – A Next Generation Semantic Layer

Pragmatic Data Governance as Enablement

No MDM initiative exists independently of data governance. But governance itself must evolve.

For too long, governance has been framed primarily as a mechanism of control. Policies, approvals, and enforcement have taken precedence over enablement. In many organizations, governance has become synonymous with friction.

This is not a failure of intent. It is a failure of orientation.

Effective data governance exists to reduce ambiguity, accelerate decision-making, and enable safe use of data in pursuit of business outcomes. Control is a byproduct, not the objective. The most

A pragmatic governance approach:

• Aligns rigor with risk

• Evolves alongside value delivery

• Emphasizes stewardship as influence, not ownership

• Prioritizes clarity over completeness

MDM provides a natural focal point for this evolution. Because it sits at the intersection of identity, trust, and use, MDM surfaces governance needs organically. Early successes reveal where rules are necessary and where flexibility is acceptable.

Governance that emerges from doing is far more resilient than governance imposed in advance.

In an AI-driven world, where speed and experimentation are the norm, governance that cannot keep pace with value creation will be ignored. Governance that enables value will be embraced.

Conclusion: Truth, Meaning, and Trust

Meaning without truth is confusion.
Truth without meaning is limited.

In the age of AI, trust emerges only when the two are intentionally combined. Master data management - pragmatic, explainable, and outcome-driven - remains essential to that convergence.

MDM may never be glamorous. But in a world where AI increasingly shapes decisions, truth is not optional.

It is foundational.

Thank you for reading this article and I sincerely hope you got value from it. If you are interested in learning more about MDM, then:

- Please connect with me on Linkedin: (19) Malcolm Hawker | LinkedIn

- Please check out the wealth of resources available on the Profisee website related to MDM best practices: www.profisee.com

- Please check out my podcast, CDO Matters: https://profisee.com/podcast/

- And last, but certainly not least, check out my latest book – The Data Hero Playbook. It’s a detailed exploration of the mindsets that data leaders need in order to succeed in an era of AI: https://a.co/d/6qxRTW5

And finally – a huge thanks to the legendary Bill Inmon for curating this content. I am humbled beyond words that he approached me to make this contribution.

Malcolm Hawker - February, 2026

Malcolm Hawker is the CDO of Profisee and is a thought leader in the fields of Data Strategy, Master Data Management (MDM), and Data Governance. As a former Gartner analyst, Malcolm has authored industry-defining research and has consulted some of the largest businesses in the world on their enterprise data and analytics strategies. Having served as a Chief Product Officer, Head of IT, and strategic business consultant, Malcolm is an industry leader with over 25 years’ experience at the forefront of data-enabled business transformations. Malcolm is a frequent public speaker on data and analytics best practices, and he cherishes the opportunity to share practical and actionable insights on how companies can achieve their strategic imperatives by improving their approach to data management. He is the author of the Wiley book ‘The Data Hero Playbook’, which details the critical role that a growth mindset plays in helping companies to realize the transformative value of data. When not sharing his passion for data or recording episodes of the CDO Matters podcast, Malcolm is an avid hobbyist landscape photographer and lives with his wife and two dogs in a small beach town in Florida.

By Jeff Wilts

At some point, someone in your organization will stand up in a meeting and say something like:

“We’re moving off Teradata. Databricks is modern. It’s cheaper. It’s faster. And we can finally stop worrying about all that old-school data warehouse discipline.”

This usually sounds very confident. It is also usually partially wrong.

What follows is not a story about technology failure. Databricks is an impressive platform. Teradata is a mature one. Both do exactly what they are designed to do. The failures happen because people assume they are philosophically the same thing. They are not.

Teradata is a full-featured enterprise data warehouse. Databricks is a unified data platform that can behave like a data warehouse — if you deliberately make it do so. That “if” is where most migrations can go off the rails.

Databases encode assumptions. When you change platforms without understanding those assumptions, you don’t get a clean modernization. You get what I like to call a Lakehouse Liability: something shiny, expensive, and possibly oddly untrustworthy.

Let’s talk about why.

Built-In Discipline vs. Earned Discipline

Teradata enforces a lot of discipline for you, whether you like it or not.

Declare a primary key? Uniqueness is enforced.
Define a SET table? Duplicates are rejected.
Create triggers? They fire exactly when rows change.
Use views? You can insert, update, and layer them in surprisingly powerful ways.

Databricks does none of this by default.

Primary keys are informational.
Uniqueness is a suggestion.
Triggers exist, but not in the way Teradata people expect.
Views are read-only abstractions.

This is not a flaw. It is a design choice.

Databricks assumes you will manage correctness through pipelines, patterns, and governance. Teradata assumes correctness is the database’s job. If you don’t notice this difference early, you will notice it later — usually during a reconciliation meeting where nobody is smiling. This is earned discipline. Not optional discipline.

Example: Primary Keys

Teradata

CREATE TABLE customer (

customer_id INTEGER PRIMARY KEY,

name VARCHAR(100) );

Duplicate insert:

INSERT INTO customer VALUES (1, ‘Alice’);

INSERT INTO customer VALUES (1, ‘Bob’);

Result: Error. Constraint enforced.

Databricks

CREATE TABLE customer (

customer_id INT,

name STRING,

CONSTRAINT pk_customer PRIMARY KEY (customer_id) );

Same duplicate insert:

INSERT INTO customer VALUES (1, ‘Alice’);

INSERT INTO customer VALUES (1, ‘Bob’);

Result: Both rows succeed.

The constraint is informational. It helps tools. It does not protect your data.

So you enforce uniqueness in pipelines:

MERGE INTO customer t

USING staging_customer s

ON t.customer_id = s.customer_id

WHEN MATCHED THEN UPDATE SET *

WHEN NOT MATCHED THEN INSERT *

Discipline moves from engine → pipeline.

If you don’t redesign for that, duplicates will accumulate quietly until someone runs a reconciliation.

Example: Duplicate Rejection

Teradata SET tables reject duplicates automatically.

CREATE SET TABLE orders (

order_id INTEGER,

amount DECIMAL(10,2) );

Databricks equivalent behavior must be simulated:

CREATE OR REPLACE TABLE orders_clean AS

SELECT DISTINCT *

FROM staging_orders;

Or enforced in streaming:

df.dropDuplicates([”order_id”])

Collation: The Quiet Way to Break Trust

Collation is one of those topics everyone skips because it sounds boring.

Until joins start dropping rows.

Collation defines how text is compared and sorted: case sensitivity, accent handling, language rules. Teradata gives you extensive control over this at the database, user, and column level. Databricks largely relies on UTF-8 binary comparisons: fast, simple, and unforgiving.

Teradata often behaves case-insensitively. Databricks often does not.

If you don’t explicitly decide how text comparisons should behave, you will still get a decision — just not one you made consciously. The result is dashboards that looks like it “randomly” change, distinct counts that drift, and analysts who start adding workarounds downstream.

Once trust is lost, performance improvements stop mattering.

Example: Case Sensitivity

Teradata

SELECT *

FROM customers

WHERE name = ‘alice’;

May match: Alice, ALICE, alice

Databricks default

SELECT *

FROM customers

WHERE name = ‘alice’;

Matches only exact case.

So this join:

SELECT *

FROM a

JOIN b

ON a.name = b.name;

can silently lose rows after migration.

Mitigation pattern:

SELECT *

FROM a

JOIN b

ON LOWER(a.name) = LOWER(b.name);

But now you’ve introduced:

· CPU overhead

· semantic drift

· inconsistent behavior across teams

The correct solution is not sprinkling LOWER() everywhere. The correct solution is a collation strategy and documented comparison rules.

Data Types: “It Loaded” Is Not the Same as “It’s Right”

Teradata’s type system is strict, expressive, and full of edge cases people forgot they were relying on.

BYTEINT.
INTERVAL.
High-precision DECIMAL.
TIMESTAMP WITH TIME ZONE.

Databricks supports many of these concepts, but not always in the same way, and not always as first-class citizens. Types get promoted. Precision limits matter. STRING becomes the universal escape hatch.

The classic failure looks like this:

You migrate a fact table.
The data loads cleanly.
The rollups run.
The totals are off by a very small amount.

That “very small amount” will cost you in credibility.

Floating-point math behaves differently. Implicit casts behave differently. Rounding behaves differently. None of this is obvious until someone compares old numbers to new ones and asks the most dangerous question in analytics:

“Which one is correct?”

If you don’t want that conversation, build a type-mapping matrix before you migrate and test the ugly cases deliberately.

Example: DECIMAL Precision Drift

Teradata fact table:

amount DECIMAL(38,10)

Spark aggregation:

SELECT SUM(amount)

FROM fact_sales;

If implicit casting occurs:

• precision loss

• floating behavior differences

• rounding changes

A 0.01% drift becomes a credibility problem.

Spark Mitigation:

SELECT CAST(SUM(amount) AS DECIMAL(38,10))

FROM fact_sales;

And validation:

SELECT old_sum, new_sum, old_sum - new_sum AS delta

FROM reconciliation_table;

You don’t test averages.

You test pathological rows.

Golden rows:

• max precision

• negative values

• null combinations

• extreme timestamps

Example INTERVAL

Teradata:

INTERVAL ‘01:30:00’ HOUR TO SECOND

Databricks workaround:

duration_seconds BIGINT

Or ISO string:

duration STRING -- “PT1H30M”

The Data Model: The Discipline You Have to Rebuild on Purpose

Teradata explicitly forces you to think in terms of a data model. Not because it’s philosophical. Because if you don’t, things break. Keys matter, relationships matter,
table design has consequences.

When teams move to Databricks, one of the first invisible casualties can be the data model — not because anyone deletes it, but because the platform no longer insists on it. The engine will happily accept loosely related tables, duplicate identifiers, drifting semantics, and “temporary” structures that quietly become permanent.

Nothing may explode immediately. That’s what makes it dangerous. A data model is not just a diagram someone made once for documentation. It is the explicit statement of what the business believes is true: What is a customer? What makes an order unique? Can an account exist without an owner? What does “active” mean?

Teradata encodes these beliefs structurally. Databricks assumes you will encode them socially — through conventions, pipelines, and shared understanding.

That assumption only works if you are deliberate.

Example: Identity Drift

Imagine a customer table without enforced uniqueness:

customer_id | email
1 | alice@example.com
1 | alice@workmail.com

Both rows load. Both rows propagate. Both rows show up in analytics.

Now every downstream metric involving “customer count” becomes an opinion instead of a fact. The data model is what prevents identity from becoming negotiable.

In a lakehouse environment, the model becomes a contract enforced by process instead of database constraints:

· MERGE rules define identity

· promotion pipelines validate uniqueness

· quality checks reject semantic violations

· published tables represent certified truth

If you don’t formalize that discipline, the platform will not do it for you.

Bronze/Silver/Gold architectures are not just performance layers. They are modeling layers:

Bronze = raw events
Silver = conformed entities
Gold = business meaning

The data model is what turns data into entities and entities into decisions.

Without it, you don’t have a lakehouse. You have a very fast junk drawer.

Ontology: When the Problem Isn’t Structure — It’s Meaning

A data model answers: How is the data structured? An ontology answers a harder question: What does the data mean? Two teams can share the same schema and still disagree about reality.

That is not a technical problem. It is a semantic one. An ontology is a shared vocabulary that defines concepts, relationships, and intent. It sits above the schema and prevents meaning from drifting as the system grows.

In small systems, ontology lives in people’s heads. In large systems, that stops scaling.

Example: The Word “Customer”

· Marketing customer: someone who signed up

· Billing customer: someone who paid

· Support customer: someone with an open ticket

· Data warehouse customer:
whoever the ETL decided last quarter

Same table name. Different ontology. The schema didn’t fail.
The shared meaning did.

Lakehouse platforms amplify this problem because they remove central bottlenecks. More teams can publish data faster. That’s good for velocity. It’s terrible for semantic consistency if you don’t define a common language.

An ontology gives you:

· shared definitions

· stable business concepts

· explicit relationships

· machine-readable meaning

This matters for more than humans. Downstream systems increasingly rely on semantic clarity:

· BI layers

· metrics stores

· data catalogs

· AI models

· governance tooling

If your ontology is implicit, every tool invents its own. That’s how organizations end up arguing over dashboards instead of decisions.

Practical Ontology Pattern

· Define canonical entities: Customer, Order, Product, Subscription

· Define their relationships: Customer places Order, Order contains Product
Customer owns Subscription,

· Align schemas to those concepts instead of letting each pipeline improvise terminology.

Unfortunately Databricks Unity Catalog does not have native capabilities for capture and managing an ontology layer. It robustly manages technical frameworks for data governance (access, control, auditing, lineage, physical schema) but you will need a different solution for business context.

You are not adding bureaucracy. You are preventing semantic entropy. The bigger and more flexible the platform becomes, the more important ontology becomes. Not because technology demands it — but because humans do.

Databricks removes structural guardrails. Ontology restores conceptual ones. Without it, the system still runs. It just slowly stops agreeing with itself.

Transactions: ACID Is Not a Yes/No Question

Teradata users are used to wrapping a sequence of statements in a transaction and expecting all-or-nothing behavior.

Databricks gives you ACID guarantees — but typically at the level of a single operation.

That distinction matters when your pipeline does something like:

• update a dimension

• insert facts

• write audit rows

• publish a completion flag

If step three fails, you may have just published a half-truth.

The solution is not to complain that Databricks is “missing” something. The solution is to design for atomicity explicitly: MERGE patterns, publish-from-staging approaches, and idempotent pipelines that can be safely re-run.

Safer Pattern: Staging Publish in Databricks

CREATE TABLE fact_staging AS

SELECT ...

Validate:

SELECT COUNT(*) FROM fact_staging;

Publish atomically:

REPLACE TABLE fact_production AS

SELECT * FROM fact_staging;

Or Delta version swap.

Time Zones: Decide Early or Pay Forever

Time zones are exhausting. Everyone knows this. That’s why teams postpone decisions about them.

Then daylight savings hits.

Teradata supports time-zone-aware timestamps natively. Spark historically treats timestamps as instants interpreted through session settings unless you deliberately use non-time-zone types.

You must decide what your timestamps mean.

Are they instants in time? Store UTC.
Are they wall-clock business times? Store local time explicitly.
Are they mixed? Store both the instant and the original offset.

Assuming “timestamp is timestamp” guarantees pain later. This is not a technical nuance. It is a semantic contract.

Best Practice Pattern

Store event instant:

event_ts_utc TIMESTAMP

Store original zone:

event_tz STRING

Display conversion:

SELECT from_utc_timestamp(event_ts_utc, event_tz)

What Happens If You Don’t Decide

• DST (Daylight Saving Time) shifts

• audit drift

• inconsistent reporting

• analysts invent fixes

SQL Differences: The Death of a Thousand Paper Cuts

Most Teradata SQL ports cleanly. The rest may make you question your life choices.

Volatile tables.
Macros.
Triggers.
Update-with-join patterns.
Stored procedures.

Each one is manageable. Collectively, they demand a systematic approach. Decide early how procedural logic will be handled: Databricks SQL scripting, notebooks, dbt, or something else. Inconsistency here is how maintenance costs quietly explode.

Example: Update Join Rewrite

Teradata:

UPDATE target t

FROM source s

SET t.value = s.value

WHERE t.id = s.id;

Databricks:

MERGE INTO target t

USING source s

ON t.id = s.id

WHEN MATCHED THEN UPDATE SET value = s.value;

Example: Volatile Tables

Teradata:

CREATE VOLATILE TABLE temp AS (...);

Databricks:

CREATE OR REPLACE TEMP VIEW temp AS (...);

Or:

df.createOrReplaceTempView(”temp”)

Don’t Just Rebuild Teradata on Databricks

One of the most common mistakes is recreating the Teradata warehouse exactly as it was, just on new infrastructure.

Same schemas.
Same layering.
Same mental model.

You can do this. You just won’t get much benefit from it. Databricks rewards discipline differently: layered Bronze/Silver/Gold models, explicit publish patterns, governance through Unity Catalog, and data quality checks as part of promotion — not as an afterthought.

The lakehouse does not eliminate the need for discipline. It removes the guardrails that forced it.

Final Thought

Teradata spent decades teaching organizations hard lessons:

· Typing matters.

· Semantics matter.

· Governance matters.

· “It runs” is not the same as “it’s right.”

Databricks does not invalidate those lessons. It simply gives you a much larger platform on which to relearn them — faster, louder, and at cloud scale — if you are not careful.

The devil is not in the details of the technology. The devil is in the assumptions and decisions you didn’t realize you were making.

Jeff Wilts is a Chief Architect and senior technology executive with 25+ years of hands-on experience designing, modernizing, and scaling enterprise data, AI, and platform architectures across FinTech, IoT, retail, and large digital marketplaces.

Currently a VP and Chief Architect in the Data & AI Practice at Ness, he is engaged in one of the world’s largest and most complex Teradata to Databricks modernization programs. Jeff has deep expertise in distributed systems, cloud-native architectures, data engineering, M&A carve out integrations, and regulated data environments, and has built and led global teams delivering high-availability, mission-critical systems. His work bridges architecture strategy and execution, with a strong emphasis on operationalizing data, AI/ML, and automation to drive reliability, performance, and measurable business outcomes.

By W H Inmon and Jessica Talisman

Once there was only structured data. Transactions ran in banks and airlines reservations systems. Data bases kept track of inventory. The world of business was running just fine based on the transaction based, structured data that served the environment.

Then one day someone noticed that there was data that was not structured in the corporation. That “foreign” data was not being generated by the transactions of the corporation.

The “foreign” data that had been discovered was text. Text was everywhere in the corporation. On the Internet. On emails. In spreadsheets. In conversations. In print. Everywhere.

Furthermore, there was valuable data wrapped up in the form of text. The problem was that accessing, getting to the text, and making sense of the text and the data that it held was a challenge. Text was random – there is no order to text. People can say or write anything they want. There simply is no rhythm to creating and collection of text. Text requires context in order to be meaningful. And context is seldom obvious when wrapped in the form of text. But context was an absolute necessity when dealing with text. Text can come in many forms – voice, print, spreadsheets, email and so forth.

In a word, reading text and extracting meaningful data from text was and is a daunting task.

ONTOLOGY

To this end, there is a technology/discipline that greatly abets the challenges of extracting meaningful data from text. That technology/discipline is called an ontology

.So what in the world is an ontology? How could you tell an ontology from your Aunt Alice? From your cocker spaniel? From your favorite football team?

DEFINING CHARACTERISTICS

What are the defining characteristics and structures that are found in an ontology?

A simplified characterization of an ontology is that an ontology is a carefully vetted vocabulary designed to unravel and classify a body of raw text. The ontology contains what can be called a series or collections of taxonomies.

TAXONOMY

And what exactly is a taxonomy? A taxonomy is a vocabulary designed to classify something. The something that can be classified by a taxonomy can be anything – a tangible object, a concept, a discipline, a football team, a method of teaching swimming, a car, and so forth. Anything that can be classified can have a taxonomy built for it. And anything can be classified.

As such there are an infinite number of possibilities for the creation of a taxonomy.

The defining characteristic of a taxonomy is that each word in the taxonomy has a similar relationship to the object that is being classified as all of the other elements contained in the taxonomy. The relationship of each element found in the taxonomy may not be exactly the same as the relationship of every other element found in the taxonomy, but the relationship of the element to its classification is at least very similar.

SOME EXAMPLES

As some examples of taxonomies, consider the following –

A classification for a car can be a Porsche, Ford or Honda. The classification of a sport may be golf, football, or tennis. The classification for food might be a steak, potatoes, or tomatoes.

Each of the elements found in the classification that shapes the taxonomy has the same basic relationship to the classifying object as each other element in a taxonomy.

INTERRELATIONSHIPS

The elements of a taxonomy may or may not have a relationship to other elements in other taxonomies in the same ontology. For example, suppose there were three taxonomies for country, state and city. The states Texas, New Mexico and others all relate to the country USA. And El Paso is found in Texas, Santa Fe is found in New Mexico, and Denver is found in Colorado.

These are examples of a relationship between taxonomies.

However, notice that Atlanta, Miami, and Nashville do not relate to the state taxonomy because the states that those cities belong to are not in the state taxonomy.

And China, Russia, and Brazil have no cities in the city taxonomy.

The relationship between elements of a taxonomy to the elements in another taxonomy then are casual. The relationship between elements in a taxonomy may or may not exist.

And this is a perfectly normal state.

DEFINING A TAXONOMY

So what is not a taxonomy? Consider the vocabulary seen in the following figure. The vocabulary looks like a taxonomy. The heading is country and England, France and the USA are found in the vocabulary. But there are elements in the vocabulary that are decidedly not countries. There is Chablis, egg whites and Babe Ruth that are found in the vocabulary. It is unclear if there is any meaning at all to the grouping of the words found in the vocabulary that has been shown.

The vocabulary shown is definitely not a taxonomy.

There are then different types of collections of vocabularies. The following vocabulary is just a list of words.

ACRONYMS

Taxonomies can include more than just words. Taxonomies can include acronyms as well as words. Acronyms are widely used, especially in medicine, legal writings and military writings. However, in addition to those commonly found places, acronyms are used everywhere.

Taxonomies need to include acronyms as well as words in order to be able to make sense of the text that is being examined.

MISSPELLINGS AND SLANG

In addition, taxonomies can include common misspellings of words. When a misspelling is common, the misspelled version of the word can be interpreted as the properly spelled version of the word.

In addition, slang – ain’t, rizz, cheugy, sus, and others – can be placed in a taxonomy

.STEMMING

Another structure that can be placed in a taxonomy are word families. A word family is a form of what is termed “‘stemming”. A word stem is a word that can be stated in more than one form.

The use of stemmed words depends on the existence and characteristics of the processor that will do the transformation of raw text into a data base. Different processors have different properties and characteristics. In some case the processor does its own stemming. In other cases, the processor only uses the literal embodiment of the word as found in the taxonomy.

WORD CASE, LANGUAGES

And there are other considerations in the making of the taxonomy, such as the considerations of case of the words – upper case/lower case – different languages, and so forth.

Cross language interpretation of taxonomies can be very complicated. As words are translated into different languages, nuances can and often do occur. When language translation of a taxonomy is done, it is a really good idea to have a knowledgeable native speaker review how the translation has been done.

MULTIPLE LEVELS OF CLASSIFICATION

Taxonomies always have some one unifying category to draw the elements in the taxonomy together. However, taxonomies may have multiple levels of categories that are contained inside the taxonomy as well.

In the following example, sports have the subclassification of team sports and individual sports. Then underneath team sports are found football, basketball and baseball. And in the same manner, food is subclassified by meat, vegetable, and dessert.

THE FOCUS OF THE ONTOLOGY

In its most effective form, ontologies have a general focus. Typically, some ontologies may focus on –

Medicine

Banking

Airlines

Insurance

And so forth

.It is noteworthy that the taxonomies that reside in an ontology contain only taxonomies that relate to the focus. For example, you would expect an ontology that was for airlines to include such taxonomies for –

Reservation systems

Baggage handling

Frequent flyer programs

Flight scheduling

And so forth

Unless there were really unusual circumstances, you would not expect to find airlines needing taxonomies for –

Deep sea exploration

Building construction

Bulldog rodeo riding

And so forth.

The focus of the ontology has an influence on the contents of the taxonomy that is placed inside it.

For example, suppose the ontology was focused on the North American landscape. This would include mountain ranges, deserts, seashore, trees, rivers, and so forth.

Now suppose that the taxonomy for rivers was chosen to be part of the ontology. The ontology for rivers would contain only rivers found in North America. The taxonomy would not contain such rivers as the Amazon or the Nile, as they exist well outside of North America.

GENERIC TAXONOMIES

The taxonomies that reside inside the ontology may or may not have relationships to each other. In the example shown for a bank, the taxonomies for credit card, savings and loans have a relationship to each other. But the taxonomies for security and accounting exist independently.

Together the taxonomies collectively describe the banking environment. But portions of the ontology for the bank exist in a very loosely connected fashion.

Because there are independent disciplines that can exist inside an ontology, there are what can be termed “generic’ taxonomies. A generic taxonomy is a taxonomy that can be used in many places.

For example, the GAAP rules which constitute much of the accounting taxonomy are universal. The GAAP rules and their subsequent text that describes them are the same regardless of whether you are an airline, a bank, an insurance company, or a telecommunications company.

When building the ontology for banking, for example, the analyst can simply apply the accounting taxonomy that has been created for insurance, or airlines.

MAINTAINING THE ONTOLOGY

A final and important consideration for the building and usage of an ontology is the consideration of the maintenance of the ontology. The ontology and its taxonomies will change over time, as the business changes.

As those basic business changes occur, the changes will need to be reflected in the taxonomies that represent them.

The good news is that most businesses only change marginally over time. It would be very unusual for Ford Motor Company to declare that it was going into the making of ice cream. Therefore, the biggest challenge in building and using a taxonomy lies in the first iteration of the building of the taxonomy.

Once the first iteration is built, all subsequent updates to the ontology are marginal.

THE WORK BEHIND THE WORDS

The maintenance of an ontology, points to something deeper about what makes an ontology work in the first place. An ontology is not simply a collection of words arranged in categories. An ontology represents decisions—thousands of small, careful decisions about how concepts relate to one another, where boundaries should be drawn, and what matters enough to include.

Consider the banking ontology mentioned earlier. Someone had to decide whether “mortgage” belongs in the loans taxonomy or deserves its own taxonomy. Someone had to determine whether “refinancing” is a type of loan or an action performed on a loan. These are not trivial questions. The answers shape how the ontology will perform when it encounters raw text about banking.

This is why building an ontology is a human activity, not a mechanical one. The ontology builder must understand the words of a domain AND the conceptual architecture underneath those words—the way practitioners in that domain actually think about their work.

SCOPE AND GRANULARITY

Every ontology must answer two fundamental questions: How wide should we cast the net? And how fine should we cut the cloth?

The first question concerns scope. An ontology for medical text could, in theory, include every term ever used in medicine. But such an ontology would be unwieldy and expensive to maintain. More importantly, it would likely, perform poorly. An ontology designed to extract meaning from radiology reports has different requirements than one designed for clinical trial documentation or insurance claims processing. Clinical trials, radiology and insurance claims are subdomains of medicine, and have different characteristics that define each as unique.

The most effective ontologies are purpose-built. They serve a specific need within a specific context. The ontology builder must resist the temptation to capture everything and instead focus on capturing what matters for the task at hand.

The second question concerns granularity. This question pertains to both the taxonomy and ontology. How detailed should each taxonomy be? Should the taxonomy for “medical procedures” include broad categories like “surgery” and “diagnostic testing”? Or should it drill down to specific procedures like “laparoscopic cholecystectomy” and “computed tomography angiography”?

And how about granularity as it pertains to an ontology? How detailed and nuanced must properties and relationships be? It is very easy to paint oneself into a corner with descriptive hyper granularity. Is it necessary to say Is it necessary to say that laparoscopicCholecystectomy requiresInstrumentInsertionThroughIncision NotTheSameIncisionAsTheOneUsedForTheCamera

IsAdjacentToIt

InAMannerDeterminedBy

TheSurgeonsDominantHand

and some TrocarPort?

Or

Is it necessary to assert that laparoscopicCholecystectomy

isPerformedIn

ARoomWhereAtLeastOnePersonPresent

HasSecretlyWonderedWhether

TheyLeftTheStoveOn value true?

Determining the scope or coverage of a taxonomy and ontology are both decisions that must be analyzed and decided upon by humans, as ultimately, these decisions construct the profile of the organization, workflows and the things human workers care about and need, to be successful.

The answer to the granularity question depends entirely on what you intend to do with the extracted data. If you need to route documents to the correct department, broad categories may suffice. If you need to identify patterns in treatment outcomes, fine-grained distinctions become essential.

THE ROLE OF HIERARCHY

We have seen that taxonomies can contain multiple levels of classification—sports divided into team sports and individual sports, food divided into meats, vegetables, and desserts. These hierarchical structures exist as organizational vehicles while also carrying semantic weight.

A taxonomy category and its sub category has a relationship that asserts something meaningful. It says that football is a kind of team sport, that a steak is a kind of meat. This “is-a” relationship allows the taxonomy and its associated ontology to make inferences. If you know that someone ate a steak, you can infer that they ate meat. If you know that someone played basketball, you can infer that they participated in a team sport.

These inferences may seem obvious to a human reader, but they are precisely what makes an ontology useful for processing text at scale. The ontology encodes common-sense relationships that allow a system to understand that a document mentioning “the patient underwent a coronary bypass” is discussing cardiac surgery, even if the phrase “cardiac surgery” never appears.

SYNONYMS AND VARIANTS

Real text is messy. The same concept can be expressed in dozens of different ways. A document might refer to a “heart attack,” a “myocardial infarction,” an “MI,” or simply state that “the patient’s heart stopped receiving adequate blood flow.”

A well-constructed ontology accounts for this variation. Each concept in the taxonomy can carry with it a collection of synonyms, abbreviations, and alternative phrasings—all mapped to a single canonical term. When the ontology encounters any of these variants in raw text, it recognizes them as the same underlying concept.

This normalization is one of the most valuable functions an ontology performs. It transforms the chaos of natural language—where the same thing can be said in countless ways—into consistent, structured data that can be analyzed and compared.

RELATIONSHIPS BEYOND HIERARCHY

Not all relationships between concepts fit neatly into a hierarchical structure. Some relationships run sideways rather than up and down.

Consider the relationship between a disease and its treatment. Diabetes is not a “kind of” insulin therapy, nor is insulin therapy a “kind of” diabetes. Yet these two concepts are meaningfully connected. An ontology can capture this by defining relationship types beyond the simple parent-child hierarchy. This is where ontologies do their work.

A disease “is treated by” a therapy. A drug “interacts with” another drug. A symptom “indicates” a condition. A procedure “requires” equipment. These associative relationships allow the ontology to represent the rich web of connections that characterize any complex domain. These are not taxonomy relationships — we model these descriptive relationships using ontologies.

When such relationships are explicitly defined, the ontology becomes capable of answering questions that go beyond simple classification. It can tell you a document mentions diabetes, and that the document discusses a treatment approach for diabetes using a particular class of medications.

THE LIVING DOCUMENT

An ontology is never truly finished. Language evolves. Domains change. New concepts emerge while old ones fade from use.

In medicine, new diseases are identified, new treatments are developed, and new terminology enters the vocabulary of practitioners. An ontology that was comprehensive five years ago may now be missing critical concepts. An ontology that was carefully balanced may now be weighted toward obsolete distinctions.

This is why the organizations that maintain major ontologies treat them as living documents, subject to continuous review and revision. New terms and concepts are proposed, evaluated, and either incorporated or rejected. Existing terms and concepts are periodically assessed for continued relevance. Relationships are refined as understanding of the domain deepens.

For an organization building its own ontology, this ongoing maintenance is a core commitment. The ontology must have a steward, someone responsible for keeping it aligned with the reality it represents. And a business must invest in these knowledge assets and cherish them for the insights and value they deliver.

THE BRIDGE BETWEEN WORLDS

We began by observing that text presents a fundamental challenge: it is unstructured, context-dependent, and infinitely variable. The ontology serves as a bridge between this world of raw text and the world of structured data where analysis and computation become possible.

But the ontology does something more than simply extract data points from text. It imposes a conceptual framework—a way of seeing and organizing the domain. When you build an ontology for banking, you are articulating a model of how banking works, what entities matter, and how those entities relate to one another.

This is the power and the responsibility of ontology work. The ontology shapes what can be seen and what remains invisible. Concepts that exist in the ontology can be found, counted and analyzed. Concepts that are absent from the ontology will pass through unrecognized, no matter how prominently they appear in the text.

The ontologist is not a technician cataloging words . The ontology builder is making choices about what matters—choices that will ripple through every analysis that depends on the ontology upstream, downstream and in between.

ABOUT THE AUTHORS

W.H. Inmon is widely recognized as the father of the data warehouse and has spent decades helping organizations structure and leverage their data assets for analytics and decision-making. Bill has sold over 1,500,000 books in his life

Jessica Talisman is a semantic infrastructure consultant and knowledge organization expert specializing in ontology development, controlled vocabularies, and enterprise semantic architectures that bridge library science principles with modern AI and knowledge management systems. Jessica created the Ontology Pipeline, a framework for building semantic knowledge infrastructures and ontologies.

Some of Bill’s latest books include –

DATA ARCHITECTURE – BUILDING THE FOUNDATION, with Dave Rapien, Technics Publications.

STONE TO SILICON – A HISTORY OF TECHNOLOGY AND COMPUTERS, with Dr Roger Whatley, Technics Publications.

Articles and presentations by Jessica can be found on Linkedin, Substack and Youtube.

By W H Inmon

How is it that humans come to grips with complexity? The classical and time honored approach to dealing with complexity is to –

Employ abstraction to determine the shape and structure of the object being analyzed

Look across the vistas of the complex landscape

Break a complex problem down into its component pieces

As an example of dealing with complexity, consider a ship on the sea. There are many factors that affect the safety of the ship and the voyage it is going on. There are only landmarks on the sea when the ship nears the coast. In the middle of the ocean there are no hard and fast landmarks. So how does the ship know what the proper setting of the rudder is to be? The captain of the ship relies on his/her compass

.COMPLEX WORLDS

There is hardly a more complex world than today’s IT environment. The IT environment has been evolving for half a century or more now and is going to continue to evolve. What was once fairly straightforward is anything but straightforward today. And the IT environment is full of seemingly unrelated and complex topics –

Legacy systems

AI

Venture capitalists

Chip vendors

Vendors that mislead their customers

Dashboards

And a host of other disciplines and technologies.

And IT is hardly the only world of complexity that there is. In another venue there is the world of text – unstructured information. There are news reports. There are newspapers. There are government reports. There are comic books. There is the Internet. And nobody seems to agree with anyone else.

If there ever was a world that was free form and complex, it would be the world of unstructured data and text.

THE STRUCTURED DATA MODEL

So what kind of compasses exist for these very different worlds? What is the compass for IT?

For IT there is the compass of the data model. The data model holds and structures the metadata that describes and defines the IT environment.

A classical data model exists at different levels of abstraction. The classical data model has a high level of abstraction called the ERD – entity relationship diagram. Each of the entities described in the ERD has a lower level of detail for each of the entities. This level is called the ‘dis” or data item set. In the mid level of the data model – the dis – data item set – level of the model - are found keys, attributes and relationships.

At the lowest level of the structured data model is found the place where the model is described to the dbms – data base management system. This is called the ddl – data definition language. Each of the levels of the data model are related to each other, in much the same way as the globe of the world relates to Texas, and Texas relates to the city of Dallas.

When the structured data model is created and maintained properly, it is a guideline to the labyrinth of IT.

THE TEXT MODEL

The world of unstructured data has its own model – a text model. The text model is the compass for unstructured data and text. The first issue of the text model is – what level of granularity does the text model exist at?

There are at least two levels of granularity of the text model – the chunk level and the word level. A chunk is a collection of text that contains words that form some kind of thought that has been expressed. The word level is the lowest level of granularity of the unstructured environment. Each of these levels of granularity have their own advantages and disadvantages.

Unstructured data at the chunk level is good for understanding the reliability and the veracity of the thought that is being expressed. Unstructured data at the word level expresses text and the lowest level of granularity. Words are good for simple contextualization, but not for complete expression of a thought.

In any case, both chunks and words require context in order to be meaningful. Text – in any form – without context is a pointless exercise. Text without context can even be dangerous.

A simple example of the need for context is the understanding of the word – “fire”. What does fire mean?

Fire can mean a flame, a conflagration. Fire can mean the pulling of the trigger of a loaded gun. Fire can mean the involuntary cessation of employment.

You do not know what the word fire means until you understand the context of fire.

And language is FULL of the need for context.

In everyday speech people take context for granted because that is a normal part of understanding language.

THE TEXT MODEL

The text model – the unstructured equivalent of the data model – is based on an ontology. An ontology is a vocabulary of words that reflects some discipline. The discipline can be broad or narrow. Typical disciplines that might be modelled might include medicine, legal, banking, telecommunications, military, and so forth.

The ontology consists of one or more taxonomies. A taxonomy is a vocabulary that relates to some one subject. The taxonomies in an ontology may or may not have an interrelationship between themselves.

For example, the ontology for medicine might contain taxonomies for such things as medications, diagnoses, oncology, dermatology, the ICU and so forth. The ontology for a bank might include such taxonomies as account management, loans, credit cards, savings, and so forth.

The taxonomy consists of two basic elements –

The word being examined (that is the focus of the taxonomy)

The context of the word

For example, the taxonomy for medicine might contain such things as –

Word – envarsus, context -medication

Word - blood pressure, context - vital sign

Word – milliliter, context – measurement

Word – blood, context – body fluid

MODEL DIFFERENCES

The data model and the text model both act as the compass for understanding their different environments. However, the data model and the text model are very different from each other. There are only casual similarities between the two model types, at best.

STRUCTURED DATA AND TEXT DIFFERENCES

There are some important and basic reasons why the two model types are so different. The primary root of the differences is housed in the fact that structured data and text and fundamentally different.

The primary way that structured data and text are different is the fact that structured data is dominated by keys and attributes. Structured data is made up of records of data. All the records of data in the same data base have the same structure. Only the value of data in each record is different.

Text on the other hand has very few – if any – keys or attributes. It is very unusual for a person to write an email –

“My social security number is – 123-45-6789.”

Of course, a person can write such a sentence. But it is unusual for that to happen. Instead, text is free form. People say what they need to say in order to communicate. It is really unusual for key data to appear in a text format.

When examining text and structured data, the very structure of the information is very different.

CONTEXT IN THE DIFFERENT ENVIRONMENTS

Another major difference between structured data and text is that of the existence and usage of context in each environment.

Both structured data and text require context. But context is derived and treated very differently in the two environments. In the case of structured data, context is determined at the moment that the data base is defined to the system. The developer identifies the data base and the keys and attributes of the data base at the moment of definition. And from that moment onward, the structured data is tagged with its context. Context is explicitly stated at the moment of creation of the structured data.

Textual data requires context as much as structured data. However, the context of text is determined within the actual text itself. People are used to determining the context of text as a natural part of speaking or writing. The contextualization of text occurs naturally, without conscious thought.

For example, when someone says –

“the fire is hot”

Fire is understood to mean a conflagration.

But when someone says –

“he pulled the trigger and the gun fired”

Fire is understood to mean the firing of a gun.

The context of the meaning of the word is buried in the complete collection of the text itself.

Both structured data and text require context. But con

text is derived and is understood entirely differently in both environments.

REPETITION OF DATA

Another major difference between the structured environment and the unstructured, textual environment is the repetition which occurs in the different environments.

Structured data is highly repetitive. When John Smith makes a purchase at WalMart, a record of the purchase is created. Then Mary Jones makes a purchase at WalMart and a record of that purchase is made as well. The record made for John and Mary is identical and repetitive. The only difference between the two records is the data that has been recorded. But the two transactions/purchases are highly repetitive.

In a days time there will be many such records written repetitively.

Text - on the other hand – is highly non repetitive.

Mack Wilson writes an email –

“I won’t be at the game tonight. My daughter is sick with measles.”

Mack may never write that same email again.

Structured data is highly repetitive and text data is highly random and non repetitive and this difference is another dividing line between the two environments.

STRUCTURING TEXT

In order for text to be subjected to analytical processing, text needs to be recast into a structured format. Modern analytical packages require that text be presented in a structured format. This is done by using textual ETL (TETL) to read the raw text and to produce a structured data base from the raw text that has been read.

The raw text that can be read is in many different formats, such as –

The Internet

Voice recordings

Print

Spreadsheet

Email

Or other sources of text.

The raw text is ingested and passed through TETL where it meets up with taxonomies that have been focused on the subject or discipline that is of interest. A structured data base is produced.

The transformation from raw text into a data base then is made by TETL.

The taxonomy and the raw text are used as input to TETL. The raw text is scanned and it is determined if the word being examined is in a taxonomy. If not, nothing happens. But if the word being examined does reside in a taxonomy, the word and its context are placed into a data base.

The interaction between the taxonomy and the taxonomies found in the ontology are basic and simple, as depicted

.As an example of raw text that can be turned into a data base, consider the following snippet of text that is taken from a medical record -

When the raw text is passed against the medical ontology and its many taxonomies, the following data base is produced –

In the data base that has been produced there are two levels of context – class and area.

VERTICAL VERSUS HORIZONTAL STRUCTURING OF DATA

While the data base that is produced by TETL indeed is structured, the format of the textual data base is significantly different from the format of a structured data base.

A structured data is found to be in the format of a table. The structured formatting of a table can be called a vertical structuring of data. Each column heading identifies the context of the data base. In the “name” column of a structured data base will be found –

Bill

Sarah

Sylvia

And so forth

You will never find Taco or chile relleno of snow shovel or quarterback sneak in the name column of a structured data base.

The same goes for the column named “school”. In the school column you might find –

UCLA

Purdue

Harvard

You would never find Jim Thorpe or the Lusitania or tamales or leap frog in the school column.

The general demeanor of the structured data base is that the context of the data is vertical. The context heads the column and the column is then populated by instances of the context, all falling vertically from the column name.

Textual data base structure is the opposite of structured data base structure. Textual data bases are said to be formatted horizontally. Each row of data in a textually formatted data base consists of the instance of data and its context. The context describes the text in the same horizontal row. The text and its context are displayed in a row in a horizontal manner.

For example, there might be –

Instance – Porsche, context – car model

Instance – machaca, context – food

Instance – Joe Montana, Context – football player

And so forth

Looking down the context column vertically, there is no uniformity of the values in the column. The context of the data found there can represent can be anything.

The horizontal formatting of text is necessary given the random nature in which text appears. The occurrence of text appears in a random manner and the elements of context found in the context column are also random.

The structuring of structured data and text is both completed in a structured format. But the internal basic organization of the structured format compared to the text format is very, very different.

STATIC DBMS STRUCTURES

In most dbms that service structured systems, the structure of the data is reflected in the data structure of the dbms itself. The structural representation of the data in a structured data format has the effect of locking the structure into the data base. In other words, the structured representation of the structure of the data is static. Once the structure of the data has been locked into the dbms, It is not easy or pleasant to modify.

As an example of structured data being cast into the form of the dbms, consider the following diagram.

In the diagram there is a hierarchical representation of data. Th dbms is designed to house and handle hierarchical data. A man – James Gooden – is shown to be in the parent location of the structure. There is some basic information about the man in the parent grouping of the data –

His gender

His account number

The state in which he resides

Then it is seen that James Gooden has a masters degree from Georgia Tech. He got his degrees in 2023. There may be multiple occurrences of education as well. Only the latest occurrence of data is shown in the diagram. James also has a bachelor’s degree from Florida State in engineering as well but it is not shown in the diagram..

James Master’s degree was in computer science and psychology. And James is married to June (nee) Fromm who is from Tennessee.

The structure of the data is defined to the data base management system in a hierarchical structure.

Because the structure of the data is defined to the dbms, it is static. It is not particularly easy for the analyst to add the parents of James, the hobbies of James, the synagogue that James attends, and so forth. Once the basic structure is defined to the data, it is static and not easy to alter.

A DYNAMIC STRUCTURING OF DATA

Textual data is not static at all. Textual data is dynamic. Textual data structures must be dynamic because the data structures that occur in text are VERY fluid and very dynamic. And text as it appears in writing or speech is certainly free form.

As an example of the fluid structure of textual data, consider the simple data base shown in the following diagram.

In this simple data base, it is seen that the name of the person mentioned in the text is Bill. Bill is a male. Bill lives in Colorado. Bill is married, and so forth. This record of Bill was created by the examination of a textual description.

The contents of the record of Bill depend entirely on the text that is generating the data. The records created for one conversation may mention one set of things. Another record of another conversation may mention an entirely different set of things.

The discussion of Kei Lou may mention her boyfriends, her pet cat, and her favorite foods.

The contents of what ends up in the data base depends entirely on the text as it is spoken or written.

WORDS AND CONTEXT

There are two components of textual structure that are found in the simple data base – the word that has been encountered and the context of that word.

THE POWER OF DYNAMIC STRUCTURING OF TEXT

It is not obvious that there is great power in the structuring of data this way. The only obvious power is that the structuring of textual data as has been discussed fits the free form style of the way that text is created.

But there are some very important other properties of this dynamic structuring of data. The horizontal structuring of textual data allows ANY relationship of data to be depicted. Unlike a vertical horizontal structuring of data that is found with structured data, the vertical structuring of data is dynamic, not static.

Stated differently, the static structuring of data is only able to depict the data as it has been cast at the moment of definition. But the dynamic structuring of data is capable of depicting ANY structure of data.

The textual data structures that can be formed depend only on the data that has been captured. In other words, ANY structuring of textual data can be accommodated. Which of course fits nicely with text because there is no prescribed structure of data when conversation occurs. A person can write anything they want in an email. Or say anything they want in a telephone conversation.

A SMALL DATA BASE

In order to demonstrate the power of dynamically created relationships of data, consider the following small data base.

The small data base has been created from some basic medical records. The (simplified) medical records contain information about –

The person being documented

The drugs the person takes

The gender of the person

The location of the person.

Of course, the medical records may contain a lot of other information. But for the purpose of this example, only the data that has been described will be considered.

The first kind of simple query that can be examined is – how many records are in the data base.

This query can be accommodated by looking at the record columns and counting distinct rows. In this case there are six unique record types that can be found in the data base.

This however is a simple query.

Much more sophisticated queries can be accommodated

.As an example of a more sophisticated query that can be accommodated by the horizontal structuring of data, consider the query –

How many people have exactly one drug in the data base?

This query is accommodated by looking at all the rows of data for a single record. Once the rows are found, the query then goes into the rows and sees if there is any record of a drug. If there is a record of a drug, then the query determines how many drugs are in the record.

In doing so the data base yields the answer – there are two records of people who have exactly one drug in their medical records.

Yet another example of a dynamic structure of data that can be found is one that ask the question –

How many patients are from Colorado and take the drug Multaq?

The query first gathers the rows for a patient. Then the query checks to see if the patient is from Colorado. Next the query sees if the patient takes Multaq.

In the case of the simple data base, there are two patients that meet this criteria.

The simple queries that have been shown illustrate the fact that with a horizontal, dynamic structuring of data, there are many, many structures of data that can be explored. Stated differently, the only relationships of data that cannot be explored using a horizontal, dynamic structuring of data are those relationships that are not mentioned in the text. Otherwise, the relationship can be explored.

By building the data so that data relationships can be created dynamically, the variability and randomness of text can be explored and exploited.

A ROBUST EXAMPLE

As a more elaborate example of the kind of analysis that can be done from a horizontal, dynamic structuring of textual data, consider the following example.

A collection of 10,000 medical records was collected. The doctors notes were written in a free form manner. The patients that were selected for the study had all had some kind of experience with COVID.

The patients records were read into textual ETL and a horizontal, dynamic data base was produced. Then the records were loaded into a Pearson coefficient matrix.

The Pearson coefficient matrix then compares each word against each other word to see if there is a correlation between the words. A red box that represents the intersection of the words indicates a negative correlation between the words. A green box indicates a positive correlation between the words. A light red box indicates a mildly negative correlation between the words. A light green box indicates a mildly positive correlation between the words.

A white box indicates that no relation has been detected.

The study of COVID patients found some very interesting results. With the Pearson coefficient matrix the relation between COVID and other factors was uncovered –

COVID versus –

Smoking

Alcoholism

Age

Sex

Ethnicity

Other medications ingested

And any other data captured in the medical records.

TIME REQUIRED FOR PROCESSING

The really interesting thing about this analysis is that from the time the 10,000 records were collected that it took about ten minutes time to go from the raw text to the Pearson coefficient matrix.

This short amount of time required for creating an analysis enabled heuristic, analytical processing to be done. The taxonomies used to process the textual data can adjusted to affect the data bases that have been produced. Words could be added to the taxonomy. Words can be deleted. Context could be adjusted. All sorts of adjustments could be made to the taxonomies that were used.

Then the analysis could be rerun.

If the analysis had taken longer – a week, a month, or more – it would have been very difficult to do an iterative analysis of the text.