I recently had some serious email problems. (Outgoing emails would not leave the outbox.) So I used AI (specifically, ChatGPT) to help me identify what went wrong, and how to fix the problem.

After hours of struggle, I eventually restored the system, and was able to send and receive emails from that address.

Here are some conclusions based on that experience.

• AI has become a valuable, possibly an invaluable, tool for solving problems such as the one that I experienced. AI can help not only with IT problems, but also practical issues, such as fixing a string trimmer that is used to cut grass.

• We know that AI is a pattern-recognition machine. Nevertheless, it did feel as if it were offering human-like emotional support during the struggle with the email system. It felt as if ‘someone was on my side’.

• Increasingly, AI is providing direct links to sources that support its responses. Following those links can be very helpful.

• Not all AIs are the same. Given the same problem or task, they can come up with surprisingly different responses. Different models, tools, prompts and access to current information can produce substantially different advice.

My experience also raised two broader questions about where AI is heading, and how it can help us.

• The fantastic amounts of money being invested in AI are a concern. The major technology companies are already highly profitable, but they are now committing enormous additional sums to AI infrastructure. Eventually, those investments will have to generate returns sufficient to justify their cost. It is not yet clear that they will do so. If expectations prove to have been excessive, the result could be a major financial correction, with consequences extending well beyond the AI industry.

• We may see the development of ‘local AIs’. These are AI tools that can be downloaded to a large personal computer, and then run without being connected to the internet, at least for inference work. If this trend is real, it could (a) avoid many privacy concerns, and (b) reduce reliance on data centers and, for some applications, reduce overall energy consumption.

My experience with the email problem reinforced a conclusion that I have reached before. AI can be extraordinarily useful, particularly when working through a difficult problem step by step. It can provide information, suggest alternatives and even create a sense that help is at hand. But it remains a tool, not an authority. We still need to understand the problem, evaluate the advice we receive and make the final decisions ourselves.

Can established process safety management techniques also be used to improve operability and profitability?

The accompanying presentation consists of 17 narrated slides, each supported by matching text. The slides will be released individually or in small groups.

Here is the text of the video:

Hello. My name is Ian Sutton.

Over the last several decades, the process industries have developed a mature set of techniques for managing safety risk: the discipline of process safety management.

This presentation considers a simple question: Can we use those same techniques to improve equipment reliability and system availability?

The analytical methods and management disciplines are familiar. What changes is the intended outcome. In addition to asking, “How do we prevent a major accident?” we ask, “How do we ensure that the system continues to perform its required function, and how can we optimize that function?”

That is the idea behind an Availability Case.

Last December, I published the book Process Safety Management. I am now working on a follow-up book, Industrial System Reliability. (The new book examines how risk-management techniques can be used to improve operability and profitability — always within safety, environmental, and regulatory boundaries.)

I use Large Language Models (LLMs) as an editorial tool while developing the manuscript. They help me organize material, identify repetition, check consistency, and improve the structure of the text.

Writing Concisely

LLMs can also help make the writing shorter.

A quotation often attributed to Winston Churchill captures the challenge:

“I’m sorry this letter is so long; I did not have time to make it short.”

Good technical writing is not simply a matter of adding information. In fact, good writing consists of jettisoning material without losing meaning. That process often requires more effort than producing the original draft.

Comparing AI Systems

Recently, I compared two widely used AI systems by asking them to perform the same editorial tasks.

The difference was substantial. One was considerably better at understanding the structure of a long technical manuscript, recognizing duplication, and producing concise revisions without oversimplifying the material.

The other was useful as an independent check, but it was far less helpful. Using more than one system can reveal omissions, questionable assumptions, or alternative ways of expressing an idea.

The Author’s Review

No matter how capable an AI system may be, the final text must be evaluated by the author. AI does not fully understand the author’s purpose, technical judgment, intended emphasis, or the relationship between a passage and the rest of the book. It may remove an important qualification, introduce an unintended meaning, or retain material that is technically correct but unnecessary.

The author must therefore review every revision and decide what to accept, reject, modify, or remove.

I have found that this final review often makes the text even shorter. AI removes much of the obvious repetition and wordiness. The author can then see more clearly which remaining sentences are essential and which can still be deleted.

Conclusion

AI does not replace the author or editor. It proposes changes. The author remains responsible for ensuring that the final text is accurate, appropriate, concise, and consistent with the purpose of the book. Used in this way, AI is a valuable editorial assistant—but it is not the authority.

The post, like all the others in this series, is based on the theme,

This post continues our discussion of the use of AI in industry. The post, like all the others in this series, is based on the theme,

We are bombarded with conflicting messages about the economics of the companies that provide Artificial Intelligence software (these companies are sometimes referred to as frontier AI Labs). Some analysts argue that AI will transform the economy so profoundly that almost any level of investment is justified. According to this view we should not be distracted by short-term losses, high capital spending, or weak current returns.

Other analysts take a more conventional view. They point out that leading AI companies have already attracted extraordinary investment, that their valuations are very high, and that the basic economics remain uncertain. The AI companies will need to generate enormous revenues in the near future just to provide a return to existing investors. Yet it remains unclear whether these revenues can, in fact, be generated. Some of these AI companies are growing rapidly, but growth is not the same as profitability. A company can generate impressive revenue and still consume large amounts of cash.

Reliable financial information is very difficult to obtain. The most prominent frontier AI companies are privately held. Publicly traded AI businesses are often embedded within much larger technology companies, making it hard to isolate the economics of AI itself.

Still, the scale of investment is evident. Reuters reported that OpenAI’s annualized revenue surpassed $20 billion in 2025, up from $6 billion in 2024. The amount invested in OpenAI is equally uncertain, depending on what factors count as investment. Nevertheless, a total of $300 billion appears to be in the correct range. (Reuters note that the data has not been audited or confirmed.)

Reuters also reported that Anthropic’s run-rate revenue had crossed $47 billion in May 2026, while raising $65 billion at a post-money valuation of $965 billion.

The question for industrial managers is not whether AI is impressive ― it is. The question is whether the economics of AI are sustainable, and what happens to both users and the AI companies if they are not.

A Simplified Illustration

We have developed a highly simplified model of an AI company’s economics in order to gain some understanding as to what is taking place. The numbers used are very approximate and doubtless overlook many critically important factors. Nor do they describe any specific company. The projections merely provide an order of magnitude of the financial challenge.

Here are some of the parameters:

• At least $300 billion has already been invested or committed in the AI company through a mixture of equity, debt, bonds, cloud commitments, compute financing, and infrastructure investment.

• Interest, taxes, depreciation, amortization, and cost of capital are not considered.

• The providers of this capital expect their investment to earn an adequate return within about ten years.

The following table shows possible financials for two situations. All values are in billions of dollars.

The first column lists five financial variables:

• Revenue
  From paying customers, not circular financing.

• Base Operating Costs
  Normal operations such as salaries, rent and utilities.

• Incremental Operating Cost
  Assumes that the company expands its technical and customer support staff, general infrastructure, and research and development. It also assumes higher inference costs associated with the increased number of customers, and the increased per customer use.

• Total Operating Costs
  Base Operating Cost plus Incremental Operating Cost.

• Operating Income
  Revenue less Total Operating Costs.

The second column shows the AI company’s current illustrative status based on the limited information currently available. Annual revenue is $30 billion, while annual operating costs are $40 billion, so the company has an operating loss of $10 billion per year.

The Projected column illustrates the financial scale the company may need to reach within just a few years. Annual revenue increases to $110 billion. Total operating costs rise from $40 billion to $80 billion, producing an annual operating income of $30 billion. If $300 billion has been invested or committed, then $30 billion per year for ten years merely equals the original capital amount before interest, taxes, depreciation, amortization, continuing capital expenditure, and the investors’ required return. Therefore the real economic hurdle is probably higher than this simplified model suggests.

These numbers are provided for illustration only. But they show that revenue would have to increase by about $80 billion per year, almost four times the current level, within just a few years.

For comparison, companies with annual revenue in this range include Bank of America and Target. The comparison is not exact because these companies have very different business models, but it does illustrate the scale of revenue that frontier AI companies need to reach if they are to justify the committed capital.

If the Target Is Not Reached

If frontier AI companies do not achieve this level of revenue and income, the result is not necessarily immediate failure. More likely, the business model changes.

Prices may rise; usage limits may become tighter; free or low-cost access may be reduced. The most capable models may be reserved for enterprise customers, government users, or high-value professional applications. Routine tasks may be routed to smaller and cheaper models or local AI, while frontier models are used only where the customer is willing to pay for them.

There may also be consolidation. Companies that cannot fund continuing model development, data-center construction, chip purchases, electricity supply, cooling systems, security, and inference costs may merge, be acquired, or become dependent on larger cloud providers. The AI industry may come to resemble other capital-intensive sectors, in which only a small number of firms have the balance sheet needed to compete at the frontier.

The AI companies may also move from general-purpose subscriptions to usage-based pricing, enterprise contracts, workflow integration, cloud bundling, advertising-supported services, or data-related business models.

For industrial users, the risk is not simply that an AI supplier disappears. The more likely risk is that the price, model, terms of service, data policy, reliability, or availability changes after the organization has become dependent on the system. A tool that appears inexpensive during the growth phase may become much more expensive when the provider has to demonstrate a sustainable business model.

Strategic Dependency

If an organization uses AI only for occasional drafting, brainstorming, or summarizing reports, then supplier economics may not seem important. But if AI becomes embedded in procedures, engineering workflows, maintenance planning, procurement, training, audit preparation, hazards analysis, or management systems, then supplier dependence becomes a real business issue.

In such a situation, the user company should ask practical questions, such as,

• What happens if prices double?

• What happens if the preferred model is withdrawn?

• What happens if usage limits are imposed during peak periods?

• What happens if the AI tool is merged into a larger platform?

• What happens if the organization builds internal workflows around a model that is later degraded, replaced, or made available only at a much higher price?

These are not reasons to avoid AI, but they are reasons to manage AI properly. For industrial users, the prudent response is not rejection; it is disciplined adoption.

Conclusions

No one knows whether the use of AI software will fundamentally transform industry. But the economic calculations provided here suggest that frontier AI companies need to grow their revenues at an exceptionally rapid rate if they are to satisfy the investments that have been made, and that continue to be made. If that growth fails to materialize, industrial companies that use AI software could find themselves dependent on suppliers whose financial future, pricing structure, or service commitments are uncertain.

The practical response is to manage and limit such dependency. Therefore, industrial companies using AI software should preserve access to their own data, avoid workflows that depend on one proprietary model, document which AI tools are used in controlled work processes, maintain human review of safety-critical decisions, and stress-test budgets against higher AI pricing. AI adoption should proceed with the same discipline applied to other critical suppliers.

With regard to management responsibility, we reiterate the importance of the following quotation.

Subscribe now

A search engine indexes information in response to a query. It helps the user find documents, web pages, reports, standards, articles, drawings, manuals, procedures, or other material that may be relevant to a question.

However, the search engine does not tell the user what to do. It merely provides places to look. The user then decides which sources matter, which ones can be trusted, how they should be interpreted, and what action, if any, should be taken.

Generative AI changes that relationship. The user no longer asks only ‘where can I find information about this subject?’ He or she may now ask:

• What does this mean?

• Which option is better?

• What are the risks?

• What have I missed?

• What should I do?

The user is not simply asking for information; the user is looking for guidance that may lead to action.

In other words, generative AI is not merely a new type of software. It fundamentally changes the software’s role. The tool is no longer acting only as an index. It is acting as an interpreter, summarizer, analyst, critic, and — potentially — as an adviser.

Search Results and AI Answers

When a conventional search engine provides results, its limitations are usually visible.

The user sees a list of sources. Some are relevant; some are not. Some are current; some are obsolete. Some are authoritative; some are informal. Some are technical; some are promotional. Some are based on experience; others are merely opinion.

The user knows that the work of evaluation remains to be done, and that the evaluation has to be conducted by the human who made the initial query.

By contrast, an AI may produce a fluent answer in direct response to the question. It may appear to synthesize information from several sources. It may explain technical concepts. It may compare alternatives. It may provide a recommendation. Any or all of these responses may be useful.

But they need to be checked.

Why Generative AI Is Useful

Generative AI is useful because professionals are typically overloaded with information. Engineers, managers, supervisors, inspectors, and regulators generally have access to large quantities of procedures, standards, incident reports, drawings, inspection records, emails, operating logs, vendor documentation, and maintenance histories.

The problem is not lack of information; the problem is finding the relevant information, organizing it, and seeing its implications.

AI can help with that work. It can summarize a set of documents. It can identify themes in incident reports. It can compare a procedure with a corporate standard. It can list assumptions in a proposed decision. It can generate challenge questions for a management review. It can help a user move from scattered information toward a more structured understanding of the issue.

AI can also help a competent person think more clearly. It can help organize complexity. It can help identify gaps. It can help create a first draft of an analysis. It can help challenge the obvious answer.

But those responses come with a risk.

The Risk of Advice-Like Answers

Once the system moves from indexing to advice, the output is no longer merely helping the user find information; it may influence professional judgment.

Consider a maintenance engineer reviewing repeated pump failures.

In response to a search engine query, a search engine might return links to pump manuals, bearing failure articles, vibration references, vendor data sheets, and past work orders. The engineer still has to read the material, compare it with site conditions, and decide what is relevant.

An AI system will go further. It may suggest that the most likely causes are cavitation, misalignment, poor lubrication, incorrect bearing installation, or operation away from the best efficiency point.

Each response may be helpful. It may also be wrong. The responses are hypotheses, not final conclusions. They must be tested by the maintenance engineer against field evidence, operating data, inspection findings, vibration analysis, maintenance history, process conditions, and the experience of the people who know the equipment.

Questions to Ask

In response to an AI’s suggestions, the user should ask questions such as,

• What information did the system use?

• What information did it not use?

• Was the information current?

• Was it approved?

• Was it site-specific?

• Did the system distinguish fact from inference?

• Did it identify uncertainty?

• Did a qualified person review the result?

• Who is responsible for the final action or decision?

Subscribe now

We continue our discussion to do with the use of AI in industry. This post, like all the others in this series, is based on the theme,

One of the uses of Artificial Intelligence (AI), specifically Large Language Models (LLMs), is idea generation. AI can help individuals and project teams think more broadly about a problem. It can suggest alternative designs, identify safety concerns, propose possible causes of failure, develop training scenarios, or offer different ways of organizing work. Its value lies not in providing ‘the answer’ ― there is no final answer ― but in helping competent people ask a wider range of questions.

Consider a team investigating repeated pump seal failures.

The team could ask an AI system to generate possible technical, operational, maintenance, and organizational causes. The resulting list might include misalignment, cavitation, unsuitable seal materials, incorrect startup procedures, inadequate flushing, excessive vibration, or changes in operating conditions. It might also suggest less obvious possibilities, such as poor communication between operations and maintenance, incomplete handover between shifts, or failure to incorporate previous incident findings into the maintenance program.

Such a list is useful. But it is not a diagnosis.

The AI has not inspected the pump. It has not reviewed the operating data. It has not examined the failed seal. It has not walked down the installation, checked the piping arrangement, reviewed the maintenance records, or spoken with the operators and mechanics involved. It may have access to general language-based information about pumps and seals, but it has no direct knowledge of this pump, this service, this site, or this failure.

Therefore,

Each AI-generated suggestion should be treated as a question to be investigated, not as a conclusion to be accepted.

AI may also help prevent premature closure of a discussion, where a group settles too quickly on the most familiar or convenient explanation. The AI can be asked to suggest alternative causes, identify assumptions, generate ‘What if?’ questions, challenge an emerging consensus, or point out categories of causes that the team may not yet have considered.

For example, if the initial discussion of a seal failure focuses only on mechanical causes, AI may prompt the team to consider operating procedures, maintenance practices, training, management of change, spare parts quality, or communication between departments. These suggestions may or may not prove relevant. But they can help ensure that the investigation is not framed too narrowly.

Using AI in this manner is generally low-risk because the AI suggestions do not have to be acted upon directly. They are simply suggestions; prompts for further review. That’s all.

Nevertheless, there are still risks.

AI-generated suggestions may be impractical, unsafe, irrelevant, or based on false assumptions. They may also appear more complete than they really are. A polished, well-written, fluent list can create a false sense that the problem space has been fully covered. The team may stop the investigation simply because the AI has produced a plausible and well-organized answer.

For these reasons, AI should be used to broaden professional thinking. It can help generate possibilities, structure a discussion, and challenge assumptions. But the work of investigation, judgment, and decision-making remains with the people who understand the equipment, the process, and the consequences of being wrong.

Subscribe now

An earlier post in this series Expert Systems and LLMs: Mistakes and Hallucinations noted that Expert Systems are one approach to the use of Artificial Intelligence (AI). That post discussed the fact that Expert Systems can make mistakes, but they do not normally hallucinate in the way LLMs can. They do not generate fluent unsupported explanations; they apply the rules they have been given.

Further thoughts to do with the use of Expert Systems are provided in this post.

Expert Systems

Expert systems are a rule-based form of AI. They are an important predecessor and companion to modern AI.

Unlike large language models, which generate responses from statistical patterns in data, expert systems apply explicit rules developed by human experts. They are narrower and less flexible than LLMs, but they can be more transparent, auditable, and suitable for structured industrial decision support.

For example, if the flow from a pump is lower than expected, an expert system might follow a rule sequence such as:

• IF the pump discharge pressure is low,

• AND the suction pressure is normal,

• AND the motor current is low.

• THEN possible causes include worn impeller, internal recirculation, an open recycle or bypass valve, or incorrect pump speed.

The system would then ask for additional information, such as pump speed, valve positions, strainer condition, recent maintenance history, and trend data, before assigning priority to the possible causes.

An expert system does not ‘learn’ from large quantities of text in the way an LLM does. It follows a structured set of rules created by engineers, specialists, or knowledge engineers. This means that the output can be consistent, traceable, and reliable within a defined operating envelope, but only if the rules are correct, current, and complete. It also means that the system is less flexible and adaptive than an LLM.

The following Table summarizes some of the differences between Expert Systems and LLMs.

As discussed in the earlier post, the distinction between making a mistake and hallucinating is important. Humans, expert systems, and LLMs can all make mistakes. Hallucination is different. If the prompt, retrieved material, or available context does not support a correct response, an LLM may still generate a plausible-sounding answer that is unsupported or false. A disciplined human expert should say, “I do not know”, or should identify the assumptions and evidence needed before reaching a conclusion. An expert system, if properly designed, should remain within its rule set. An LLM may instead produce fluent language that sounds authoritative even when the answer is wrong.

Components of an Expert System

An expert system usually has three parts:

1. Knowledge Base
   Rules, facts, decision trees, procedures, and domain knowledge.

2. Inference Engine
   The logic mechanism that applies the rules to a specific situation. It may work forward from known facts toward a conclusion, or backward from a possible conclusion to determine what facts must be confirmed.

3. User interface / Explanation
   The part that asks questions, gives recommendations, and that can explain why it reached a conclusion.

The explanation function was one of the strengths of expert systems. They could often say, in effect, ‘I reached this conclusion because rules 4, 7, and 12 were satisfied.’ This is unlike LLMs, whose reasoning is not normally traceable in the same formal way.

Industrial Relevance

Many industrial decisions are already rule-based: if a condition exists, then a review, approval, inspection, safeguard, or procedure is required. Examples include:

• Equipment troubleshooting

• Alarm response support

• Permit-to-work screening

• Maintenance diagnostics

• Checklist support

• Regulatory compliance checks

• Operating procedure guidance

• Process safety management audits

• Management of Change screening

• Safety instrumented function proof-test logic

• Startup and shutdown readiness checks

• Relief system documentation checks

• Inspection deferral screening

• Operating-envelope monitoring

For example, an expert system could support an MOC (Management of Change) review, which depends on structured questions and decision logic, by asking:

• Does the change affect equipment design?

• Does it alter operating limits?

• Does it change alarms or interlocks?

• Does it require procedure updates?

• Does it affect training requirements?

In safety-critical applications, the expert system should support the decision process; it should not bypass engineering review, operating discipline, or formal approval.

Limitations

Expert systems are limited by the rules they contain, and may fail when:

• The situation falls outside the anticipated rule set.

• The facts entered by the user are incomplete or wrong.

• The plant, equipment, or operating mode has changed but the knowledge base has not been updated. Hence, the original rules were based on assumptions that are no longer valid.

• Informal operating knowledge has not been captured.

• The rules interact in unexpected ways.

• The rules may be based on outdated assumptions.

• Conflicting rules require a defined priority system or human resolution.

These limitations are familiar to industrial organizations. Procedures, checklists, alarm response guides, and engineering standards all have similar weaknesses. They are valuable only when they are current, technically sound, and used by competent people.

Relationship to Modern AI

Expert-system concepts have not disappeared. Rather, they are often embedded inside modern AI software under names such as:

• Decision support systems

• Rule engines

• Diagnostic systems

• Compliance engines

• Workflow logic

• Alarm rationalization logic

• Safety lifecycle tools

• Knowledge management systems

Modern AI can also be combined with expert systems. For example:

• An LLM can interpret a user’s question.

• A retrieval system can find relevant procedures.

• An expert system can apply formal rules.

• A human expert can approve the final recommendation.

This combination may be more useful than a free-standing LLM. Some tasks require broad language capability, but others require disciplined rule-following, traceability, and formal review. In industrial applications, the objective should not be to replace rule-based systems with LLMs. The better objective is to combine language capability, verified documents, rule-based logic, engineering standards, and accountable human review.

Subscribe now

A theme of my writing to do with Artificial Intelligence (AI) can be summarized in the statement,

Ronald Reagan expressed the same idea when he said, “Trust, but verify”.

With that thought in mind, I decided to test my LLM (Large Language Model) on its ability to understand a flowsheet, and how processes operate.

Here is the flowsheet.

Note that this drawing is neither a P&ID nor a Process Flowsheet. A P&ID would have much more equipment detail, and a Process Flowsheet would provide information to do with pressures, temperatures and flow rates. In other words, the drawing is a sketch. (For background, P-101A is normally operating; the steam-driven pump P-101B is a backup.)

Here is the ‘conversation’ that I had with my LLM.

The Conversation

Question:
Shown on the diagram are a tank, two pumps, a pressure vessel, some piping, and a simplified control system. This drawing is neither a P&ID nor a process flow diagram. It is a simplified process sketch. A P&ID would show more piping, valves, instruments, line numbers, equipment details, and safeguards. A process flow diagram would normally include process conditions such as flow rates, temperatures, pressures, and material balance information.

Tell me:

• Which pump is electrically driven?

• Which vessel is PSV-101 protecting?

• What is the identification number of the level sight glass on T-100?

LLM:

• P-101A

• V-101

• LI-100

These responses are correct.

Question:
What is the type of level control system in T-100?

LLM:
The level control system on T-100 is a cascade level control system.

Specifically, LRC-101 measures/controls the level in T-100 and sends a signal to FRC-101, which controls the flow through the control valve on the pump discharge line. So the arrangement is: T-100 level → LRC-101 → FRC-101 → flow control valve. In control terminology, level is the primary/master control variable, and flow is the secondary/slave control loop.

Once more, the answer is correct.

Question:
Should there be check valves on the discharge of the two pumps?

LLM:
Yes. For the arrangement shown, there should normally be a check valve on the discharge of each pump. The two pumps, P-101A and P-101B, appear to discharge into a common header. If one pump is running and the other is stopped, the running pump could otherwise push liquid backward through the idle pump. That can cause:

• Reverse rotation of the standby pump,

• Loss of delivered flow to V-101,

• Overheating or damage to the idle pump,

• Possible leakage or mechanical seal problems,

• Unintended circulation back toward T-100.

The usual arrangement would be:

pump discharge → check valve → discharge isolation valve → common discharge header.

Each pump should also normally have suction and discharge isolation valves so that one pump can be maintained while the other remains available.

On this drawing, the check valves are not shown. For a simplified flowsheet that may be acceptable, but on a P&ID or operating drawing I would expect to see a non-return/check valve on the discharge of both P-101A and P-101B, unless there is a documented reason why another protective arrangement is being used.

Again, the answer is correct.

Finally,

Question:
What could cause T-100 to overflow?

LLM:
T-100 would overflow if the inflow to T-100 exceeds the liquid removed by P-101A/P-101B. On this drawing, the main protection against high level is the level/flow control system: LRC-101 → FRC-101 → discharge control valve. Anything that defeats that balance could cause overflow.

Possible causes include:

The most direct scenario is:

RM-12 continues entering T-100, while P-101A/P-101B do not remove enough liquid. For a more complete P&ID, I would expect to look for high-level alarm, high-high-level trip or shutdown, overflow/vent routing, pump permissives, and clear fail positions for the control valve.

This was a strong response.

Conclusions

• The LLM made correct inferences from the drawing, even though it did not ‘understand’ the drawing.

• The LLM correctly understood the need for discharge check valves, and what would happen were they not present. It identified five potential hazards, some of which could have been overlooked by a human process engineer.

• Furthermore, the LLM suggested the need for suction and discharge isolation valves, and explained why. It provided this suggestion without being asked to do so.

• The analysis to do with high level in T-100 was thorough. It provided a starting list of causes that a HAZOP team could review, challenge, and supplement with plant-specific knowledge.

• It made suggestions for adding safeguards to control the high level scenario.

• It used the term ‘non-return valve’ as well as ‘check valve’. In other words, it understood language differences between different engineering cultures.

• The LLM provided the above conclusions and analyses instantly.

This simple example illustrates the value of AI in industrial work. It can provide a fast, structured first analysis. It can identify omissions. It can suggest challenge questions. It can help prepare for a HAZOP, design review, or operating procedure discussion.

But it is still not authoritative. The LLM did not know whether the drawing was current. It did not know the actual valve positions, pump curves, instrument ranges, control-valve fail position, operating history, alarm settings, or site practices. A qualified engineer or operating team still has to verify the output.

The conclusion remains the same:

Subscribe now

Artificial Intelligence is often treated as a new technology. In one sense, that is true. The current generation of Large Language Models (LLMs) has capabilities that were not available to most engineers and managers even a few years ago. But, in another sense, AI is not new at all.

Expert Systems

One form of AI — the expert system — has been used in engineering work for many years. For example, about twenty years ago, I used an expert system to assist with the design of shell-and-tube heat exchangers. The software had been written by a heat exchanger expert. Its purpose was to evaluate proposed designs against applicable codes, standards, and engineering requirements, including those from organizations such as TEMA, ASME Section VIII, and API 660.

The software did not ‘think’ in the way people now discuss AI. It followed rules, checked calculations, compared design details with specified requirements, and flagged potential problems. The software could check whether a proposed heat exchanger design satisfied the applicable code, standard, and purchaser requirements.

The software also offered judgment. For example, it might say that a liquid velocity on the tube side was technically allowable, but nevertheless appeared to be too high. The software was not merely checking pass/fail requirements, it also captured some of the practical judgment of the expert who had developed it.

Mistakes and Hallucinations

At first, expert systems and Large Language Models may appear similar. Both can evaluate a situation, review information, offer guidance, and can suggest that something looks wrong, incomplete, or risky. There is, however, a crucial difference between expert systems and LLMs.

An expert system can make mistakes. An LLM can hallucinate.

An expert system is based on explicit rules, calculations, decision trees, and engineering assumptions. Those rules are written by people, and people can be wrong. The rule may be incomplete. The expert may have misunderstood the standard. The standard may have changed. The software may be used outside its intended design envelope. The input data may be incorrect. The program may contain a bug.

But, in principle, the error can usually be traced. The user can ask: What rule was applied? What calculation was performed? What assumption was made? What standard was being used? What edition of the standard applied? Was the software being used for a case it was designed to handle?

An LLM is different. It does not apply a fixed set of engineering rules in the same way. It generates responses from patterns in language and information. It can produce a fluent, confident, and technically plausible answer that is not actually grounded in the applicable standard, drawing, calculation, field condition, or operating practice.

In other words, it hallucinates.

The LLM may cite a requirement that does not exist. It may misstate a code clause. It may combine requirements from different standards. It may present a general engineering practice as if it were a mandatory rule. It may use the right terminology in the wrong context. It may sound authoritative even when it is not.

This does not mean that expert systems are safe and LLMs are unsafe. Nor does it mean that expert systems are obsolete and LLMs are superior. The point is more practical than that.

Different forms of AI fail in different ways.

Expert systems can fail within a defined structure of rules, assumptions, and input data. They are written and developed by human beings, and human beings make mistakes. LLMs, on the other hand, can fail by producing fluent statements that are plausible but unsupported.

Given this background, expert systems may be more trustworthy for certain narrowly defined engineering tasks. Their rules are explicit. Their scope is limited. Their assumptions can usually be examined. If an error is found, the rule, calculation, or decision logic can often be corrected.

This does not make expert systems authoritative. It means that their failure modes are usually easier to trace than those of an LLM.

Helpful, Not Authoritative

Regardless of the type of software being used, AI output should never be treated as authoritative; it must always be checked by competent people against the actual standard, the actual drawing, the actual equipment, and the actual operating situation.

AI can be helpful. It can organize information, identify issues, challenge assumptions, and support engineering judgment. But it cannot replace responsibility. In industrial work, judgment remains with the engineer, manager, operator, or organization that uses the output.

Subscribe now

We continue development of the book Artificial Intelligence in Industry. An overview of the book is available at Artificial Intelligence in Industry.

Chapter 1 — Helpful, But Not Authoritative is available in .pdf format. It can be downloaded here: https://suttontechnicalbooks.gumroad.com/l/svmrim for $9.99.

The chapter introduces one of the central themes of the book: AI can be extremely useful in industrial work, but it must not be treated as an authority. AI can help draft, summarize, organize, compare, and challenge ideas. It can support engineering, operations, maintenance, management, and executive work. But it cannot verify field conditions, replace operating experience, make ethical judgments, or accept responsibility for a decision.

The Table of Contents for Chapter 1 is:

• Structure of This Book

• Background

• Example

• Responsibility

• Strategies

  • Idea Generator

  • Drafting / Editing Assistant

  • Analytical Support / Systems Thinking

• Critical Thinking

• Management of Change

• A Standard Example

• Conclusion

A key insight in the chapter is that AI in industry does not require an entirely new management philosophy. Many of the risks associated with AI can be managed using techniques that industrial organizations already understand.

In particular, AI can be managed through well-established Management of Change (MOC) principles. Before AI is introduced into procedures, engineering work, maintenance planning, incident analysis, training, or management decision-making, organizations should ask familiar MOC questions:

• What is being changed?

• Who is affected?

• What new hazards or failure modes are introduced?

• What controls are needed?

• Who reviews and approves the change?

• How will the change be communicated?

• How will the organization know whether the change is working as intended?

That framing is practical. It keeps the discussion away from speculation about machine consciousness or technological inevitability and brings it back to industrial discipline.

AI may be new. The need for control, verification, accountability, and sound management is not.

Subscribe now

Artificial intelligence is moving quickly into industrial work. Managers, engineers, operators, maintenance specialists, and executives are using AI to draft procedures, summarize documents, compare options, analyze incidents, support decisions, and improve communication.

This Substack series follows the development of a book titled Artificial Intelligence in Industry. The book is written for people who work in industrial organizations: engineers, technicians, maintenance personnel, operations superintendents, facility managers, executives, consultants, regulators, and others whose decisions affect safety, reliability, production, environmental performance, cost, and public trust.

A central argument of the book is that AI is extremely useful, but it must be understood realistically. It is not a licensed engineer, an experienced operator, or an accountable decision-maker. It can assist professional work, but it cannot replace field knowledge, engineering discipline, management systems, or ethical judgment. For example, Chapter 1 ends as follows.

Would I be willing to sign my name to this AI-generated analysis, instruction or report, knowing that it might later be examined during an audit, investigation, or legal proceeding?

If the answer to the above question is ‘No’ then the AI output should be used with caution. That output may be useful. It may be interesting. It may be thought-provoking. It may point to issues that deserve further investigation. But it is not trustworthy.

The book examines:

• What AI can realistically do in industrial settings

• Practical applications for engineering, operations, maintenance, management, and executive work

• Economics, productivity, and cost

• Risk management and decision-making

• Reasoning and creativity

• Confidentiality, verification, and accountability

• Governance standards such as ISO/IEC 42001

• Innovation and new management models

The current Table of Contents is:

Chapter 1 — Helpful, Not Authoritative
Chapter 2 — Capabilities and Limits
Chapter 3 — Professionalism and Governance
Chapter 4 — Risk Management
Chapter 5 — AI Within the Organization
Chapter 6 — Innovation

Subscribe now

We continue writing the manuscript of the book System Reliability.

A key feature of the book is the development of an understanding as to how risk is understood and managed.

When it comes to safety the level of risk can never be too low. We see this in company slogans such as,

No One Gets Hurt

Every Person Home Safe Every Day

The same way of thinking applies to environmental performance. For example,

Every Release Matters

No Leaks. No Spills. No Excuses.

The catch with slogans such as these is that they are aspirational, but not fundamentally realistic. Hazards always exist, those hazards have a finite likelihood of occurrence, and they have safety and environmental consequences.

Risk can never be zero.

When it comes to system reliability this way of thinking no longer applies. It is possible to be ‘too reliable’. For example, an engineer may propose spending $1 million dollars on new equipment to improve reliability. If that investment leads to cost reductions of $1 million annually then it is clearly justified. However, if the savings are just $100,000 per year, then the project will not move forward ― assuming that safety and environmental performance are not affected.

This way of thinking leads to a different set of slogans:

Reliable Operation Through Disciplined Management

Right Reliability, Right Cost

Subscribe now

An old Greek saying, usually attributed to the poet Archilochus, states:

The fox knows many things, but the hedgehog knows one big thing.

The phrase was made famous in modern times by Isaiah Berlin in his essay The Hedgehog and the Fox. Berlin used the phrase to distinguish between thinkers who organize the world around one large central idea and those who draw on many ideas, observations, and experiences.

The distinction is useful in process safety.

The hedgehog looks for the governing principle. The fox looks for the details. The hedgehog asks, “What is the central issue?” The fox asks, “What is actually happening?” The hedgehog seeks coherence. The fox seeks evidence.

Both ways of thinking are needed.

The Hedgehog: One Big Thing

In process safety, the hedgehog’s ‘one big thing’ is simple:

The facility must operate within clearly defined safety and environmental boundaries.

This principle is not negotiable. Production goals, maintenance schedules, capital budgets, staffing pressures, and business targets are all important. But they are subordinate to the requirement that the facility be designed, operated, maintained, and modified within safe limits.

The hedgehog view is essential because process safety programs can become fragmented. The programs include operating procedures, training records, hazard analyses, mechanical integrity programs, management of change reviews, pre-startup safety reviews, incident investigations, audits, emergency response plans, and many other elements. Each is important. But each can also become a separate administrative activity, detached from the larger purpose.

The hedgehog reminds us that all these activities serve one purpose: keeping hazardous materials and energies under control.

This is the strength of the hedgehog approach. It provides clarity. It allows management to say, “This is the boundary. We will not cross it.”

For example:

• We do not start up a unit until the required safeguards are in place.

• We do not bypass an interlock without proper review and authorization.

• We do not operate outside established limits in order to meet a production target.

• We do not treat a temporary repair as permanent.

• We do not make a process change without understanding its consequences.

These are hedgehog statements. They are based on governing principles.

But hedgehog thinking also has a weakness. It can become too simple. It may produce slogans instead of understanding. ‘Safety first’, ‘zero incidents’, or ‘follow the procedure’ may be true in a general sense, but they do not by themselves explain why a pump seal keeps failing, why operators are overriding alarms, why a startup procedure is confusing, or why a maintenance backlog is growing.

For that, we need the fox.

The Fox: Many Things

The fox is empirical. The fox does not assume that one explanation is enough. The fox looks at the evidence from several angles.

Consider a recurring pump seal failure. A hedgehog response might be:

We need better mechanical integrity.

That may be true, but it is incomplete. A fox-like investigation would ask a wider set of questions.

Was the seal correctly specified for the service? Has the process composition changed? Are there off-design operating conditions? Is the pump operating away from its best efficiency point? Are there vibration problems? Are operators starting or stopping the pump in a way that damages the seal? Was the seal installed correctly? Were flush systems working? Was there a management of change review when upstream conditions changed? Are maintenance technicians trained on this seal design? Are spare parts correct? Are previous failures being analyzed, or simply repaired?

The fox does not reject the big principle. The fox makes the principle operational.

This matters because process safety events rarely arise from one isolated failure. They usually involve combinations of technical, human, organizational, and management system weaknesses. A relief valve may be incorrectly sized, but the deeper issue may include deficient process safety information, an incomplete hazard analysis, poor management of change, inadequate training, and weak auditing. A line may rupture because of corrosion, but the explanation may involve inspection intervals, dead-leg identification, materials of construction, operating history, and budget decisions made years earlier.

The fox is useful because process safety is a system problem.

The Danger of Being Only a Hedgehog

A purely hedgehog approach can lead to overconfidence. Management identifies a single theme and forces all evidence into that framework.

After an incident, the explanation may become:

• The operator failed to follow the procedure.

• The procedure was inadequate.

• Maintenance did not do its job.

• The culture is poor.

• This was a training problem.

Each of these statements may contain some truth. But each may also be too narrow. A single-cause explanation is often attractive because it is administratively convenient. It allows a single corrective action to be written. It allows the investigation to be closed. It allows the organization to move on.

But the apparent simplicity may be false.

For example, saying “the operator failed to follow the procedure” may avoid harder questions:

• Was the procedure actually usable?

• Did it match field conditions?

• Were the steps in the correct sequence?

• Were critical steps identified?

• Was the operator under time pressure?

• Were alarms distracting or excessive?

• Had the procedure been changed?

• Was the operator trained on the current version?

• Did experienced personnel routinely use informal workarounds?

• Did supervision know about those workarounds?

The hedgehog explanation may be neat. The fox explanation may be messy. But the messy explanation is often closer to reality.

The Danger of Being Only a Fox

The fox has its own weakness. A purely fox-like approach can produce endless detail without judgment.

Investigations can become compilations of facts. Hazard analyses can become long lists of scenarios without a clear sense of priority. Audits can produce many findings, but no understanding of the underlying management system weakness. Meetings can end with action items but no strategic conclusion.

The fox may know many things but fail to identify which things matter most.

This can be a common problem in process safety management. Organizations collect large amounts of data: inspection results, alarm counts, overdue action items, near-miss reports, training records, audit findings, incident reports, and maintenance history. But data accumulation is not the same as understanding risk.

At some point, management must ask hedgehog-style questions:

• What is the most important risk here?

• What boundary is being approached?

• What safeguard is weak?

• What decision must be made?

• What will we stop doing?

• What will we fund?

• What will we not tolerate?

Without that discipline, the process safety program can become busy but ineffective.

Hedgehog First, Fox Second, Hedgehog Again

The best process safety thinking uses both styles.

First, start with the hedgehog. Define the governing principle.

For example:

The process must not be operated outside its safe operating limits.

Then work like a fox. Examine the evidence.

Are limits clearly defined? Are they in the control system? Are operators trained on them? Are alarms set appropriately? Are excursions investigated? Are temporary limits controlled? Are operating procedures consistent with the limits? Have process changes affected the limits? Are instruments reliable? Are supervisors reinforcing the limits or informally encouraging production beyond them?

Then return to the hedgehog. Make a clear decision.

If the facility cannot demonstrate that safe operating limits are known, controlled, and respected, then production must be constrained until that weakness is corrected.

This pattern can be applied to many process safety issues such as the following.

Management of Change

Hedgehog: No change should be made until its safety and environmental consequences are understood.

Fox: What counts as a change? Are temporary changes included? Are software changes included? Are organizational changes included? Are changes to feedstock, operating envelope, equipment, control logic, procedures, staffing, or inspection intervals being captured? Are MOC reviews technically competent? Are action items closed before startup?

Hedgehog: If the change has not been reviewed, approved, communicated, and implemented correctly, then it is not ready.

Operating Procedures

Hedgehog: Procedures must support safe and consistent operation.

Fox: Are procedures current? Are they written for actual field conditions? Do they distinguish normal, startup, shutdown, emergency, and temporary operations? Do operators use them? Are critical steps visible? Are consequences of deviation explained? Are procedures revised after incidents and changes?

Hedgehog: A procedure that is not accurate, usable, and controlled is not an effective safeguard.

Mechanical Integrity

Hedgehog: Equipment that contains hazardous materials must remain fit for service.

Fox: What equipment is safety-critical? Are damage mechanisms understood? Are inspection intervals justified? Are overdue inspections risk-ranked? Are temporary repairs controlled? Are recurring failures investigated? Are spare parts correct? Are maintenance deferrals reviewed?

Hedgehog: Equipment that is not demonstrably fit for service should not be treated as available for normal operation.

Process Safety Needs Both Disciplines

A strong process safety culture does not choose between foxes and hedgehogs. It needs both.

It needs hedgehogs because some principles must be clear. Safety and environmental boundaries matter. Management must be willing to define them, communicate them, fund them, and enforce them.

It needs foxes because real facilities are complex. Equipment degrades. Procedures drift. People adapt. Production pressures change. Small deviations become normalized. Records are incomplete. Workarounds become routine. The actual system is always more complicated than the formal system.

The hedgehog protects the boundary.

The fox understands the field reality.

Used together, they provide a practical model for process safety leadership. Start with the governing principle. Investigate with breadth and humility. Then return to a clear decision.

Subscribe now

Last December we published the book Process Safety Management. Before that we published Plant Design and Operations. We are now working on the third book in this series: System Reliability: Decision Making in an Age of AI. An introduction to the series is provided here.

This new book is very much a work in progress. Nevertheless, we have releasing draft materials in order to encourage feedback and discussion. Please provide your comments.

Chapter 1

The first set of released materials will be four sections from Chapter 1: Boundaries and Outcome. There are four download modules:

• Chapter 1a

• Chapter 1b

• Chapter 1c

• Chapter 1d

The current Table of Contents for Chapter 1 is shown below.

Chapter 1 — Boundaries and Outcome
Overview
Book Structure
Book Series
Understanding Risk
Risk Management Strategy
Asymmetry
Benefits of Improved Availability
Optimum Reliability Investment
Production and Profitability
Reduced Maintenance and Downtime Costs
Customer Satisfaction and Public Relations
High Consequence Events
Leadership Benefits
Worked Examples
Pump System
Heat Exchanger
Concluding Transition

Overview of Chapter 1

Chapter 1 introduces a central idea of this book: that industrial performance is governed by two fundamentally different types of constraint. Safety and environmental protection define the boundaries within which a facility must operate. These boundaries are non-negotiable and reflect ethical, regulatory, and societal expectations. Within those constraints, the primary determinant of outcome is system availability — the ability of the facility to operate as intended, when required, and to recover quickly from disruption.

The chapter distinguishes between external factors that are largely beyond management control, such as market prices and input costs, and internal factors — reliability, maintenance, operations, and organizational effectiveness — that directly influence availability. It shows how these internal variables shape both production and cost, and therefore overall financial performance.

Although safety and availability both involve the management of risk, they are not the same type of problem. Safety decisions are constrained by ethical considerations and do not lend themselves to simple optimization. In contrast, availability can be analyzed, managed, and improved using economic and engineering principles.

The chapter concludes by outlining the structure of the book and introducing two worked examples that are used throughout to illustrate key concepts.

In other words:

Safety defines the boundaries.
System availability defines the outcome.

Subscribe now

The CCPS (Center for Chemical Process Safety) has published the May 2026 Process Safety Beacon Training After a Change. This Beacon describes a situation where instrumentation was changed, but there was no follow up training, or an update of the operating procedures. This was a near-miss situation — no one was injured.

The following is an analysis of the Beacon using the ‘top three’ of the 20 elements of process safety as defined by the CCPS.

PSM Elements

In this series of posts we evaluate incidents for process safety insights using the management elements defined by the Center for Chemical Process Safety (CCPS). The elements are:

1. Process Safety Culture

2. Compliance

3. Competence

4. Workforce Involvement

5. Stakeholder Outreach

6. Knowledge Management

7. Hazard Identification and Risk Management

8. Operating Procedures

9. Safe Work Practices

10. Asset Integrity / Reliability

11. Contractor Management

12. Training / Performance

13. Management of Change

14. Operational Readiness

15. Conduct of Operations

16. Emergency Management

17. Incident Investigation

18. Measurement and Metrics

19. Auditing

20. Management Review

The ‘Top Three’

Of the elements listed, the following are particularly pertinent to this incident.

Management of Change

This is the dominant element. The installation of vibration monitoring sensors was a physical and procedural change. Later, replacing the motor was also a change affecting the equipment configuration. The change process apparently did not ensure that the sensor remained part of the equipment package, that the affected procedures were updated, or that workers and contractors understood the new requirements.

Training and Performance Assurance

The Beacon’s title points directly to this issue. Workers responsible for operating, maintaining, inspecting, or replacing the equipment needed training after the vibration-monitoring system was installed. Training should have made clear that the sensor was now part of the equipment’s required configuration and had to be reinstalled or reconnected after motor work.

Operating Procedures

The motor replacement procedure apparently did not include instructions to remove, protect, reinstall, reconnect, and verify the vibration sensor. If procedures are not revised after a change, field personnel will often revert to the old task method, which is what appears to have happened here.

Secondary elements

Asset Integrity and Reliability

The vibration-monitoring system is part of the equipment reliability and early-warning strategy. If the sensor is not installed or functioning, the organization may falsely believe that critical rotating equipment is being monitored.

Operational Readiness

After equipment modification or replacement, there should have been a readiness check or inspection confirming that the motor, sensor, wiring, data signal, and alarm/monitoring display were all restored and functional.

This item is a close runner-up for the top three.

Contractor Management

The Beacon states that contractors may not have reinstalled the sensor or may not have had the updated procedure. If contractors perform maintenance on changed equipment, they must receive the same relevant technical information and procedural expectations as employees.

Process Safety Information / Knowledge Management

Drawings, equipment records, inspection checklists, maintenance instructions, and monitoring-system documentation should have reflected the installed sensor. Without updated information, the change becomes invisible to later workers.

Subscribe now

Last December we published the book Process Safety Management. Before that we published Plant Design and Operations. We are now working on the third book in this series: System Reliability: Decision Making in an Age of AI. An introduction to the series is provided here.

This new book is very much a work in progress. Nevertheless, we have releasing draft materials in order to encourage feedback and discussion. Please provide your comments.

Chapter 1

The first set of released materials will be four sections from Chapter 1: Boundaries and Outcome. There are four download modules:

• Chapter 1a

• Chapter 1b

• Chapter 1c

• Chapter 1d

The current Table of Contents for Chapter 1 is shown below.

Chapter 1 — Boundaries and Outcome
Overview
Book Structure
Book Series
Understanding Risk
Risk Management Strategy
Asymmetry
Benefits of Improved Availability
Optimum Reliability Investment
Production and Profitability
Reduced Maintenance and Downtime Costs
Customer Satisfaction and Public Relations
High Consequence Events
Leadership Benefits
Worked Examples
Pump System
Heat Exchanger
Concluding Transition

Overview of Chapter 1

Chapter 1 introduces a central idea of this book: that industrial performance is governed by two fundamentally different types of constraint. Safety and environmental protection define the boundaries within which a facility must operate. These boundaries are non-negotiable and reflect ethical, regulatory, and societal expectations. Within those constraints, the primary determinant of outcome is system availability — the ability of the facility to operate as intended, when required, and to recover quickly from disruption.

The chapter distinguishes between external factors that are largely beyond management control, such as market prices and input costs, and internal factors — reliability, maintenance, operations, and organizational effectiveness — that directly influence availability. It shows how these internal variables shape both production and cost, and therefore overall financial performance.

Although safety and availability both involve the management of risk, they are not the same type of problem. Safety decisions are constrained by ethical considerations and do not lend themselves to simple optimization. In contrast, availability can be analyzed, managed, and improved using economic and engineering principles.

The chapter concludes by outlining the structure of the book and introducing two worked examples that are used throughout to illustrate key concepts.

In other words:

Safety defines the boundaries.
System availability defines the outcome.

Subscribe now

This post is the sixth in a series to do with process safety and artificial intelligence. Previous posts are:

1. Artificial Intelligence in Process Safety Management

2. Artificial Intelligence and Process Safety: Helpful ― Not Authoritative

3. Artificial Intelligence and Process Safety: Understanding Paradox

4. Process Safety and AI: Who Will Pay?

5. Artificial Intelligence and Process Safety: Accountability Cannot Be Automated

In this post we start a series on the theme ‘The Process Safety Singularity’.

The word singularity is dramatic. It suggests black holes, time travel, and a future in which familiar rules no longer apply.

That is not the sense in which the word is used in this series.

In physics, a singularity is associated with a point at which current theories cease to provide a complete description of reality. For example, around a black hole, the known laws of physics, specifically general relativity, work well. But at the singularity itself, the theory reaches its limit. An Event Horizon — the boundary around a black hole beyond which nothing, not even light, can escape — marks the point of no return.

The singularity/event horizon image is useful for thinking about artificial intelligence and process safety, although ― like all analogies ― it should not be pushed too far.

Process Safety Singularities and Event Horizons

In the extreme, a process safety singularity would occur when AI takes over decision making, including decisions that involve ethical judgment, risk tolerance, and management accountability. A deliberately tongue-in-cheek definition for this type of horizon would be,

A process safety event horizon is the boundary beyond which no one — not senior managers, not process safety professionals, not operators, and not regulators — can understand or control how the process safety management program is designed, structured, or managed. Once crossed, the program is outside meaningful human control.

That is not the scenario being discussed in these posts. It is a warning image.

A more practical definition would be on the following lines,

A process safety singularity occurs when AI-enabled systems change the identification, interpretation, and management of process safety risk faster than the organization can understand, validate, or implement those changes.

Managers remain accountable. The challenge is whether that accountability can be exercised effectively when AI systems influence decisions in ways that are difficult to inspect, explain, or challenge.

This definition removes the science fiction flavor. AI does not have to become conscious. It does not have to replace engineers or operators. It does not have to seize control of the plant. The danger is not that AI becomes evil or conscious. The danger is that the organization no longer fully understands how its own risk judgments are being formed.

The more realistic concern is subtler: AI may become so deeply embedded in process safety work that it begins to shape the questions asked, the risks emphasized, the weak signals noticed, and the management priorities selected.

At that point, the organization has crossed an unremarkable but real event horizon. AI is now no longer merely helping people perform familiar tasks. It is changing the way they think.

The process safety singularity will probably not announce itself. It will not arrive as a dramatic corporate decision that says, “From this day forward, AI will manage process safety.” More likely, it will arrive through useful tools such as the following.

AI-assisted PHA preparation

• AI-generated incident summaries

• AI review of operating procedures

• AI comparison of audit findings

• AI screening of Management of Change records

• AI-assisted analysis of near misses

• AI-generated dashboards

• AI review of regulatory changes

• AI-supported risk models

• AI agents that monitor data streams and recommend action.

Each step will appear reasonable. Each tool may save time. Each application may improve consistency. Each recommendation may look helpful. Taken together they become a type of singularity.

Over time, the organization may cross a boundary. It may begin to rely on AI not merely to assist process safety work, but to frame the questions, select the priorities, identify the weak signals, interpret the evidence, and shape the management agenda.

However, this does not mean that AI cannot help us understand various process safety ‘event horizons’. It can encourage people to ‘think the unthinkable’. AI points to a boundary beyond which familiar methods of understanding and prediction may no longer be sufficient. That is the sense in which this series uses the term process safety singularity.

Three Potential Event Horizons

This series will examine three possible “singularities,” or threshold changes, in the way AI may affect process safety.

Systems Thinking

Most process safety programs are divided into management elements. AI may help organizations see interactions among those elements rather than treating them as separate compartments.

Dynamic Linkage of Real-Time Data with Risk Analysis

AI may move process safety from periodic review toward more continuous interpretation of operating conditions, regulatory changes, maintenance status, staffing changes, and other signals.

Development of Broader Risk Models

Traditional fault trees and related methods often focus on equipment and instrumentation failures. AI may help create models that also incorporate human factors, organizational decisions, staffing constraints, maintenance backlogs, procedural weaknesses, and management-system degradation.

Each of these thresholds is practical. None requires science fiction. Each represents a possible shift in how process safety is understood and managed.

Conclusion

The phrase ‘process safety singularity’ should not be understood as a prediction that AI will take over process safety management. Nor should it be read as a claim that AI will eliminate fires, explosions, toxic releases, equipment failures, or human error. The physical realities of process safety remain unchanged. Materials still burn, corrode, react, leak, and explode. People still make decisions under pressure. Organizations still face economic, regulatory, and operational constraints.

The singularity lies in the management of knowledge.

AI may allow process safety information to be connected, interpreted, and challenged at a speed and scale that traditional management systems cannot easily match. It may identify patterns across procedures, training records, MOC files, maintenance histories, incident reports, audit findings, and operating data. It may help an organization see connections that were always present but previously hidden by departmental boundaries, document systems, and periodic review cycles.

That is the opportunity.

The danger is that the same tools may begin to shape process safety judgment without the organization fully recognizing the shift. AI may start by helping people answer questions. Over time, it may influence which questions are asked, and which risks receive management attention.

Managers remain accountable for process safety. That principle does not change. What changes is the practical difficulty of exercising that accountability when AI systems influence decisions through complex, dynamic, and sometimes opaque forms of analysis.

At that point, the organization has crossed an event horizon. AI is no longer merely a productivity tool. It has become part of the process by which the organization understands risk.

The correct response is not to reject AI; the correct response is to govern it. Process safety professionals and managers should use AI to broaden inquiry, identify hidden connections, and challenge stale assumptions. But they must also insist on technical validation, operational reality checks, source transparency where possible, and clear human ownership of decisions.

This series will therefore examine three possible event horizons: the movement from element-based thinking to systems thinking; the dynamic linkage of real-time information with risk analysis; and the development of broader risk models that include not only equipment and instrumentation, but also human behavior, organizational decisions, and management-system degradation.

That is why the process safety singularity matters.

Subscribe now

This post is the fifth in a series to do with process safety and artificial intelligence. Previous posts are:

1. Artificial Intelligence in Process Safety Management

2. Artificial Intelligence and Process Safety: Helpful ― Not Authoritative

3. Artificial Intelligence and Process Safety: Understanding Paradox

4. Process Safety and AI: Who Will Pay?

This post discusses professional accountability.

Large Language Models (LLMs) are moving rapidly into areas that were once considered the exclusive domain of trained professionals. These models can review legal contracts, summarize medical records, identify accounting anomalies, draft engineering reports, and prepare regulatory filings. In many cases they can do so quickly, cheaply, and with a high level of fluency.

Not only can LLMs help professionals to become more productive, they can also help non-specialists ask better questions before seeking professional advice. For example, a process safety professional can submit a proposed employment contract to an LLM and receive valuable guidance as to whether they should sign that contract.

Opportunity and Danger

This situation creates both an opportunity and a danger.

The opportunity is clear. Professionals can use these tools to reduce repetitive work, improve consistency, identify omissions, and accelerate analysis. A lawyer can review hundreds of pages of contracts in minutes. A physician can use an LLM to organize patient histories and identify possible drug interactions. An accountant can detect patterns in financial data that would otherwise be overlooked. A process safety engineer can use one of these tools to prepare draft procedures, summarize incident reports, or identify common failure themes across multiple facilities.

Used properly, an LLM can improve the quality of professional work, and drastically reduce the effort needed to complete that work.

The danger arises when assistance is confused with responsibility.

A legal contract still requires a lawyer who understands the law, the client’s objectives, and the implications of the wording. A medical recommendation still requires a physician who understands the patient as a human being rather than as a collection of data fields. Financial statements still require accountants who are accountable for their integrity. Engineering decisions still require licensed professionals who understand both the technical details and the potential consequences of failure.

These distinctions matter because LLMs do not possess judgment in the professional sense of the term. LLMs are just pattern-recognition machines ― they do not think, they do not possess cognition. Nor do they understand ethics, liability, public welfare, or professional duty. They do not possess experience, accountability, or moral agency. They do not lose sleep over a failed design, a lost-time accident, or the pollution of the local waterways. They do not worry about the possibility of criminal prosecution. They merely generate statistically plausible responses based on patterns contained within training data.

The critical point is that responsibility remains with the human professional who accepts, modifies, rejects, or implements the findings and recommendations.

This principle is already embedded within many professional disciplines. A junior engineer may perform calculations, but a licensed engineer approves the final design. A paralegal may draft legal material, but an attorney signs the filing. A medical resident may prepare an assessment, but the attending physician remains accountable.

LLMs should be viewed in much the same way: they can provide highly capable assistance, but not authority. In fact, the more powerful these systems become, the more important professional responsibility may become. The risk is not merely that these systems make mistakes. The greater risk is that professionals gradually stop exercising independent judgment because the AI output appears confident, complete, and authoritative.

Professionalism requires more than technical competence. It requires accountability. And accountability cannot be automated.

At the end of the day, someone, not something has to sign off on the report that was issued or the recommendations that were made.

Process Safety Management

The same principles apply to process safety management. Many process safety activities already contain formal mechanisms that define accountability and professional responsibility. For example,

• A process safety audit may involve large amounts of data gathering, analysis, and report preparation. An LLM can provide invaluable assistance with this work. However, before the audit is submitted to senior management it must be certified by a human auditor.

• An LLM can help prepare or revise operating procedures by organizing information, improving formatting, and identifying inconsistencies. But, before the procedure is issued for use in the field, it requires approval and signature by an Operations Superintendent or equivalent authority who accepts responsibility for the instructions being issued.

• An LLM may assist with incident investigations by organizing timelines, summarizing witness statements, or identifying possible causal factors. Nevertheless, the final report for a lost-time incident must still be reviewed and signed by the lead investigator, who remains professionally accountable for the findings and recommendations.

• A HAZOP (Hazard and Operability) team can use an LLM to help identify deviations, summarize previous incidents, and evaluate safeguards. But the determination as to whether risk is acceptable remains the responsibility of the team leader and plant management.

• Mechanical integrity inspections increasingly use AI-based image recognition or predictive analytics. Yet the decision to continue operating equipment, defer repairs, or shut down a unit still requires engineering judgment backed by professional accountability.

Confidentiality

The use of LLMs creates a new potential problem: if information is provided to an internet-based system, it may enter the public domain depending on the permissions that were given to the LLM.

Increasingly, local LLMs will be used.

But we still come full circle ― if LLMs are used to write and analyze documents, where and when does the human expert fit into the cycle?

That is the topic of a future post.

Subscribe now

This post is the fourth in a series to do with process safety and artificial intelligence. Previous posts are:

1. Artificial Intelligence in Process Safety Management

2. Artificial Intelligence and Process Safety: Helpful ― Not Authoritative

3. Artificial Intelligence and Process Safety: Understanding Paradox

This post considers how the economics of Large Language Models (LLMs) may affect their use in process safety.

Most of the companies providing LLMs offer a multi-tier pricing strategy, including a free entry-level service. The rate for the first paid layer is typically $20/month. (There is very little public information to do with the paid/free ratio, but it is probably around 5%.)

Until now, the companies offering these models have justified their decision to offer a free option for the following reasons.

• It helps establish market presence.

• Users become part of the company’s ecosystem ― the brand becomes familiar. There is a non-financial cost to do with leaving that system.

• As the user needs more capability, he or she is likely to move to a paid level.

The problem with offering free and low cost options is that the company providing the LLM struggles to make a profit. For example, one report states that OpenAI reported $13.5 billion in losses for the first half of 2025, while generating only $4.3 billion in revenue.

For process safety professionals, these costs matter because LLMs are unlikely to remain a near-free utility. If they become embedded in HAZOP preparation, procedure writing, incident analysis, training, or mechanical integrity workflows, then access to high-capability models will become a budgeted professional tool, not a casual convenience.

Financial Reports

It must be stressed that many of the AI companies do not provide public information on their finances, so their reporting is hard to verify. Nevertheless, among the specialist AI companies, expenses appear to exceed revenues by a wide margin.

A further difficulty is that the reported revenue number may be misleading. Some of that ‘revenue’ comes from the provision of services and compute capacity to companies which are also suppliers to the LLM company.

If these companies were just garage-based startups then their lack of profitability would not matter too much. But the figures on both the revenue and expense side are huge.

On-Going Investment

The financial gap has, to this point, been covered by investors who hope to reap immense profits at some time in the future. However, in the year 2026, many of these investors are becoming restless. They would like the AI companies to make a profit sooner rather than later.

Given this background, it appears that the AI companies will be compelled to raise their fees, and they will have to phase out, or drastically reduce, their free offerings.

A simple back-of-the-envelope calculation suggests that they will have to increase the first-tier price to $100 per month ― assuming that such a price increase will not lead to a loss of customers ― a highly unlikely assumption.

The other approach that they could take is to charge ‘by the drink’, i.e., by the number of tokens used, for which there would be a monthly allowance. The difficulty with this metered approach from the customer point of view is that it is not known ahead of time how many tokens a particular prompt may use. Budgeting will become challenging.

Process Safety Implications

The implication for process safety work is straightforward. Organizations should no longer treat AI as a free or low-cost add-on to professional work. If LLMs are used for HAZOP preparation, incident investigation, procedure development, or training, then a cost for this service should be included in budgets. The days of unrestricted free or low-cost access to frontier AI models are probably coming to an end. This means that AI companies will increasingly focus on enterprise customers who can justify higher fees through productivity, risk reduction, and workflow integration.

Caution

There remain, however, two cautionary notes. First, most of the financial reporting from privately-held companies is not publicly verified. Estimates provided in articles such as this one are just that: estimates. (There has been continuing speculation that some leading AI companies may go public in the near future. If they do, their financial reporting will become more transparent.)

Second, financial projections generally assume that demand for ‘compute’ and energy will continue to rise exponentially. It is possible that new generations of AI hardware and software will be considerably more efficient, and therefore less demanding of resources. If so, costs may not be as high as previously anticipated.

Subscribe now

This post is the third in a series to do with the use of Artificial Intelligence (AI) and Large Language Models (LLMs) within the process safety management discipline. The first two posts are:

• Artificial Intelligence In Process Safety Management: A Structured Approach to Decision, Execution, and Analysis

• Artificial Intelligence and Process Safety: Helpful ― Not Authoritative

In this post we consider the role of paradox in Process Safety Management (PSM).

Proper Names

The following quotation is often attributed to Confucius.

The term Artificial Intelligence is not the ‘proper name’ for programs such as ChatGPT, DeepMind and Claude. Specifically, LLMs are not conscious minds; they are not intelligent. They do not possess understanding, judgment, intuition, or experience. LLMs are simply software systems trained to identify statistical relationships between tokens: words, symbols, and patterns contained within enormous data sets. They are powerful tools, but they are not conscious minds.

Attributing the term ‘intelligence’ to such systems is more than misleading ― it attributes a capability to them that does not exist. This means that professionals in all disciplines need to be very cautious about allowing AI to make decisions.

Paradox

This caution applies particularly to process safety because the discipline often involves something deeper than pattern recognition.

Process safety involves paradox.

For example, many incident reports identify the lack of clear, comprehensive operating procedures as being an important contributing factor to the incident in question. The resulting recommendation is that operating procedures should be thorough, complete, accurate and up to date. The procedures should also anticipate as many operating situations as possible.

The catch is that no written procedure can describe every possible operating condition, equipment failure, weather event, utility disruption, or human interaction. Real facilities are too complex.

Therefore, safe operation requires two apparently opposing behaviors:

• Strict procedural compliance,

• Intelligent adaptation when reality differs from the procedure, often during an emergency.

This is more than a balancing act. The very discipline that prevents unsafe improvisation can also become dangerous if it prevents intelligent response to conditions the procedure did not anticipate.

An experienced process safety professional understands this paradox, and can work within it. An LLM, on the other hand, may have trouble responding to this type of challenge because the situation will contain multiple valid but conflicting objectives that cannot be resolved through straightforward pattern matching.

In other words,

Can LLMs handle novel conditions?

LLMs do not reason about the world in the same way humans do. They do not possess direct experience, tacit operational understanding, or physical intuition. They generate responses based on statistical relationships learned during training. This means that they are strongest when the domain is well represented in training data, the problem resembles prior examples, and the structure of the problem is stable. They are weaker when conditions are genuinely novel, physical interactions are poorly represented, or the problem requires deep understanding of context and consequence.

Bhopal

Intuition

Experienced operators and engineers often recognize danger before they can fully articulate the reason for their concern. They have hunches based on anomalies such as,

• An unusual sound

• An unexpected trend

• A subtle operational inconsistency

• An interaction that ‘doesn’t feel right’.

This type of judgment is difficult to formalize because it often depends on tacit understanding accumulated over years of direct operational experience. Hence, it is very difficult to incorporate hunches and feelings into an LLM’s training.

More Paradoxes

Here are some more PSM paradoxes.

Too Much Data

• Modern facilities generate enormous quantities of data.

• More data should improve understanding.

• But excessive data can obscure critical signals.

• Important weak indicators become buried.

• Organizations become ‘data rich but insight poor’.

Knowledge

• The more organizations know about hazards, the more they realize how much uncertainty remains.

• They learn that rare interactions, abnormal conditions, organizational behavior, and human adaptation cannot be fully modeled.

• Hence increased expertise often produces greater humility rather than greater certainty.

• That characteristic sharply contrasts with many AI systems, which often generate responses with high apparent confidence regardless of actual uncertainty.

Conclusion

LLMs are invaluable tools for organizing information, identifying patterns, generating procedures, and reviewing large bodies of technical material. But process safety management also involves judgment under uncertainty, recognition of anomalies, and an understanding of paradoxical conditions. These capabilities remain deeply dependent on human experience and operational understanding.

Here is a final paradox: the safer a facility appears to become, the greater the risk that vigilance will decline. Long periods without incidents may create confidence, normalization, and complacency.

In other words,

Safety success can itself become a source of vulnerability.

Subscribe now