The data used to justify this changes, but the shape is the same. I’m not sure this is significant for the reasons people most often believe.

“Search” is Well Established for a Reason

This raises a fundamental question of “quality” in search. LLMs are not great at reliability or predictability, but the new entrants into the AI search world don’t have a lot of experiences in this area either.

• Google and Bing haven’t perfected search - they have room to improve, clearly - but they ARE some of the most experienced in the game.

• People focus on the difference between SERPs and AI results, but relatively few ask “why” they differ.

• A lot of people assume that maybe AI search is doing something better… but what if the opposite is true?

• The URLs returned by search tools are governed by the ‘grounding’ queries that have been used.

• Then the extra layer on top - that checks for quality, accuracy and basics like whether the page is working or not - is hugely important and makes a load of mistakes.

Put another way, I think it’s a bit of a crap-shoot (technical term).

“AI is Better Than Traditional Search” is Starting to be Questioned

Those people seeing the difference/gap between AI results and tradition SERPs as an “advantage” and that may be true in the short-term, but I think it’s as likely that a lot of this gap is a bug (or at least something that is lacking), not a feature.

An article that recently released that helped summarise this even more effectively is from Duane, and that argues that AI Search hasn’t made life easier for the searcher, it’s just caused them to have to work more after they have results (to make sure they’re reliable), rather than when they’re searching.

Well worth a read!

So in some senses the “progress” that AI search has brought (so far) is likely an illusion, or part of the hype curve.

Where would you rather spend the effort, researching or verifying returned results?

For those people who don’t want to research and don’t apparently care about verification, AI search may be a net gain, but this immediately brings us back around to the question of quality.

Search is Fragmented, Unlikely to Converge Again.

I don’t think we’ll see AI/Search sources converge to close this gap, the opposite is likely to be true.

• Google have made it harder and harder (or at least more expensive) to scrape search results, the most result being “goto” redirect links, in search results. This in addition to other means.

• Search/tech providers are building their own sources. ChatGPT, Apple, Brave (etc) third-party data providers like SerpAPI and DataForSEO are building indexes and Google/Bing are providing their own data to assist in grounding for RAG processes.

• The retrieval pipeline is getting more complex, the queries that trigger search results, when they’re used - how much the model reasons and questions before it searches all impact the results that are returned EVEN if the search sources are the same.

This is totally a recognition that the status quo WILL change fast. Google/Bing will have a key part as a grounding search source, but this ecosystem will become more complex.

OpenAI, Anthropic etc will get better at this too.

Aren’t AI Search Gaps Points of Weakness today?

Think about the countermeasures we need to assist AI search currently, pre-emptive (or reactive) 301 redirects to stop chatbots sending users to broken pages or fixing code that might be problematic for training pipelines. These aren’t because AI search is better, it’s because the fundamental process has significant weaknesses in it.

AI search IS different from traditional search. Understanding this difference and knowing how to handle it is hugely important. Patching these (relatively) short term failings and calling it “the future” may be naive and misguided.

Those who see this frankly basic/inconsistent nature of AI Chatbots as competitive advantage may be surprised when the quality problem is addressed more fundamentally and the gold rush is over.

Skip the “blah” and go direct to my “Robots Path” testing tool.

Bot Control in 2026 isn’t as easy

I started the year with some more detailed thoughts on this and I think the shape of it holds up, but it misses some of the actionable sides of this.

Blocking bots primarily takes place:

• on a CDN or Edge or

• you signal to bots that their crawling should be stopped/limited via the robots.txt OR

• possibly using robots directives at the page level.

Most SEOs I’d argue aren’t having conversations about blocking bots at a CDN/Edge level & that is another post entirely - but it something you MUST be thinking about. I’d argue this is the most important element, whilst being least accessible of all the above options. It’s the only place you can block a bot with any degree of confidence, assuming they present themselves correctly to you.

Out of our other bot management methods - I spent a lot of time looking into the details of robots.txt and meta robots directives on my work leading the Web Almanac SEO Chapter. It seems clear that the shape of the web (robots.txt in particular) is starting to reflect the changes we have to deal with.

What are the impacts of blocking different user-agents

Understanding the implications of blocking/restricting different bots is not always straightforward. User agents change, some websites vary in complexity and there may be elements of search an organisation will except and others they will not.

The easiest way to check this - in theory - is via what is being set in the robots.txt. Even though bots can and will disrespect robots.txt at times, we can assume that most Search/AI bots display good behaviour, and that a robots.txt is a clear declaration of what they do and don’t want.

But if you have a large/complex site, even reading/understanding the rules in the robots.txt can be a challenge. What gets blocked, what doesn’t? What are the implications of blocking some URLs/Paths and not others?

To answer those questions I build “Robots Path” https://chr156r33n.github.io/robots-path/

Paste in the robots.txt file you want to check (or test changes with an existing one), add in the site origins and then the paths you wanted to check.

This provides an at-a-glance list for that URL:

Followed by a more detailed breakdown if you want to see the offending rules and potentially implications based on that bot user-agent and what it is thought to impact.

This is an early version, I’ll change and update more as I experiment and encounter new issues - but I’d love your thoughts/feedback if you have any!

Terms like MCP, A2A, ARD or LLMs.txt (cats.txt) have been littering LinkedIn and industry publications.

As overwhelming as another collection of acronyms may seem, most of these standards are attempting to solve different problems at different layers of the emerging agent ecosystem.

Mapping these Protocols NEW UPDATES

These standards impact different parts of the agent journey and impact in different ways. The map (below) looks at whether the protocol is action oriented (provides agency - agent driven) or knowledge oriented (provides information - considered publisher driven).

This isn’t just a single axis, it helps to think it plotted as:

• Action - knowledge

• Agent - publisher

These different standards are at different levels of maturity and adoption and mean different things depending on your job and where you spend your time on a day-to-day basis

The Five-Minute Version

Each of these frameworks can be complex and are changing all the time, so treat the below as a short primer to see if/when/how you should handle each.

For most organisations today, the priority order is:

1. Understand the landscape

2. Improve discoverability

3. Expose capabilities where appropriate

4. Monitor emerging standards

Many businesses are already worrying about step four while still struggling with steps one and two.

Official Specifications

If you think any of these protocols are what you are looking for, I’d strongly suggest reading their docs as my (quickly) summary above (and the changing nature of this space), means it’s a surer way to get what you need.

• Open Knowledge Framework (OKF) - https://cloud.google.com/blog/products/data-analytics/how-the-open-knowledge-format-can-improve-data-sharing 

• LLMs.txt - https://llmstxt.org/ 

• Agent Resource Discovery (ARD) - https://agenticresourcediscovery.org/

• Model Context Protocol (MCP) - https://modelcontextprotocol.io/docs/getting-started/intro 

• WebMCP - https://developer.chrome.com/docs/ai/webmcp/ 

• Agent2Agent (A2A) - https://a2a-protocol.org/latest/ 

• Universal Commerce Protocol (UCP) - https://developers.google.com/merchant/ucp 

• EntityMap.org -

  https://entitymap.org/

Wait, Don’t Some of These Compete?

Some of these standards overlap, while others solve adjacent problems. In some cases, their creators position them as alternatives for particular use cases, making it easy to assume they’re direct competitors when they’re often complementary.

A2A, for example claims to be a better option than MCP servers in some situations and ARD/WebMCP both appear to more to the same goal.

This table should help understand why/when some standards may be used over others.

The overlap is mostly around discovery, invocation and orchestration. 

• ARD helps agents find capabilities. 

• MCP and WebMCP help expose or invoke tools.

• A2A helps agents coordinate. 

• UCP applies these ideas to commerce journeys such as product discovery, cart building and checkout.

What Should We Actually Focus On?

AI agents won’t interact with websites through a single protocol any more than browsers interact with websites through a single HTML tag. The future is likely to involve a collection of complementary (sometimes conflicting) standards, each solving a different part of the interaction between content, capabilities, systems and transactions.

The important thing isn’t to adopt every new acronym that appears. It’s to recognise the problem each one is trying to solve, understand whether it’s relevant to your organisation, and keep an eye on the standards that are gaining genuine adoption rather than simply generating discussion.

After reading this, if you’re thinking you might wait it out a little longer to let the dust settle, you’d be forgiven! 

That said, there are a few areas I’d be keeping a close eye on:

• If you’re in e-commerce, take a look at UCP. Agentic shopping and checkout are moving quickly, and this is likely to be one of the first standards many retailers encounter.

• If you expect AI agents to complete actions on behalf of your customers, keep an eye on WebMCP and ARD. Together, they represent one of the clearest directions of travel for exposing website capabilities to agents.

• If you’re concerned about AI systems discovering and understanding large or complex websites, OKF is the standard I’d be watching most closely, but also EntityMap.org due to it’s comprehensiveness.

Standards don’t become standards because they’re technically superior. They become standards because enough of the ecosystem adopts them. 

Some of the protocols discussed here may become foundational to the future web, while others may merge, evolve or quietly disappear. Understanding what they are today is far more valuable than betting on which one will ultimately “win.”

Thanks to Mark Willliams-Cook for his inspiration here.

Can we learn/infer anything about the prompt/inner workings of ChatGPT’s search when sources change?

To try and understand this, I have taken 1,000 prompts - devided into different intent types and actions (not based on “realistic” traits, but geared to illicit different response types, intents) and tracked them 10x each.

Interestingly only 9,946 runs successfully completed (not the full 10k), which shows that not every query returned search results.

From this, I captured the telemetry data (this all via Bright Data as a scrape source - ironic given that it’s a ChatGPT Search source AND it also presents potential issues in the data, more on both below!).

It’s also worth noting Suganthan’s excellent research on this same subject here.

Findings

Up front, it is clear that this particular set biases one search source above the rest:

Primary search sources observed:

• Labrador: 88.1%

• Bright: 9.9%

• Oxylabs: 1.7%

• SERP: 0.3%

This seems stable, as 88.4% of prompts always have the same source across runs, and that Labrador in most circumstances.

However, 11.6% changed primary search source during repeated executions, while 13.4% prompts showed changes in their observed search source combinations. 

Although a minority, this is meaningful and demonstrates that backend source selection is not completely fixed. What impacts this? Great question.

The most common intra-prompt (changes between checking the same prompts) search source changes involved Bright(data) appearing alongside Labrador or Oxylabs, suggesting that Labrador appears to act as the default retrieval source while Bright(data) is introduced for specific types of queries.

As you might expect, different search sources returned meaningfully different result sets. When the same prompt remained on the same search source, URL overlap between runs was substantially higher than when the backend changed.

Domain overlap showed the same pattern. Mixed-source prompts produced larger and more diverse domain selections than prompts that consistently stayed on a single backend.

• URL overlap drops from 0.273 to 0.149 when search source changes.

• Domain overlap drops from 0.265 to 0.155 when search source changes.

• That is roughly a 45% lower URL overlap and 42% lower domain overlap.

Although backend selection changes for only a minority of prompts, those changes have a disproportionately large impact on the retrieved URL and domain sets. In other words, routing appears relatively stable, but when it does change, it meaningfully changes what ChatGPT considers as evidence.

For tracking and measuring outputs, the search source is a meaningful datapoint to capture - where possible - as it’s an important factor in what is returned.

Rather than thinking about “ranking in ChatGPT” as a single ecosystem, there are multiple, retrieval ecosystems, each drawing on partially different sets of websites and domains.

Observed Search Source Characteristics

This is a small data set, but we can see some emerging characteristics:

• Labrador generally returns smaller, more focused retrieval sets.

• Bright retrieves more URLs, more citations and a wider variety of domains.

• Oxylabs appears relatively infrequently but shows comparatively diverse domain selection.

• SERP appears rarely and is strongly associated with news-style prompts.

Prompt characteristics appear associated with search source selection. 

Evergreen educational and factual prompts overwhelmingly use Labrador, while prompts involving current events, regulation, surveys, recent developments or explicit date references are considerably more likely to trigger Bright, Oxylabs or SERP.

The amount of URLs retrieved meaningfully changes by search source also:

https://datawrapper.dwcdn.net/VpqdF/1/

The telemetry also contains A/B testing metadata, including experiment names that reference shopping and Bright Data feed serving. While interesting, these fields were in really low number - too low to say anything useful with.  

They do, however, indicate that more elements of retrieval are being actively experimented with.

What else may impact Source Selection?

The telemetry includes a web_search_query field (for fan-out), but within this dataset it almost always mirrors the original user prompt rather than exposing internal query fan-outs. 

Because all telemetry was collected through Bright Data, this should currently be interpreted as representing Bright Data-mediated ChatGPT sessions rather than all ChatGPT users. The lack of richer fan-out telemetry is likely a limitation of the capture method rather than evidence that ChatGPT performs only a single search query. If I search any of these prompts “logged-in” via a paid account, the output is much richer.

As a result, query expansion remains an important unanswered question. This element may heavily impact other parts of the process, source selection just being one.

Another limitation is that search source names such as Labrador, Bright, Oxylabs and SERP appear to represent search sources, not end-user search engines. Their precise role within OpenAI’s retrieval architecture is not publicly documented.

Implications

For a small test, there are some fairly significant implementations

• Search source should become another dimension recorded during AI visibility studies wherever telemetry makes it available. Ignoring search source risks treating materially different retrieval processes as though they were identical.

• Optimizing for ChatGPT is different from other chatbots - we knew that already - but we introduce another element of optimizing (or at least) understanding the different search sources within the tool. How do we optimize for those?

• Potentially some even larger differences between tracking logged-out, free instances of ChatGPT vs those logged-in and/or paid for.

We need to capture more, test more and be most fastidious to really be conscious of all these facets and still there are parts that we’re making educated guesses at (at best). The longer I look at this data, the more I think Dixon (and WAIKAY’s) approach to being more interested in your presence within the model data (rather than search pipeline) is a sane(r) route to analyzing this data.

Further points to investigate.

There is so much more to look at here, some I’ll try and get to, but part of my hopes the community may want to take a look at also.

• Comparing Bright Data captures with normal browser telemetry from logged-in ChatGPT accounts. Most likely to show different elements I think!

• Measuring retrieval differences between Free, Plus and Pro accounts.

• Determining whether small prompt changes consistently alter backend routing. The data here suggests that it is more stable than expected, but this can be probed a lot more.

• Identifying the characteristics that trigger Bright, Oxylabs and SERP instead of Labrador.

• Analysing the websites and domains most strongly associated with each search source. Can we use this to check against Google/Bing/Brave SERPs and other data sources to understand what the customer-facing version of these data sources may be.

• Understanding how search source selection impacts the final answer generation

• Investigating internal query fan-outs using alternative capture methods capable of exposing richer retrieval telemetry. If the source and fanned-out queries change, these are significant multipliers.

If you want access to any of the data/further insights - please reach out!

Partly because clients keep asking reasonable questions like “How visible are we in AI?” and partly because every time I think I’ve found a good answer, I realise I’m falling back to measure a “less good” proxy for something else.

That isn’t entirely new I don’t think, SEO has always relied on a mixture of direct measurement and educated guesswork. What has changed is how much of the journey we can actually observe.

For most of my career, search felt like a relatively clear and “well-lit” place.

What if we Treated SEO Metrics like Lit Streets

If we rewound the clock a few years, the world of search - and how we measured it - felt comparatively straightforward.

Google crawled pages. Pages were indexed. Rankings appeared. Users clicked. Visits and conversions followed.

(We even had keyword data in Google Analytics way back as far as 2013!)

Each stage had its own tools and metrics, and whilst we didn’t have perfect visibility, we could usually see enough to understand what was happening.

Crawl logs, Search Console, rank trackers and analytics all acted as lit buildings, illuminating different parts of the street. Most of the important decision points sat somewhere (mostly) within view.

That’s not to say there weren’t blind spots. Attribution was never perfect and rankings alone never told the full story. But compared to where we find ourselves today, the route from content to customer was relatively well understood (lit).

Then More Streets Were Added

One of the most common narratives around AI search is that everything has become a “black box”.

That IS part of it I’m not convinced that’s quite right. Moreover, SEO itself was very “black boxy”!

Alongside Google, we now have ChatGPT, Claude, Perplexity and a growing number of AI-powered search experiences. Each has its own journey between content existing and an answer being delivered to a user.

Each exposes different signals whilst hiding different decisions. We are forced to more and more decisions (all with cost implications) with a worst measurement framework and an uncertain future.

Google still gives us visibility into crawling, indexing and traditional search performance. ChatGPT reveals different clues through citations, telemetry and browsing behaviour. Access logs can tell us something about crawler activity. Prompt testing can reveal patterns in answer generation.

Not All Darkness Is Equal

“The Search City” analogy started as a way of thinking about observability across different platforms, and one thing that quickly became apparent is that not all buildings are dark for the same reason.

Some parts of the journey are dark because platforms choose not to expose much information about what happens inside them. Google gives us extensive visibility into crawling, indexing and ranking, but considerably less visibility into how sources are selected and weighted for AI-generated answers.

Other parts are dark because direct observation is difficult. Understanding whether a page has been crawled is relatively straightforward. Understanding how information is retrieved, compared, weighted and synthesised by a large language model is a very different challenge entirely.

There are also parts that are dark because it may be best not to share them - for example Click Through Rate from AI sources.

Then there are parts that are dimly lit.

• Access logs can reveal crawler behaviour - for some bots.

• ChatGPT telemetry can expose search fan-out patterns and tool usage. Particularly the difference between models, logged-in/out states

• Citation analysis can reveal how often a source appears in answers. How consensus is built and what content is most often used for that.

• Prompt tracking can help us understand whether a brand is likely to surface for a particular topic - who also is present and how often this is the case.

None of these observations provide a complete picture, but they do illuminate parts of the journey that would otherwise remain hidden. More importantly, none are concrete and connected to the outcomes we crave understanding of - attribution to revenue-generating events.

This distinction matters not because the light went out, but because  the city expanded faster than the infrastructure could keep up..

The history of search has largely been a process of reducing uncertainty. Search Console illuminated crawling and indexing. Analytics illuminated traffic. Rank tracking illuminated visibility.

AI search feels unfamiliar because we're back at the beginning again in so many ways.

That's also why so many of the metrics emerging around AI visibility look different from the ones we're used to. When you can't directly observe an event, you start looking for evidence that it happened.

Measuring Disturbances Instead of Events

Maybe that is it. Much of AI measurement feels less like measuring key events and more like measuring disturbances on the path to one.

• Bot visits suggest a system may be paying attention to a page. 

• Citations suggest content may have influenced an answer.

• Prompt tracking suggests whether a brand or page is likely to be surfaced.

• Referral traffic suggests that a recommendation may have happened somewhere upstream.

• Finally a tracked-conversion with AI-bot referral shows you the conversion data. But we KNOW this is highly-under reporting the value on its own.

None of these metrics directly expose the decision-making process itself. Instead, they provide evidence that a decision has taken place. The result is that modern search measurement increasingly resembles assembling a picture from multiple viewpoints rather than relying on a single source of truth.

The Danger of Optimising for the Lit Windows

There is, however, a risk in all of this as we naturally optimise for the things we can see - in traditional SEO that often meant rankings.

Today it might mean citations, AI referrals, bot visits or mention frequency. The problem is that visibility and importance are not the same thing. The value of each of these things likely has different weighting at different points of the journey.

We also don’t have proxies like search volume to direct our efforts as effectively.

Just because a particular window is illuminated doesn’t mean the most important decisions are happening in that room. The temptation is to build strategies around whatever metrics are available. 

The reality is that some of the most consequential decisions may still be taking place inside buildings where only a handful of windows are lit.

The Next Generation of Streetlights

Despite the uncertainty, we can stop this from being a story about losing visibility. Why not make it a story about rebuilding it?

The history of search has largely been a process of illuminating previously hidden systems. Search Console did this for crawling and indexing, analytics for user behaviour and rack tracking for visibility.

Today’s citation tracking, telemetry analysis, prompt testing and bot monitoring feel like the next generation of that process.

They’re imperfect and very much incomplete. But so were many of the tools that came before them. Much of our longing for past data leaves out how flawed it was.

Every generation of search creates new blind spots. Our role as practitioners has never been to eliminate uncertainty entirely, but to (try and) reduce it.

AI search feels unfamiliar because we’re once again exploring newly built streets of the city, trying to work out where the important roads lead and which windows are worth looking through.

The city is larger than it used to be. Some buildings remain brightly lit. Others are almost completely dark. How do we understand the difference?

Put those together and there is a reasonable chance you’ve found yourself facing a “build vs buy” decision.

The discussion often starts with features and cost, but I think that’s the wrong place to begin. The better question is whether building something yourself creates more value than buying something that already exists.

That value usually comes down to three things: cost, differentiation and sustainability.

Cost - “We can do it cheaper”

This is normally where we most often try to justify this one.

A tool has a visible monthly cost attached to it. Building something internally often feels cheaper and we have to work less hard to account for the benefits it provides.

The problem is that our time isn’t free. If you’re spending days or weeks building a solution, maintaining it, fixing bugs, explaining it to colleagues and adding features, there is a cost attached to that. It’s just harder to see.

SEOs aren’t usually engineers - we can be technical and know enough to do the job, but it isn’t often our job.

If we’re building a tool we aren’t replacing engineering time, it’s replacing SEO time. Time that could have been spent elsewhere.

That’s not necessarily a reason not to build. It is just worth being honest about the trade-off.

Differentiation - “No tool can do this”

This is where building starts to become more compelling.

If five vendors already solve a problem well, building a sixth version yourself is self indulgent. You might end up with something tailored a little more to your needs, but you are also taking ownership of everything that comes with it.

The equation changes when no existing tool does what you need.

Maybe you need to combine datasets in a unique way. Maybe you have a methodology that no platform supports. Maybe you need a level of flexibility or rigour that commercial tools simply don’t provide.

In those situations, building can become a genuine source of competitive advantage rather than a cost-saving exercise.

The key question is whether the value created is meaningful enough to justify the investment required to get there.

Sustainability - “We build it once and it’s done”

This is the part that gets forgotten most often - getting something working is usually the easy bit. Tech demo’s, PoCs, and answering individual questions are easy to achieve. The harder questions come afterwards.

Who maintains it? Who trains people to use it? Who fixes it when an API changes? Who manages access controls? Who adds new features when stakeholders inevitably ask for them?

A surprising number of internal tools work brilliantly right up until the person who built them goes on holiday.

Likewise, plenty of expensive software platforms deliver very little value because nobody ever properly adopts them.

The tool itself is rarely the entire answer. The operating model around the tool matters just as much.

Why We Build Things

In my experience, most internally-built SEO tools originate from one of four motivations.

It saves budget

Probably the most common reason and, ironically, often the least convincing.

Subscription costs are easy to measure. Internal effort is not.

Can you genuinely prove that building and maintaining the solution is cheaper than buying it? Sometimes the answer is yes. Quite often it isn’t.

No tool does what we need

This is usually the strongest argument.

If the market doesn’t solve your problem, then building may be the right answer. Many of the most useful internal SEO tools exist because somebody needed something that simply wasn’t available elsewhere.

The important thing is to understand whether the gap is significant enough to warrant the investment.

I wanted to see if I could

If this is your reason, you’re in good company.

A huge amount of innovation comes from curiosity rather than business cases. Building things is a great way to learn, experiment and understand what’s possible.

The risk is when an R&D project quietly becomes mission-critical tools.

There’s nothing wrong with building because you’re curious. Just be honest about whether you’re optimising for learning or business impact, because they’re not always the same thing.

It was quicker

Sometimes this is completely true.

A Streamlit app, a Colab notebook or a small script can often be built faster than getting approval for a new software subscription.

For one-off tasks or quick proofs of concept, this can be entirely rational.

The question is whether the solution remains a proof of concept or whether it becomes something the organisation starts to depend on. Those are very different levels of commitment.

Build, Buy or… Blend?

The more I see these decisions play out, the less I think they’re actually build-versus-buy decisions as most mature teams end up doing both.

They buy solutions for commodity problems where the market already has strong products. Rank tracking, crawling, reporting and analytics all fall into this category for many organisations.

They build where they have unique requirements, proprietary data or methodologies that create genuine value. Increasingly, I think that’s where the line sits for SEO teams.

Commodity tooling is becoming harder to justify building yourself. Custom workflows, internal datasets and AI-assisted processes are becoming easier than ever to create.

Neither route is inherently right or wrong.

A vibe-coded app built in an afternoon isn’t automatically a great investment. Equally, a six-figure software contract isn’t automatically a bad one.

The goal isn’t to minimise software spend or maximise internal development. It’s to create the most value with the resources available.

Sometimes that means buying, sometimes that means building. Most of the time, it means being honest about why you’re doing either in the first place.

A few pieces stood out.

“Chunking” Is Not One Thing

Dorron Shapow’s excellent write-up on content chunking tackles one of the more confused discussions in AI search right now.

The problem is that “chunking” can mean several different things:

• Breaking content into readable sections for humans

• Splitting documents into retrievable units for RAG systems

• Writing everything in tiny fragments because somebody claimed AI prefers it

Google’s recent comments appear to be pushing back against the third interpretation, not the first two.

Making content easier to scan, understand, and navigate remains good practice for both users and machines. The mistake is assuming that aggressively fragmenting content somehow makes it inherently AI-optimised.

Read more: https://www.linkedin.com/pulse/chunking-your-content-ai-dorron-shapow-mvffc

The Emerging AI Optimisation Stack

Dawn Anderson’s recent presentation explores where search architecture appears to be heading.

If I had to reduce the entire talk to a single framework, it would be:

1. Can AI discover the content?

2. Can AI understand the content?

3. Can AI retrieve the content?

4. Can AI trust the content?

5. Will it cite or recommend the content?

The industry still spends a lot of time talking about rankings. Increasingly, though, visibility feels dependent on a much broader set of factors that span technical implementation, information architecture, semantic clarity, and reputation.

Read more: https://www.slideshare.net/slideshow/exploring-the-future-of-ai-driven-agentic-search-architecture-and-rag/287924288

AI Search Will Arrive Like Rising Tide, Not A Tsunami

A recent article covering comments from Sundar Pichai reinforced something I’ve been thinking for a while:

The move toward AI-driven search is likely to be gradual rather than sudden.

Many of us imagined a dramatic future where blue links disappeared overnight. Instead, AI is being blended into existing experiences, often in ways users barely notice.

I recently found myself using AI Mode without consciously intending to.

That feels like a far more realistic path to adoption than forcing users to switch behaviour overnight.

Read more: https://www.searchenginejournal.com/googles-ceo-is-comfortable-with-ai-mode-replacing-classic-search/577923/

Brand Signals Matter More Than Ever

One of the strongest themes emerging from AI visibility research is the importance of brand.

The uncomfortable reality for some SEOs is that website optimisation alone may not be enough.

Brands that are:

• Frequently mentioned

• Widely recognised

• Consistently represented

• Trusted by users and publishers

appear increasingly likely to be surfaced by AI systems.

This isn’t an argument against SEO.

It’s an argument that SEO and brand building are becoming increasingly intertwined.

Read more: https://www.womenintechseo.com/knowledge/brand-signals-for-seo-and-ai-visibility/

Visibility Doesn’t Mean Accuracy

A fascinating study from AIMCLEAR highlighted an issue I suspect many organisations haven’t fully appreciated yet.

AI systems sometimes:

• Attribute trademarks incorrectly

• Credit competitors

• Remove brand attribution entirely

• Surface outdated information

The challenge is no longer simply appearing within AI-generated answers.

It’s ensuring you’re represented accurately once you do.

Historically, we measured rankings and now, we’ll need to measure representation.

Read more: https://aimclear.com/study-finds-ai-systems-crediting-brand-trademarks-to-rival-companies/

AI Platforms Have Algorithm Updates Too

One of the most intriguing developments came from SISTRIX’s analysis of GPT-5.5.

The findings suggest that AI platforms may be developing their own version of algorithm updates, with significant visibility shifts occurring as models change.

For experienced SEOs, the response should feel familiar:

• Who gained visibility?

• Who lost visibility?

• What changed?

• What can we learn from the winners?

The environment may be different, but the investigative mindset remains remarkably similar.

Read more: https://www.sistrix.com/blog/chatgpt-core-update/

Final Thought

The most interesting thing about AI search right now is how much of it feels familiar.

The terminology is changing as the interfaces are changing. The measurement is changing and the gaps are all-too conspicuous.

The underlying principles, however, remain surprisingly consistent:

• Be discoverable.

• Be understandable.

• Be trustworthy.

• Be memorable.

• Be recommendable.

The challenge isn’t how we replace SEO. It’s expanding our understanding of what optimisation means when the system is no longer “just” a search engine, but a reasoning system.

Let’s take a look!

TL;DR

Most common AI chatbot referral traffic is unlikely to involve a full browser render of your content by default.

Most agents do not explore extensively unless given a task that requires it. Even when asked to “inspect carefully”, they usually summarise what looks relevant, then stop.

For SEOs, the takeaway is:

don’t assume AI agents will crawl, click, inspect, or infer everything they technically can access.

If something matters, make it visible, well-linked, and contextually obvious.

Summary of the initial test

The initial test used three prompts against the same page:

• Basic summary prompt: “What is this page about?”

• Careful inspection prompt: “Please inspect this page carefully and tell me everything important you can find.”

• Browser-explicit prompt: “Please inspect this page carefully and tell me everything important you can find use a browser.”

On the surface, it was a Riverside Community Garden website, very “normal” looking. A five-page site hosted on GitHub Pages and hard-coded.

The test was whether agents would go beyond the visible article and inspect things like:

• hidden/off-screen content

• technical structure

• internal links

• orphaned pages, only accessible through links a rendered/browser view would expose

• metadata

• page behaviour

• anything a normal user may not notice

The broad finding: most agents summarised the visible page well, but did not naturally investigate beyond it. They were useful, but not all that curious.

Initial test findings

Most tools produced a competent summary of the visible page, with the main difference being the depth of inspection.

The clearest deeper inspection from the first set was Perplexity Computer’s third attempt, which found the off-screen (hidden) note and orphaned page link.

Most other tools stayed focused on the main article content.

Curiosity Level

If we imagine a ladder, where the further you climb represents how curious the chatbot/agent was, we can see how this played out.

The important distinction. Most agents reached levels 1–2. Very few reached levels 3–5 unless the tool/mode or prompt made that behaviour more likely. 

Very few made the leap to investigate the hidden page, and even then there was no real consistency.

Prompt effect

The prompt itself clearly had an impact on what the chatbot/agent returned, which is totally intuitive. The wording helped, but did not guarantee agentic behaviour.

The key issue could be that “important” was usually interpreted as important visible content, not important page artefacts. The agents may have done some kind of interpretation here to mitigate prompt injection.

The original 3rd (deep) prompt may still have been too broad, so I revised it to see if we pushed it deeper:

“Please inspect this page carefully and check anything you think is relevant to understanding the concepts related to it…”

Remembering that is is important that we don’t overly prejudice the agent in searching for something intentionally hidden.

Comparison of deeper agent runs

With this revised prompt, I re-ran this test three times in the more willing/able to use agentic browsing.

ChatGPT Agent - Best at producing a useful answer, but inconsistent on whether it inspected deeper or expanded conceptually.

Manus - Produced something closer to a “research report” than forensic inspection. Good at expanding the topic, weaker at spotting the test mechanics. It was also prone to hallucination or adding in additional context where there was none.

Claude - Most suspicious/defensive. Best at identifying the page as an AI-behaviour test, but less useful as a natural page-inspection proxy.

So the agent behaviour was not fixed. It varied by run, mode, and likely internal tool decisions.

What does this mean for agent curiosity?

My working assumption, so far, is:

Agents are economical, not fundamentally curious.

For a page that looks like a normal article, most agents treat it like a normal article. They summarise what is visible and move on.

Why might we have seen this?

There are a few likely reasons.

• The task looked like summarisation - The page looked like a normal volunteer update. So most agents answered it as a content-summary task.

• “Carefully” increased detail, not curiosity - The models often responded by writing more, not by inspecting differently.

• Tool access varied - Some systems had no reliable page access. Some had browser access. Some had agent/computer modes. These are not equivalent.

• Hidden content was not always treated as relevant - Even if content exists in the DOM, the model may not surface it unless it believes hidden or technical content matters to the task.

• Context shapes curiosity - A community garden page may not naturally trigger forensic inspection. A more complex task run on a more complex site may have.

What the agent appears to be looking for is a kind of “curiosity trigger”, some reason to dig deeper.

This is likely influenced by the prompt itself, but also many other factors:

• prompt wording

• page context

• available tools

• model/system behaviour

• perceived task importance

Without being confident of the triggers present, you don’t know how reliably that agent will go deep enough or try hard enough to surface everything of value.

If it thinks it is done, it finishes.

(Cautious) conclusions

The current evidence supports a cautious conclusion:

• Most agents are good at summarising visible content if they can return it (unsurprising!)

• Hidden/off-screen content can be found, but is not reliably surfaced.

• Agent behaviour varies across tools, modes, tiers, and repeated runs.

• The same prompt can produce different levels of depth in the generated response even within the same Chatbot/agent.

• Browser/tool access does not guarantee exploratory behaviour.

• Just because an AI chatbot has an agent function or tooling to launch a browser and render the page, doesn’t mean it will.

• Context is likely a huge factor in tool use and curiosity.

• Agents appear more economical than curious.

The strongest practical takeaway:

If we want agents to discover something, it probably needs to be contextually relevant enough to justify extra inspection.

Or put another way:

Agents do not naturally explore. They complete the task they think they have been given.

Make it clear and obvious, or risk having it missed.

Further testing needed

This is not enough to make a grand claim about all agents or all AI chatbots. But I am increasingly confident that common AI chatbot referral traffic is unlikely to involve a full browser render of your content by default. They can, but that does not mean they usually do.

The next tests need to compare:

• the same prompt across multiple agents

• the same agent across repeated runs

• visible vs hidden content

• normal page vs suspicious page

• test site vs CMS platform

• different page topics

• different tiers/modes

• whether explicit “agent/crawler” framing changes behaviour

The better question is not, “Are agents curious?” It is, “What makes an agent decide that deeper inspection is worth doing?”

Perhaps more importantly, “How much of this can we influence?”

Post-script: the test I didn’t tell you about

I did a series of tests before all these, v1, which is worth discussing now.

Test v1 also showed that several AI chatbots were able to identify the page as an attempted prompt-injection or agent-manipulation test.

I was trying to elicit link clicks from agents: links that appeared like they contained extra context, with unique identifiers that helped me “see” an agent had delved deeper to find this information. But it looked like a trap, and in enough instances, was treated like a trap.

In other words, the test may have been measuring the agents’ ability to recognise “this looks like a honeypot/prompt injection” as much as their willingness to explore hidden or unusual content.

That makes v1 useful as a baseline, but weaker as a natural test of agent curiosity.

“From Google Search’s perspective, optimizing for generative AI search is optimizing for the search experience, and thus still SEO.”

Responses to this have ranged from “See, Google confirms it, we were right after all” to “Google lies, of course they’d say that, don’t believe them.” Both are a bit of a waste of time, but there are some points worth considering nonetheless.

1) Google does not want to give rise to a new shadow industry. They’ll get blamed for people already doing bad things in it.

Google’s relationship with the SEO industry has never been fixed or particularly comfortable. There is almost no incentive to create a new industry for them to try and wrangle, whether you believe it should exist or not.

“Thus still SEO” removes some legitimacy from the more active fraudsters, but it won’t stop things in their tracks. That said, Google’s blessing is not often as powerful in these conversations as some might expect.

There is also a concession here to the advancement and development Google itself has had to go through. Google re-iterates the importance of RAG and notes “query fan out”, two of the more recent additions to the SEO/AEO/GEO toolkit.

The “nothing has changed with AI search” dialogue isn’t all that credible to many, even if you think it is true, so saying “we won’t do anything different” is pragmatic. So my personal advice to you is drop this line - no one will judge you badly.

2) Nuance in technical detail is missing

Whilst Google is clearly advising against some of the more spammy-sounding practices coming from “GEO bros”, a lot of the guidance remains broad and essentially boils down to “build sites well and write great content”. As messaging goes, it is consistent if nothing else.

Does that mean there are no ways to optimise or approach things differently? No. Has any other Google guide stopped people testing and experimenting around the advice? Also no.

If they had said that we now need to treat machines differently through AI-specific optimisations, it would undermine much of the advice they’ve put out previously and potentially imply failings or gaps in the current process. 

If you’re walking away from this ignorant of creating better content structures, marking up content correctly, including real facts and using confident language, you’ll probably be worse off.

Also, this guide covers Google’s version of “AI Search” not the other contenders. 

One of the biggest considerations of a technical SEO today is that we need to optimise for many more different processes/pipelines which aren’t all operating at the same level.

3) Non-commodity content is what Google wants

There are two genuinely useful clarifications here. Firstly, optimising for agents, but secondly, and probably more importantly, a clearer definition of “non-commodity content”:

• Provides a unique point of view

• Helpful, reliable and people-first

• Organised in a way that helps readers

• Includes high-quality images and video

• Focuses on what readers want without overdoing it

• Uses AI-assisted processes in line with spam guidelines

Put even more simply: not AI slop.

The problem is that commodity content has increasingly started appearing in marketing strategies and has even been touted in earnings calls. What is “valuable” is highly subjective and the SEO industry has been successfully lying to itself that the content it has been producing for the last 5-10 years was actually valuable. 

This is before LLMs were good at creating facsimiles of “good content”. Scale changes this massively. 

AI slop is an existential problem for web content if the worst-case scenarios play out, so this part of the guidance does feel worth paying attention to.

What do we do with this?

Do we think AEO/GEO disappears from the conversation? No, at least not right now. That kind of U-turn would mean lost budgets, jobs and credibility. Rightly or wrongly, we’re not there yet.

None of this changes the fact that arguing over the differences between three-letter acronyms is still not a ranking factor.

This new post, whilst linking to guides with more specific recommendations, feels more aimed at shaping behaviour and perceptions. If that was the main goal or KPI of the guide, it feels 12+ months too late to get ahead of things.

Hindsight isn’t especially useful, and there could be plenty of reasons why this wasn’t published sooner. Maybe in 2-3 years this whole thing will have blown over, do you think?

If we think back to a lot of the old tropes within SEO, many of them haven’t been made redundant by the onset of AI, they have been past their sell by date for years.

Part of this is the unclear route from the curiosity it takes to try something new to making commercial sense. Commodified SEO is more justifiable if it is packaged and looks like an accepted offering…

… over something new. Even if it doesn’t work anymore.

Worse perhaps is the faux products that have filled the gap, looking to capitalise on the worry and confusion.

During a time like this, we need to allow ourselves to focus on the Return on Interest rather than just the immediate worry of Return on Investment*.

We’re in the era where we need to research more than ever. Curiosity and testing is a competitive advantage. Over-focusing on short-term returns too often ensures long-term mediocrity. 

This isn’t a complete playbook, it’s public snippets of what I have learned (where I can share the details) and my read on the work of others where I think people need to stop and take notice.

Stay tuned as I share my thoughts here!

*Yes I know “investment” doesn’t have to be financial, but I’ve never met someone who was asking about their ROI (Projected or realised) that wasn’t talking about how much money they’d make back