Why would an AI agent read your site - and what happens when it can't

Wait - why would an agent read my site at all?

Because your customers no longer always arrive through Google. Some of them ask ChatGPT "where should I buy X" or "how much does Y cost", and the model goes out to the web to answer. If it reaches your site, the answer is built from your information: your prices, your terms, your products.

And if it can't? It does not tell the customer "I couldn't find it." It answers anyway, from whatever it found elsewhere: a competitor's page, a forum thread, an outdated listing. In the research published alongside the new spec, answers built without the business's own site were 3.7 times more likely to contain none of the facts the user actually asked for.

That is the practical difference. Not "being on trend with AI", but who supplies the information about your business when someone asks about it.

What launched

AgentReady is an open specification - not a binding standard and not a certification badge. It tries to answer a question that had become genuinely confusing: there is now llms.txt, MCP, WebMCP, A2A, Web Bot Auth and a handful of other protocols, so which of them does a business actually need, and in what order?

Its answer is split into layers, following the order in which an agent meets a site. Two of them apply to every site:

• Find - so it can get there. Allow agents in robots.txt and in your bot protection, and publish the information on a first-party page of your own.
• Read - so it can understand. The important answer has to be in the initial HTML, with real HTTP status codes and links to resources like llms.txt and structured data.

There is a third layer, Act, about letting an agent complete a task rather than just read. It applies to software products and services an agent is meant to transact in - not to a business site or a store - so we won't go into it here.

Requirements are graded MUST, SHOULD and MAY, and are based on more than 250,000 real agent runs. Ora's scanner checked around 124,000 sites and found that only 6% score a B or above; the average is 42.

What actually broke the agents

This is the useful part. In the controlled experiment a single documentation site was presented in 19 different configurations, and only two changes broke the agents: blocking access (a 403 response), and content that only loads after JavaScript runs. In the other 16 configurations - which included changes to llms.txt, to structured data and to Markdown versions - the agents retrieved the answer regardless.

That isn't a dismissal of the rest of the work, it's a priority order: first make sure the agent can get in and receive text, then worry about the purpose-built files.

Two further findings are worth attention. The sitemap is barely used: agents reached documentation pages in 83% of runs and the homepage in 69%, but sitemap.xml in only 4%. They browse like people do, through the front page and the links. Which also means llms.txt on its own does nothing - a file built for machines only gets read if something already being read points to it.

Pricing: the widest gap in the research

In the broader study that accompanied the launch - 37,927 agent journeys across 1,056 businesses - one finding stood out. When the answer was built from the business's own pages, 60% of pricing answers were accurate; when it was built from elsewhere, 37%. That is the widest gap of any question type tested.

The implication is plain. If your prices sit in an image, behind a dynamic calculator, or behind a "contact us for a quote" form, someone is still answering that question on your behalf - usually not accurately.

What to do now

1. Check that you aren't blocked. Start with robots.txt, then confirm with your developer or host that bot protection isn't blocking AI crawlers. A note of caution: a quick command-line check can mislead you - some protections filter on connection characteristics rather than just the crawler's name, and a simple testing tool looks suspicious to them in its own right.
2. Make sure the content is in the source. Save a product page as HTML and check whether the price and description are in it. If not, some agents see an empty page.
3. Move prices and terms into text. On the evidence, this is the highest-return change available.
4. Link llms.txt from the homepage and your main pages, rather than leaving it stranded.

Several tools implement these recommendations if you want to self-check: Ora's scanner, is-agentic.com built with Vercel, and Mintlify's Agent Score. Each uses its own scoring method, so don't treat any single score as absolute.

One closing note: most of the data here comes from Ora, which also sells a scanning tool and therefore has an interest in the topic feeling urgent. The controlled experiment ran on a single documentation site. The direction looks right to us and the priority order is sensible, but don't expect technical readiness on its own to double your recommendations.

Sources

• AgentReady - The open standard for agent readiness
• Ora - agent readiness scanner and research
• is-agentic.com - agent readiness check built with Vercel

← Back to the blog