The same website scored 95 on our technical audit and 20 on Cloudflare's
Both scores had an explanation. Three of the faults were ours.
On 10 September 2026 we ran three websites through our AI Readiness audit and through Cloudflare's agent readiness scanner. We gave them 95, 98 and 93; Cloudflare gave 20, 20 and 27. Most of the gap comes from measuring different layers: ours checks whether an AI search engine can read the page, and 8 of Cloudflare's 15 checks are about APIs, authentication and MCP servers. When we then audited the homepages of 23 client websites with the corrected engine, none had a single agent signal, 17 scored 70 or more, and 4 always rejected GPTBot, OpenAI's training crawler, while letting every search crawler through.
- Sample
- 3 websites run through both tools · 23 client homepages audited with the corrected engine
- Period
- 10 September 2026
- Method
- The three websites were run the same day through Cloudflare's scanner (isitagentready.com) and through our AI Readiness audit. The portfolio homepages were then audited with the corrected engine, storing the full audit and the score the previous version would have given on the same data. On the sites that rejected the request presented as GPTBot, the test was repeated five times, alternating with a browser.
- Language
- Spanish-language websites (Spain)
What each score measures
| Website | Our AI Readiness | Cloudflare scanner | Checks passed on Cloudflare |
|---|---|---|---|
| www.vilnux.com | 95 | 20 | 3/15 |
| www.joseredondo.es | 98 | 20 | 3/15 |
| www.forummontau.com | 93 | 27 | 4/15 |
The isitagentready.com score is the number of checks passed out of 15: three passed give a 20 and four give a 27. They fall under discoverability (robots.txt, sitemap, Link headers, DNS records for agents), content (serving Markdown to whoever asks for it), bot access control (AI crawler rules and Content-Signal) and a block of 8 checks on APIs, OAuth and MCP that only makes sense if you offer something an agent can use. For a clinic or a factory, more than half the score depends on things it does not need.
Our score looks at something else: whether AI crawlers can get in, whether the page has structured data, content that can be cited and authorship signals. A website can be very well prepared for an AI search engine to read it and have nothing for an agent to operate. The three websites were exactly that.
What was wrong with ours
- It did not read
noindexornosnippet. Withnoindexa page drops out of Google and Bing, and Google documents thatnosnippetandmax-snippet:0stop its text being used as a direct source for AI Overviews and AI Mode. A page carrying that tag could walk away with a 95. None of the 23 homepages had it, but the audit would not have noticed. - It only looked for the sitemap at
/sitemap.xml. A website with five sitemaps declared in its robots.txt was told it had none, because on its www version that path returned 404. And a sitemap index was counted as if each sitemap were a page: a site with more than a hundred pages came out as "3 URLs". Across the portfolio, 16 of the 23 homepages publish an index. - Checks that "do not score" did weigh on the recommendations. The score ignored them, but the calculation that orders the actions turned their weight of 0 into 1.
And one more that does not affect the score: audits saved in June and July were still showing in the dashboard a citation multiplier we withdrew in July for having no source.
How much the score changed once fixed
Very little, and upwards. On the same audits, 13 of the 23 homepages went up and none went down; the largest rise was 4 points. That makes sense: fixing a false negative raises the score of whoever suffered it, and every site passes the new check.
Median 81 · 17 of 23 at 70 or above · 9 at the top level
That leaves the underlying criticism standing. Correcting the instrument does not make it more demanding: among websites that are reasonably well built, our score still separates them poorly. So we changed the method that same day. In version 2, whatever mechanically stops an AI from using the page (not responding, a noindex or nosnippet, search crawlers blocked in robots.txt, or a robots.txt returning a server error) no longer adds or subtracts points: it blocks, and the audit puts it first even if everything else scores 95. The top level stopped being called "Leader" and is now "Well prepared", and from there the audit says that improving the site moves little. The figures on this page come from the 10 September capture, taken before that change.
Agent signals, counted across the portfolio
The agent layer of our audit reviews 12 signals: serving Markdown, Content-Signal, Link headers, WebMCP, an MCP server card, a skills index, an API catalog, OAuth, protected resource metadata, an ARD catalog, auth.md and the bot signature directory. On 23 of 23 homepages there was not one. That matches what Cloudflare published in April 2026 when it launched its scanner: 3.9% of sites negotiate Markdown, 4% declare AI usage preferences and fewer than 15 published an MCP server card.
That is why this layer does not score. A grade that fails every website does not help tell them apart, and no AI search engine has confirmed it uses these signals to decide whom to cite.
What turned up without looking for it
- 4 of 23 websites rejected GPTBot, the crawler OpenAI uses to collect content for training its models, while serving the page to a browser: three with a 503 and one with a 406, every time, over five alternating attempts. The robots.txt of all four lets GPTBot in. With search crawlers all 4 returned 200: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot and Bingbot. One also rejected ClaudeBot, Anthropic's training crawler. This does not stop ChatGPT citing them. What there is, is a contradiction between what their robots.txt says and what their server does, and whoever decides should know which of the two they want.
- 2 homepages took more than 1.8 seconds to start responding, web.dev's slowness threshold for people. No AI crawler publishes how long it waits, so we do not turn it into a score.
- 2 websites have a sitemap but do not declare it in robots.txt, which is where crawlers that do not probe paths blindly look for it.
Correction of 10 September 2026, hours after publishing: the first version said these four websites rejected "the request that presented itself as GPTBot" without making clear that GPTBot is the training crawler. When the test was repeated with the search crawlers, all of them got through. It read as a block that does not exist.
And ours
vilnux.com scored a 20 on Cloudflare's scanner. We sell an audit that checks whether a website publishes its MCP server card, and our own MCP server had no card. Since 10 September the site publishes that card at /.well-known/mcp.json, an API catalog, Link headers on the homepage and a Content-Signal that allows its content to be used for search, for answers and for training. Run through the scanner again that same morning, it scored a 47 (7/15 checks). It does not serve Markdown on request yet. Our own audit score did not move: none of those signals counts towards it.
What this finding does NOT prove
- Three websites compared with both tools are not a sample: they show what each tool measures, not which one is more accurate.
- Our score measures the homepage, not the whole site. A website can have a sound homepage and poor pages where AI actually cites it.
- Tests using crawler strings do not show what happens to the real crawler, which arrives from its operator's addresses: a firewall that verifies by IP would treat it differently. Only the server logs confirm it.
- Neither score measures whether AI cites the brand. A website scoring 95 can be absent from every answer.
- These are 23 websites from the portfolio of a single measurement engine, from specific sectors and in Spain. They do not describe the internet.
- The adoption figures for agent signals are Cloudflare's, published in April 2026, and we have not verified them ourselves.
Frequently asked questions
Why does the same website get such different scores on two AI readiness audits?
Because they measure different layers. An AI visibility audit checks whether a search engine can read and understand the page: crawler access, sitemap, structured data and content. Cloudflare's scanner mostly checks whether an agent can discover and use the website: API catalogs, OAuth, MCP servers and Markdown. Almost no website has the second, so almost all fail there.
Do I need an MCP server card or Markdown serving to show up in ChatGPT?
No AI search engine has confirmed it uses those signals to decide whom to cite. They make sense if you offer an API or an MCP server you want an agent to use. For appearing in answers, it matters more that crawlers can get into your website and that you are talked about outside it.
How do I know whether my website blocks AI crawlers even though robots.txt allows them?
Search your server logs for requests answered with 403, 406 or 503 and split them into two groups. If it happens to OAI-SearchBot, ChatGPT-User, PerplexityBot or Claude-SearchBot, which search in order to answer, those AIs cannot use your site. If it only happens to GPTBot or ClaudeBot, which collect content for training, it does not affect whether you are cited. In our sample, 4 of 23 websites rejected GPTBot and let every search crawler through.
What can stop a well-built page from being cited?
A noindex tag removes it from Google and Bing results, and Google documents that nosnippet and max-snippet:0 stop its text being used as a direct source for AI Overviews and AI Mode. It is checked in the robots meta tag and in the X-Robots-Tag HTTP header.