practice

Make your site readable by AI agents

· HonestClaw

Agents run past a browser stuck on a loading bar toward another browser that says Agents welcome.
An agent that only gets the first HTML never sees an answer that only appears after JavaScript.

You can spend a week, or thousands of dollars, on a homepage animation that looks great for people but an AI agent might never see it. And if an agent cannot read your public pages, it cannot research you, compare you, or use what you publish.

What teams treat as best practice for beautiful human websites often hides the answers from agents. Polished client-only JavaScript for people in browsers has been the default for years. But agents that only read the first HTML miss answers that appear only after that JavaScript runs (an empty first response or a page shell that needs JS).

Put every answer you want agents to use in the first HTML response. Server-render it, prerender it, or use a hybrid. If your key text only shows up after the browser runs JavaScript, an agent never sees it.

TL;DR

  • Find: if your server returns 403 to an agent, the run ends there. Block only on purpose.
  • Understand: if pricing, plans, and limits sit behind a click or inside an image, agents guess or move on. Put them in plain text, and link the docs from the homepage, because that is where most agent runs go next.
  • Use: only matters if agents need to operate your product. Then publish how to get credentials and where the API, MCP, or CLI contract lives, and link it from your docs.
  • Test: Run the copy-pasteable check below. It saves the response while printing its status, extracts readable text with Python’s standard library, then counts the fact. A 403 status means you are blocked. A success status with a zero count in the extracted text means the fact is not in the first response. Swap in any agent’s name; it needs no account with anyone.
  • The evidence: two independent measurements, 2024 and 2026, found no major AI crawler runs JavaScript. Two large studies agree llms.txt does not get you cited; it only helps agents already on your site. As of April 2026, 3.9% of the top 200,000 domains serve Markdown on request.
  • This is the how-to for agent readiness on published pages. HonestClaw does not sell scores, reviews, or placement.

Find: the agent has to get the page and see what you offer

If an agent gets blocked, nothing else on this page matters. Three things go wrong here, and two of them usually happen without anyone deciding to.

  1. Bot rules that catch agents. Firewalls tuned for scrapers often return 403 to agent user-agents or to the shared cloud IP ranges agents fetch from. In the controlled probes Vercel published in 2026 (190 fetch probes across 19 site configurations), a 403 stopped both the plain fetch client and the JavaScript-running client, five out of five times. It is also easy to do without meaning to: since September 15, 2026, new domains on Cloudflare block agent bots by default on pages that show ads. If you block a path, block it on purpose and know which one.
  2. A robots.txt written for search engines. Cloudflare’s scan of the 200,000 most visited domains found that 78% have a robots.txt, and most of those are written for search crawlers, not agents. A robots.txt rule that disallows AI crawlers stops a compliant agent before it sees the answer. A public site does not need a robots.txt to permit crawling. Use it only for deliberate exclusions, and read it once a quarter.
  3. No page that states the offer. Agents base their answers on pages they actually fetched. In the same run study (1,033 agent runs across 25 sites), 82.5% of answers traced back to a fetched page, and docs pages were the largest single source. If what you sell and what it costs live only in a PDF, a sales deck, or a support thread, an agent cannot find them. Put them on a URL you control, in plain text the first response can read.

A sitemap helps crawlers list what exists. Agents fetched sitemap.xml directly in 4% of runs. Keep one, but do not expect it to do more than that.

Understand: the facts have to be in the first response

This section is the rule at the top of the page, applied to your own site.

The first HTML test. Fetch the page without running JavaScript. If the fact you want an agent to use is in that response, you pass. If the response is an empty page shell and a script tag, you fail, no matter how good the page looks in a browser. Vercel and MERJ measured GPTBot and ClaudeBot across hundreds of millions of requests in late 2024 and found no JavaScript execution. GPTBot downloaded JavaScript files about 11.5% of the time and did not run them. Nothing published since then says otherwise. What passes is crawlable HTML, and nothing else.

Write the fact as a sentence. A page an agent can fetch but cannot make sense of still loses. An agent will not pull a price out of a screenshot or guess a limit from a headline. Put the number, the limit, or the requirement in plain words on the page.

The homepage is a map. Agents start there. In the run study the homepage was reached in 69% of runs and was the first fetch in 92% of those. The most common next hop was a docs page, in 59% of the runs that reached the homepage. Docs pages were reached in 83% of all runs, and agents that got there read about three and a half of them. So the homepage has one job: say what the product is and link to the docs that answer how. A vague call to action, a menu that only exists in JavaScript, or a dead docs link ends the run before it starts.

Link your discovery files. Publishing them is not enough. llms.txt, openapi.json, and .well-known paths only help if agents find them, and agents find them by following links, not by guessing paths. In the run study, 97% of openapi.json fetches, 93% of .well-known fetches, and 86% of llms.txt fetches arrived through a link. Put those links in the footer, the docs sidebar, or the HTML head with rel="alternate". If nothing links to the file, nobody finds it.

Offer Markdown, once the HTML works. Agents ask for it. Across the study harnesses, 65% of all web fetches requested Markdown, and when a fetch named any format, it named Markdown 96% of the time. Cloudflare reports that a blog post costing about 16,000 tokens as HTML costs about 3,000 as Markdown. Serve it through content negotiation on Accept: text/markdown, or through a linked .md alternate. In the controlled probes, adding Markdown mirrors, llms.txt, JSON-LD, or a sitemap did not change whether a client could retrieve an answer that was already in the server-rendered HTML.

Send honest status codes. Agents read status codes literally. A missing page returns 404, not a 200 with “not found” on it. Throttling returns 429 with Retry-After, not a silent challenge page. A Markdown response carries Content-Type: text/markdown and Vary: Accept so caches keep the two versions apart. To a machine, a soft 404 or a hidden rate limit looks like a real page.

JSON-LD is worth adding where it is accurate. No published study has measured it improving an agent’s answer. Keep the same facts in visible text, because some converters drop script blocks.

Use: only when your product needs it

Find and understand cover a marketing site or a docs site completely. If agents also need to do something inside your product, the next how-to material has to be published and readable too. Three things count here.

  1. A documented way to get credentials without a human in a browser: self-serve API keys with named scopes, or OAuth metadata published under .well-known.
  2. A public API contract, usually an OpenAPI document, or an MCP or CLI entry point, linked from your public docs or a published API catalog so an agent finds it after a normal fetch.
  3. Enough documentation that an agent can get from authentication to a first successful call without guessing.

Adoption is early. On the seven study sites that shipped an openapi.json, agents fetched it in about one run in five. This check stays observational: whether the materials are published and readable. Neither this page nor a scan claims a live agent finished a product task with them.

What the evidence says, and where it disagrees

The checklist above rests on four public datasets plus the scanners themselves. They do not all agree, and the places where they disagree are the useful part.

The JavaScript finding is settled. Vercel and MERJ measured crawlers in December 2024. Vercel ran controlled fetch probes in mid 2026. A year and a half apart, with different methods, they found the same thing: text that only exists after client JavaScript is invisible to the fetch-only clients most agents use. The one documented exception is Google’s crawler, which runs JavaScript because Googlebot does. Do not plan around the exception.

Most of the web is not ready. Of the 200,000 most visited domains Cloudflare Radar scanned in April 2026, 3.9% serve Markdown on request, 4% declare AI usage preferences in robots.txt, and fewer than 15 publish an MCP server card or API catalog. Cloudflare republishes the chart weekly, and this page is updated to match.

A file on the server is not a capability. Most readiness scanners, Cloudflare’s included, check whether a file exists and is well formed. That is easy to satisfy. One developer took a blog from 3 out of 100 to a perfect score mostly with placeholder files. Another wrote up stubbing OAuth metadata on a site with no OAuth server, and noted it passed. A third showed the same site scoring 33 or 67 depending on one preset toggle. A file that claims something the site cannot do is worse for an agent than no file at all. That is why every check on this page looks for the answer in the response, not for a file at a path.

llms.txt is where the sources split. SE Ranking looked at roughly 300,000 domains and found no relationship between having an llms.txt and how often a domain is cited by major LLMs. Trakkr scanned 37,894 domains that AI systems already cite and put the citation advantage at statistically zero; among the 50 most cited domains, only 6% have one. The run study found the opposite direction: among runs that reached an llms.txt, 36% grounded the final answer in it. Being cited and being usable by an agent are different things, and getting cited vs agent-ready covers that gap.

All three are right, because they measure different things. SE Ranking and Trakkr ask whether the file gets you cited. The run study asks whether an agent already on your site uses it. The file does not bring agents to your site. It helps the ones already there. Treat it as a reading list, nothing more.

Markdown for agents has critics. Google’s John Mueller has said publicly that serving Markdown-only pages to AI systems is a bad idea, and Microsoft’s Fabrice Canel has warned it could double crawl load. The worry is fair if your Markdown says something your HTML does not. Serve the same facts in both and it does not apply to you.

Every number on this page comes from a sample, not the whole web. Six sites account for two thirds of the run study. Cloudflare scans the top of the web, not the long tail. Treat the numbers as direction and test your own site.

Check your own site in 30 seconds

Run this against the page you most want agents to read, and pick a fact that should be on it:

curl -sS -A 'GPTBot' -o body.html -w 'HTTP %{http_code}\n' https://yoursite.com/pricing
python3 -c 'exec("""from html.parser import HTMLParser
import sys
class Text(HTMLParser):
    def __init__(self): super().__init__(); self.hidden = 0; self.text = []
    def handle_starttag(self, tag, attrs): self.hidden += tag in {"script", "style"}
    def handle_endtag(self, tag): self.hidden -= tag in {"script", "style"}
    def handle_data(self, data):
        if not self.hidden: self.text.append(data)
p = Text(); p.feed(sys.stdin.read()); print(" ".join(" ".join(p.text).split()))""")' < body.html > page.txt
grep -Foc 'the fact you expect' page.txt

The -A part names the request as a known agent so your firewall treats it the way it treats real agents. GPTBot is OpenAI’s crawler; ClaudeBot, PerplexityBot, or any other published agent name works the same, and none of them need an account. The Python step uses a common standard-library parser to discard tags, comments, scripts, and styles before the count. If the status line is HTTP 403, you are blocked. If the status is HTTP 200 or another success and the count in page.txt is zero, the fact is not in the first response. That is the whole test.

Two caveats. Your laptop’s IP is not an agent’s IP, so a pass here does not prove real agents get through; check your server logs for requests from the published OpenAI and Anthropic crawler ranges. And a 200 with the text present proves the fact is reachable, not that an agent understood it.

Limitations

  • Fetch checks show whether content was reachable in the first response. They do not measure whether an agent understood it or completed a task with it.
  • The numbers here come from four published studies and one weekly scan, each with its own sample. None describes the web as a whole.
  • A free scan on HonestClaw is a separate observational measurement of one published URL. It stays logged out, fetches without JavaScript, honors your robots.txt, and adds an agent comprehension check: whether an agent reading your first response can answer plain questions about what you are and what you offer. No scanner that only checks for files asks that question.

Next steps

  1. Run the 30-second check on your homepage, your pricing page, and your docs landing page.
  2. Fix any 403 first, then any first-HTML miss, then HTTP honesty, then discovery links. Each one hides the next, so the order matters.
  3. Add the use items only if your product needs agents to call it.

Sources

See what agents see

Check how clearly an AI agent can find, read, and use your website.

← All posts