Bot / Scanner info

This page describes how HonestClaw scanner traffic identifies itself and how to control it in common WAF setups.

User-Agent identity

The User-Agent starts with HonestClaw-Scanner/<version>. Current releases link tohttps://honestclaw.ai/scanner. Upcoming releases link tohttps://honestclaw.ai/bot.

Allowlists should match the prefix HonestClaw-Scanner/ either way. The version segment changes for debugging.

Agent-UA probes are still declared as HonestClaw and add one probe token in the comment. Current releases use the /scanner contact URL, and upcoming releases use/bot:

Current releases:
HonestClaw-Scanner/<version> (+https://honestclaw.ai/scanner; agent-ua-probe=ChatGPT-User)
HonestClaw-Scanner/<version> (+https://honestclaw.ai/scanner; agent-ua-probe=Claude-User)
HonestClaw-Scanner/<version> (+https://honestclaw.ai/scanner; agent-ua-probe=Perplexity-User)
HonestClaw-Scanner/<version> (+https://honestclaw.ai/scanner; agent-ua-probe=ClaudeBot)

Upcoming releases:
HonestClaw-Scanner/<version> (+https://honestclaw.ai/bot; agent-ua-probe=ChatGPT-User)
HonestClaw-Scanner/<version> (+https://honestclaw.ai/bot; agent-ua-probe=Claude-User)
HonestClaw-Scanner/<version> (+https://honestclaw.ai/bot; agent-ua-probe=Perplexity-User)
HonestClaw-Scanner/<version> (+https://honestclaw.ai/bot; agent-ua-probe=ClaudeBot)

Those probe requests are measurement probes, not a different crawler identity.

Robots controls

The robots token is HonestClaw-Scanner. To block this scanner inrobots.txt, publish:

User-agent: HonestClaw-Scanner
Disallow: /

Disallow: / stops all page and file fetches. We still read/robots.txt and send one HEAD request per host to check it is reachable.

What we fetch

  • Public pages.
  • robots.txt.
  • Well-known files.
  • If your site declares an MCP endpoint, a short MCP session (initialize, list tools and resources, a protocol-capability request, a session close with DELETE, and OAuth metadata reads). MCP sessions skip robots checks for that endpoint.
  • A limited per-scan request budget.

No login. No form submission. Browser rendering is currently off.

Pacing

  • During a scan: at most one request in flight per host, and at least 250 ms between request starts to the same host.
  • At scan start, reachability checks send one HEAD to the root and up to nine subdomains in parallel.
  • At most one retry: after a timeout or connection error, or a 408, 429, 500, 502, 503 or 504 response, or a 403 carrying Retry-After orx-ratelimit-remaining: 0.
  • For rate-limit responses we wait as long as Retry-After or the rate-limit reset asks, and skip the retry if that does not fit in the scan's time budget. Other retries wait at most 10 seconds.
  • We never retry a plain 403 or a robots disallow (challenge pages served as 200 or a plain 403 included), and never retry with a different identity or source IP.

IP list

Source ranges will be published at /bot/ips.json.

The list is coming soon while egress publication is in progress.

Allowlist guidance

Until source IPs are published, use the User-Agent prefixHonestClaw-Scanner/ for identification and narrow rate-limit exemptions.

Do not rely on the User-Agent header alone for a security bypass. Apply bypass rules only after you also verify source IPs from /bot/ips.json.

Keep your normal bot controls for traffic that does not match your HonestClaw allowlist conditions.

Contact

Contact listed in /.well-known/security.txt:security@honestclaw.ai.