One site, one week, and specific enough to be worth writing down. The site is a documentation and catalogue subdomain created on 2026-09-02, with 141 URLs in its sitemap, no inbound links to speak of and no history. The window is the seven days ending 2026-09-09, of which the log holds 165 hours, so the effective window is 6.875 days. Every request in it was classified by user agent and matched against the sitemap.
| Crawler | Sitemap URLs fetched | Requests | Coverage |
|---|---|---|---|
| ClaudeBot | 140 | 144 | 99.3% |
| Amazonbot | 127 | 131 | 90.1% |
| YandexBot | 113 | 119 | 80.1% |
| MJ12bot | 112 | 135 | 79.4% |
| SemrushBot | 98 | 99 | 69.5% |
| GPTBot | 90 | 149 | 63.8% |
| meta-externalagent | 67 | 95 | 47.5% |
| bingbot | 35 | 62 | 24.8% |
| Applebot | 33 | 36 | 23.4% |
| PerplexityBot | 5 | 16 | 3.5% |
| Googlebot | 2 | 8 | 1.4% |
| AhrefsBot | 2 | 3 | 1.4% |
| Claude-User | 1 | 2 | 0.7% |
| DuckDuckBot | 1 | 2 | 0.7% |
| ChatGPT-User | 1 | 1 | 0.7% |
| DotBot | 1 | 1 | 0.7% |
All 141 sitemap URLs were fetched by at least one search or AI crawler in the window. Nothing was missed by everybody. The difference between the top of the table and the bottom is not access, it is appetite.
Every one of the 141 URLs was checked through the Search Console URL Inspection API on 2026-09-09, one call per URL against the verified domain property:
Serving is not the problem. Googlebot receives HTTP 200 and full text, robots.txt allows everything, and the sitemap is submitted and fetched repeatedly. A crawler that reads your sitemap and then takes two pages has looked at what you offer and declined most of it. On a new domain with no authority that is ordinary behaviour, and it takes months rather than days to change.
Training and indexing crawlers versus user-triggered fetches.
Claude-User and ChatGPT-User are separate agents from
ClaudeBot and GPTBot, and they appear when a person's question causes a
fetch. In this window there were two such fetches in total, against 293 requests from ClaudeBot and GPTBot alone. Broad coverage is not the same as anyone reading.
Coverage is not citation. Nothing in a server log tells you whether a page was used in an answer. What the log supports is the weaker and still useful claim that the content is available to those systems, and that Google's is not.
If your pages are reference material, the audience that has actually read them is not the one you
optimised for. That argues for pages that answer a narrow question outright in the first lines, for
keeping an accurate llms.txt, for stating dates and sources so a fact can be checked, and
against another round of pages built for a search engine that took two URLs in a week.
It does not argue for giving up on search. It argues for not spending the next month on it while a different set of crawlers reads everything you publish within a day.
One site, one week, 141 URLs, a domain three weeks old. User agents can be spoofed, so a name in this table is a claim rather than a verified identity, though the volume pattern is consistent with the published crawler documentation for each. Coverage here means fetching a URL that is in the sitemap; requests to other paths were not counted. A larger or older site would see different numbers from Googlebot in particular, because crawl budget follows authority.
Source: edge request logs for mcp.zovo.one ending 2026-09-09, joined to
the 141-URL sitemap; the Google figures from
POST https://searchconsole.googleapis.com/v1/urlInspection/index:inspect, one call per
URL against a verified domain property, run the same day. Both are recorded in
data/indexation.json in this repository.
No, and nothing in a log can tell you that. It tells you the content was retrieved. The separate user-triggered agents, Claude-User and ChatGPT-User, are the closest signal available, and here they accounted for two fetches in a week.
That depends on whether your content is the product or the advertisement for it. For reference documentation about something you sell, being read is the point. For the paid content itself, it is not.
New domain, no inbound links, no history. Crawl budget is allocated on authority, and there is none yet. The observable part is that it fetched robots.txt and the sitemap repeatedly and then took two pages, which is a decision rather than a failure to discover.
Any edge or server log with user agents, grouped by agent and joined to your sitemap URL list. For the index side, the Search Console URL Inspection API returns coverage per URL on a verified property, one call each, and it is free.
Only as claims. Each of the major crawlers publishes IP ranges or a reverse-DNS method for verification, and doing that check is the difference between a measurement and a guess. The counts here are unverified by IP and are presented as such.
All MCP servers and prices · All guides · Buy the bundle $39