AI Crawlers and sitemap.xml: 30-Day Log Data from Three Sites

TL;DR

• Across 30 days of crawler logs from three websites, AI crawlers made 78% to 89% of all sitemap.xml requests, while robots.txt requests in the same logs split roughly evenly between AI crawlers and search engines
• On a new site with only 15 pages, half of all crawl requests went to robots.txt and sitemap.xml, two files with no content in them; on the two larger sites that share was 17% and 19%
• Each blog post was crawled 8 to 14 times in 30 days, while the homepage was crawled several to several dozen times as often
• Google's documentation says small, well-linked sites may not need a sitemap, but AI crawlers show no sign of following the same rule
• Publishing one more post adds one more line to a list that crawlers reread constantly; it does not make existing posts get read more deeply

Contents

Test setup: 30 days of crawler logs from three websites
Share of crawl requests spent on robots.txt and sitemap.xml
AI crawlers vs. search engines on sitemap.xml
Crawl frequency per blog post
The effect of publishing more posts on AI crawling
Limitations of this observation

Introduction

After a site goes live, the hardest thing to confirm is whether AI systems are actually reading it. Search Console only covers Google's side, and most site owners never see server-level request logs, so the question of whether to do anything specifically for AI usually gets answered on instinct.

What follows is 30 days of crawler logs pulled on the same day from three websites, all run from Taiwan and publishing in Traditional Chinese. Most published research on AI crawlers and GEO looks at English-language sites, so this is a small view from outside that sample. I originally went in to find out which articles were crawled most often, but the clearest pattern in the data was not in the articles at all. It was in sitemap.xml.

Test setup: 30 days of crawler logs from three websites

All three sites use the same logging mechanism, which records the source and path of every crawler request. Crawler identity is not taken from the User-Agent string alone; there is an additional source verification step, and requests with spoofed user agents are excluded. Human visitors are not logged.

Crawlers are counted in three groups: real-time retrieval bots that fetch a page while answering a question (for example OAI-SearchBot), training crawlers that collect content for model training (for example GPTBot and ClaudeBot), and traditional search engine crawlers (Googlebot and Bingbot). "AI crawlers" in this article means the first two groups combined.

Share of crawl requests spent on robots.txt and sitemap.xml

The first thing to look at is how much of the crawl volume these two files take up. Neither contains any content: one declares the rules, the other lists the site's URLs.

Half of the new site's crawl requests never touched any content, because they were spent repeatedly checking which pages exist and which are allowed.

The absolute number of sitemap.xml requests was fairly close across the three sites (138, 442 and 436), while robots.txt requests grew with site size. How often the URL list gets reread does not appear to track how many pages a site has.

AI crawlers vs. search engines on sitemap.xml

Splitting the requests for these two files into AI crawlers and search engines reveals the largest gap anywhere in this data.

robots.txt works as the control here, since it is a file on the same site, over the same period, that only crawlers ever read. On all three sites robots.txt was close to one to one, while sitemap.xml ran anywhere from 3.6 to 1 up to 8.2 to 1.

That rules out the explanation that AI crawlers simply make more requests overall. On seobar.dev, the site-wide ratio of AI crawler to search engine requests was 503 to 341, about 1.5 to 1, and the 8.2 to 1 ratio on sitemap.xml sits far above that baseline.

Google's documentation on sitemaps says a site of roughly 500 pages or fewer with complete internal linking "might not need a sitemap," because Googlebot can find everything by following links. All three sites fall under that threshold, and Googlebot behaved exactly as the documentation describes, requesting the sitemap noticeably less often than robots.txt.

AI crawlers did not show the same judgment. One plausible explanation is that they lack the link graph Google has built up over twenty years, so the list a site provides is what they rely on to discover new URLs. That is speculation, though, because this data can only show that the behavior differs, not why it differs. I have written separate guides on how sitemaps are generated and submitted, and which fields Google actually reads (in Chinese), and on robots.txt syntax and common mistakes (in Chinese).

Crawl frequency per blog post

The crawl requests left over for content pages were distributed very unevenly from page to page.

Each of the 11 posts on seobar.dev was crawled 8 to 14 times in 30 days. Over the same period the homepage was crawled 86 times, the About page 73 times and the blog index 62 times. On Site B the homepage alone received 1,342 requests, while the 18 posts that made the top 50 paths added up to 748. On Site C the homepage received 803, and the 29 posts in the top 50 totaled 1,601.

Homepages and listing pages were crawled several to several dozen times as often as a single post. The homepage is where each new round of discovery starts and the listing page is where new content shows up, so both change more often than any individual article.

The effect of publishing more posts on AI crawling

I measured a different dataset earlier and found that the top 10% of posts took nearly 80% of clicks, so the number of posts does not scale with traffic. That dataset looked at the search results end, and this one looks at the crawling end.

What crawlers come back to reread every day is the list, not the articles. The direct effect of publishing one more post is one more line on that list and one more entry point that might get picked. It does not get existing posts read more deeply, and it does not raise the site's total crawl volume proportionally. sitemap.xml requests on the three sites fell between 138 and 442, even though their page counts differ by more than seven times.

Whether long-tail posts can pass authority and readers to pillar pages through internal links is something crawler logs cannot answer. They show the number of entry points but not how individual pages affect each other, and that would take a different dataset.

What you can check first in practice is the list itself: whether sitemap.xml is missing pages, whether lastmod updates when content changes, and whether robots.txt blocks paths it should not. These checks cost far less than writing another article, and they affect every page on the site.

Limitations of this observation

The popular-page statistics only include the top 50 paths, so the larger the site, the more gets cut off. The average crawls per post for Sites B and C are therefore overestimates, since only posts that made the list are counted, and comparisons across sites should stick to ratios and rankings rather than absolute numbers.

All three sites are maintained by the same person on the same technical stack with the same sitemap generation logic, which pushes their behavior toward each other. Three samples are not enough to generalize to all websites, especially sites that generate sitemaps differently or have very different internal linking.

Being crawled means an AI system fetched your content, but it does not mean it will cite you in an answer. Several more decisions sit between those two events, and this data covers only the first one.

Crawler behavior changes as each company adjusts its strategy, so the ratios above are the result of one observation, not stable constants.