AI Crawlers Account for 28% of Googlebot's Traffic
A new report from Vercel shows that AI crawlers now account for a significant share of the web's data collection. These advanced bots are changing the SEO game and placing new demands on website optimization.
Nearly a Billion Requests From AI Crawlers Every Month
According to Vercel's report, AI crawlers like OpenAI's GPTBot and Anthropic's Claude generate nearly a billion requests per month. Over the past month, GPTBot made 569 million requests, while Claude accounted for 370 million.
Other AI crawlers also contributed to the high number:
PerplexityBot: 24.4 million requests
AppleBot: 314 million requests
Together, these AI crawlers make up about 28% of Googlebot's total traffic volume, which stands at 4.5 billion requests.
What Makes AI Crawlers Unique?
The report identifies several unique characteristics of AI crawlers:
Doesn't render JavaScript: AI crawlers fetch JavaScript files but don't execute them, which can cause content requiring client-side rendering to be missed.
Often inefficient: About 34% of GPTBot and Claude's requests end up on 404 pages.
Content prioritization: ChatGPT prioritizes HTML (57.7%), while Claude focuses more on images (35.17%).
Geographic Distribution of AI Crawlers
Unlike traditional search engines, which often operate globally, AI crawlers have a more concentrated presence in the US:
GPTBot: Des Moines (Iowa) and Phoenix (Arizona)
Claude: Columbus (Ohio)
How Can Companies Optimize for AI Crawlers?
Given the growing impact of AI crawlers, it's important for websites to adapt to maximize visibility and indexing. Here are three key strategies:
1. Server-Side Rendering (SSR)
Since AI crawlers don't execute JavaScript, server-side rendering is crucial.
Recommended actions:
Implement server-side rendering for critical content.
Make sure metadata, headings, and navigation are visible in the initial HTML code.
Use static site generation where possible.
2. Structured Content and File Optimization
Vercel's data shows that different AI crawlers prioritize different types of content:
ChatGPT: Focuses on HTML (57.7%)
Claude: Prioritizes images (35.17%)
Optimization tips:
Structure HTML semantically and clearly.
Optimize image delivery with proper metadata and alt text.
Implement a correct heading structure and avoid unnecessary JavaScript dependencies.
3. Technical Considerations
High 404 error rates from AI crawlers show the importance of keeping the website technically up to date.
Recommended actions:
Keep sitemaps up to date.
Implement correct redirect chains.
Perform regular audits of 404 errors.
AI Crawlers Are Reshaping the SEO Landscape
For SEO specialists, the message is clear: AI crawlers are here to stay and are changing how the web is indexed. Even though many AI bots today rely on cached or older data, their ability to interpret fresh content will grow.
By adapting with server-side rendering, structured data, and technical optimizations, companies can ensure their content reaches both traditional search engines and AI-powered bots.