Cloudflare's Super Feature: AI Bot Blocking for Everyone

Cloudflare's new AI feature makes it easy for users to protect their content from unwanted AI bots. By using globally aggregated data and advanced machine learning models, Cloudflare can quickly detect and block new scrapers and their behaviors. This ensures that

Share
Cloudflare's Super Feature: AI Bot Blocking for Everyone

Cloudflare has launched a new feature that lets users easily block AI bots with a single click. This feature is available to all customers, even those on the free plan. The goal is to protect content creators from having their material exploited without permission by AI companies using web scraping techniques.

A Growing Problem With AI Bots

The rising popularity of generative AI has led to huge demand for content to train models and run inference. However, not all AI companies are transparent about their operations. Google, for example, has paid $60 million a year to license user-generated content from Reddit, while Scarlett Johansson has accused OpenAI of using her voice without permission. Most recently, Perplexity has been accused of impersonating legitimate visitors to scrape content from websites. This highlights the importance of protecting original content.

How the New Feature Works

To enable the block, go to the "Security > Bots" tab in the Cloudflare dashboard and toggle the switch labeled "AI Scrapers and Crawlers." The feature will update automatically as new fingerprints from malicious AI bots are identified.

Current AI Bot Activity

Data from Cloudflare shows that Bytespider, Amazonbot, ClaudeBot, and GPTBot are the most active AI bots. Bytespider, operated by ByteDance, is used to collect training data for their large language models. Amazonbot is used to index content for Alexa, while ClaudeBot is used to train the Claude chatbot. GPTBot, managed by OpenAI, collects training data for products like ChatGPT.

AI BotShare of Websites Accessed
Bytespider40.40%
GPTBot35.46%
ClaudeBot11.17%
ImagesiftBot8.75%
CCBot2.14%
ChatGPT-User1.84%
omgili0.10%
Diffbot0.08%
Claude-Web0.04%
PerplexityBot0.01%

Source: Cloudflare

Source: Cloudflare

Identifying and Blocking Fake Browsers

Cloudflare has discovered that some bot operators try to hide their activity by using spoofed user-agent strings. Through advanced machine learning models, Cloudflare can identify these deceptive activities and block them effectively. Customers who already have WAF rules in place to challenge visitors with low bot scores automatically block this type of traffic.