> ## Content Index
> Fetch the complete content index at: https://nordisk-ai.ghost.io/llms.txt
> Use this file to discover other available public pages before exploring further.

# Effectively Protecting Your Website From AI Bots
- URL: https://nordisk-ai.ghost.io/effectively-protecting-your-website-from-ai-bots/
- Published: 2024-06-12T09:41:32.000Z
- Updated: 2026-08-27T17:21:42.000Z
- Description: A longer article with a list of popular AI bots you can use to block AI bots from your website. Also many other great links and recommendations in the article to prevent your data from being stolen.
- Author: Anders Hagen
- Tags: Cybersecurity, AI, Bots, Data Privacy, IT, IT Security, Nordic, premium, Sweden, Sweden, technology, Website, Web Security, #Migrated-1787841634512, #wp, #wp-post, #Import 2026-08-27 14:41, #Import 2026-09-07 13:41

AI robots and so-called scrapers can be a major headache for website owners. These automated tools can steal content, overload servers, and even mimic human behavior, making it hard to tell them apart from real users. Fortunately, there are several effective methods for protecting your website against unwanted AI activity. Here's a long but very useful article on how to best protect yourself against AI bots trying to steal your website content.

**Tip** – Before adding the entire list, check whether there are any bots you actually need to allow to crawl your website for data.

## Table of Contents

- [Make clear in your Terms of Use (TOU) what is not allowed](#sakerstall-i-era-terms-of-use-tou-vad-man-inte-far-gora)
- [An ai.txt file can have a big impact on AI bots](#ai-txt-fil-kan-fa-stort-inflytande-pa-ai-botar)
  - [The benefits of ai.txt](#fordelarna-med-ai-txt)
  - [How to block content with ai.txt](#sa-blockerar-du-innehall-med-ai-txt)
- [robots.txt, the most common way to prevent AI bots](#robots-txt-det-vanligaste-sattet-att-forhindra-ai-botar)
  - [List of AI bots to block in robots.txt](#lista-pa-ai-botar-att-blockera-i-robots-txt)
- [Block user agents](#blockera-user-agents)
  - [What is a User Agent?](#vad-ar-en-user-agent)
  - [Analyze User Agent strings to identify bots](#analysera-user-agent-strangar-for-att-identifiera-botar)
  - [Create rules to block specific User Agents](#skapa-regler-for-att-blockera-specifika-user-agents)
  - [Pros and cons of blocking via User Agents](#fordelar-och-nackdelar-med-att-blockera-via-user-agents)
- [Block IP addresses](#blockera-ip-adresser)
  - [What does IP-based blocking mean?](#vad-innebar-ip-baserad-blockering)
  - [How to set up IP blocking](#sa-har-staller-du-in-ip-blockering)
  - [Why is IP blocking effective?](#varfor-ar-ip-blockering-effektivt)
  - [Challenges and considerations](#utmaningar-och-overvaganden)
  - [Use IP blocking as part of a larger security strategy](#anvand-ip-blockering-som-en-del-av-en-storre-sakerhetsstrategi)
- [Useful links](#vardefulla-lankar)

## Make clear in your Terms of Use (TOU) what is not allowed

It's rare to see a website's Terms of Use state what bots or scrapers are not allowed to do. It's more of a legal gray area, but there are guiding rulings. So it's always good to include this from a legal perspective in your TOU. But the most effective way is to block bots directly at the web server.

**Example of YouTube's TOU, which restricts how scrapers/bots may be used**

```
Permissions and Restrictions
You may access and use the Service as made available to you, as long as you comply with this Agreement
and applicable law. You may view or listen to Content for your personal. non-commercial use. You may also
show YouTube videos through the embeddable YouTube player.

The following restrictions apply to your use of the Service. You are not allowed to:
1. access, reproduce, download, distribute, transmit, broadcast, display, sell, license, alter, modify or
otherwise use any part of the Service or any Content except: (a) as express ly authorized by the Service;
or (b) with prior written permission from YouTube and, if applicable, the respective rights holders;

2. circumvent, disable, fraudulently engage with, or otherwise interfere with any part of the Service (or
attempt to do any of these things), including security-related features or features that (a) prevent or
restrict the copying or other use of Content or (b) lim it the use of the Service or Content;

3. access the Service using any automated means (such as robots, botnets or scrapers) except (a) in the
case of public search engines, in accordance with YouTube's robots.txt flle; or (b) with YouTube's prior
written permission;

4. collect or harvest any information that might identify a person (for example, usernames or faces),
unless permitted by that person or allowed under section (3) above.
5. use the Service to distribute unsolicited promotional or commercial content or other unwanted or mass
solicitations.

6. cause or encourage any inaccurate measurements of genuine user engagement with the Service,
including by paying people or providing them with incentives to increase a video's views, likes, or
dis likes, or to increase a channel's subscribers, or otherwise manipulate metrics in any manner;
```

## An ai.txt file can have a big impact on AI bots

The file ai.txt is a concept similar to robots.txt, but is specifically intended to manage how artificial intelligences (AI) interact with web content. The purpose of this file is to give website owners control over how their data is used by AI technologies, particularly in areas such as automatic content generation and machine learning. ai.txt is a file placed at the root of a web server.  
  
This file type is new, and it's not yet clear how the major companies view it. More and more companies are joining [Spawning](https://spawning.ai/?ref=nordisk-ai.ghost.io), who launched the initiative.

### The benefits of ai.txt

**Control over data sharing:** Website owners can precisely control which data may be used by AI systems, protecting sensitive content and personal data.

**Protection against misuse:** By limiting how AI interacts with the website, owners can prevent their content from being misused by unreliable or unethical AI applications.

**Alignment with ethical guidelines:** ai.txt can help organizations comply with ethical standards and guidelines for AI use, which is especially important at a time when AI ethics is being discussed more and more.

### How to block content with ai.txt

Create a new file named ai.txt and copy the lines below. Upload it to the root of your web server.

```
User-Agent: *
Disallow: /
Disallow: *
```

If you want to allow AI bots to pick up some of your content, we recommend the service Spawning, which is free. The service creates a customized ai.txt file based on your preferences, completely free of charge.

[Visit Spawning here](https://spawning.ai/ai-txt?ref=nordisk-ai.ghost.io)

## robots.txt, the most common way to prevent AI bots

robots.txt is a common file placed at the root of your web server that gives instructions to robots about which parts of your website they may or may not download data from. By specifying which robots are allowed to visit specific pages, you can effectively block unwanted AI robots from reaching sensitive or valuable content.

It's important to know that robots.txt is a request to bots, not an enforced rule. They can completely ignore robots.txt and still collect the data. Major companies like Google, Meta and OpenAI generally do follow robots.txt

### List of AI bots to block in robots.txt

Copy the list, or pick out the bots you want to block, and add them to your robots.txt

```
# Block one of OpenAI's GPT bots
User-agent: ChatGPT-User
Disallow: /

# Block Google's AI for Gemini
User-agent: Google-Extended
Disallow: /

# Block one of Google's general bots
User-agent: GoogleOther
Disallow: /

# Block Common Crawl's CCBot
User-agent: CCBot
Disallow: /

# Block Perplexity AI
User-agent: PerplexityBot
Disallow: /

# Block Anthropic AI
User-agent: anthropic-ai
Disallow: /

# Block one of the Claude bots
User-agent: ClaudeBot
Disallow: /

# Block one of the Claude bots
User-agent: Claude-Web
Disallow: /

User-agent: Bytespider
Disallow: /

# Block Apple's AI bot Siri
User-agent: Applebot
Disallow: /

# Block Amazon's AI bot Alexa
User-agent: Alexa
Disallow: /

# Block Microsoft's AI bot Tay
User-agent: bingbot/2.0
Disallow: /

# Block Facebook
User-agent: FacebookBot
Disallow: /

# Block ImageSiftBot
User-agent: ImagesiftBot 
Disallow: /

# Block Omgili
User-agent: ImagesiftBot 
Disallow: /

# Block Omgili
User-agent: omgili
Disallow: /

# Block Omgili
User-agent: omgilibot
Disallow: /

# Block Friendly Crawler - this one is sneaky and may need more than just robots.txt 
User-agent: FriendlyCrawler
Disallow: /

# Block 
User-agent: AwarioRssBot
Disallow: /

# Block 
User-agent: AwarioSmartBot
Disallow: /

# Block 
User-agent: DataForSeoBot
Disallow: /

# Block 
User-agent: Diffbot
Disallow: /

# Block 
User-agent: img2dataset
Disallow: /

# Block 
User-agent: ImagesiftBot
Disallow: /

# Block 
User-agent: magpie-crawler
Disallow: /

# Block 
User-agent: Meltwater
Disallow: /

# Block 
User-agent: peer39_crawler
Disallow: /

# Block 
User-agent: PiplBot
Disallow: /

# Block 
User-agent: scoop.it
Disallow: /

# Block 
User-agent: Seekr
Disallow: /

# Block 
User-agent: YouBot
Disallow: /

# Block 
User-agent: cohere-ai
Disallow: /
```

## Block user agents

User Agents play an important role in how network traffic is handled and identified on the internet. By understanding and manipulating how User Agents interact with your website, you can effectively prevent unwanted bots from accessing your web content.

### What is a User Agent?

A User Agent is a software identifier that tells web servers about the type of device and browser trying to access a website. This information helps servers deliver the right type of content adapted to the user's device. Examples of User Agents include browsers like Chrome or Firefox, but bots such as Googlebot (used by Google to index web pages) also have specific User Agents.

### Analyze User Agent strings to identify bots

To block bots, you can analyze the User Agent strings of incoming traffic. Many aggressive or malicious bots often use generic or suspiciously crafted User Agents, and by monitoring these you can effectively identify and block them. Tools such as log analyzers and security plugins can help you track these User Agents automatically.

### Create rules to block specific User Agents

You can create server rules that specifically deny access based on User Agent strings. This can be done by modifying your web server's configuration files (such as .htaccess on Apache servers).

Example of how to create a rule in the htaccess file:

```
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Textbot [NC]
RewriteRule .* - [F,L]
```

If you want to block several user agents, here's an example:

```
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^.*(SCspider|Textbot|s2bot).*$ [NC]
RewriteRule .* - [F,L]
```

AWS and Cloudflare also have good built-in features where you can directly add which user agents you want to block:

[Go to AWS documentation](https://repost.aws/knowledge-center/waf-block-http-requests-no-user-agent?ref=nordisk-ai.ghost.io)

[Go to Cloudflare documentation](https://developers.cloudflare.com/waf/tools/user-agent-blocking/?ref=nordisk-ai.ghost.io)

### Pros and cons of blocking via User Agents

Blocking based on User Agents is powerful but has its limitations. The advantages include easy implementation and the ability to quickly stop known bots. The disadvantage is that it's relatively easy for advanced bots to get around this by masking or changing their User Agent string. This method should therefore be combined with other security measures to provide more robust protection against bots.

## Block IP addresses

Blocking IP ranges is an effective method for protecting your website from unwanted or malicious traffic. By restricting access for specific IP addresses or entire networks, you can effectively reduce the risk of attacks and disruptions from known malicious sources.

### What does IP-based blocking mean?

IP-based blocking means denying access to your website for traffic from specific IP addresses or address ranges. This can be especially useful for stopping bot visits from specific regions or networks.

### How to set up IP blocking

To implement IP blocking, you can configure settings directly on your web server or via security plugins offered in various CMS platforms such as WordPress. One example is the htaccess file located at the root of the web server.

Another way to do this with htaccess is to use regular expressions.  
  
**Example:**

```
<IfModule mod_rewrite.c>
	RewriteCond %{REMOTE_ADDR} ^214.53.25.(6[4-9]|7[0-9]|8[0-9]|9[0-9])$ [OR]
	RewriteCond %{REMOTE_ADDR} ^214.53.25.1([0-1][0-9]|2[0-8])$
	RewriteRule .* - [F]
</IfModule>
```

If you want to be a bit sneaky, you can always redirect a bot via htaccess :)  
  
Example

```
<IfModule mod_rewrite.c>
	RewriteCond %{REMOTE_ADDR} ^111\.111\.111\.111$
	RewriteRule .* /duarenbot.php [R=301,L]
</IfModule>
```

If you use Cloudflare or AWS, there's good documentation on how to set up IP blocking in their interface

[Documentation - Cloudflare](https://developers.cloudflare.com/waf/tools/ip-access-rules/?ref=nordisk-ai.ghost.io)

[Documentation - AWS](https://repost.aws/knowledge-center/waf-allow-my-ip-block-other-ip?ref=nordisk-ai.ghost.io)

### Why is IP blocking effective?

IP blocking is effective because it creates an immediate barrier against bots that have proven to be malicious. It's a powerful tool for preventing DDoS attacks, spam and unauthorized data access.

### Challenges and considerations

While IP blocking is a powerful tool, there are a few considerations and potential drawbacks to keep in mind:

**Dynamic IP addresses**: Users with dynamic IP addresses may be unintentionally blocked if they are assigned an address that was previously flagged as malicious.

**VPN and proxy use:** Users using VPN services or proxy servers may appear to come from the IP addresses you've blocked, which can prevent legitimate users from reaching your website.  
  
**Monitor your logs:** IP ranges can change and new malicious sources can appear, requiring regular updates and maintenance of your block list. The easiest way is to check the web server logs for new bots.

### Use IP blocking as part of a larger security strategy

For the best protection, IP blocking should be used as part of a larger, multi-layered security strategy. Combine this technique with other security measures such as user authentication, CAPTCHA tests, and regular security audits to build robust defenses against various types of threats.

Blocking IP ranges is an important security measure that can help protect your website from potential threats and ensure a safe and reliable environment for your users.

## Useful links

**An updated list of AI bots** \- https://github.com/ai-robots-txt/ai.robots.txt  
  
**A large list of bots** \- https://udger.com/resources/ua-list/crawlers

**Bot documentation OpenAI** \- [https://platform.openai.com/docs/plugins/bot](https://platform.openai.com/docs/plugins/bot?ref=nordisk-ai.ghost.io)

**Bot documentation** **Google** \- [https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers?ref=nordisk-ai.ghost.io)

**Bot documentation** **Perplexity** \- https://docs.perplexity.ai/docs/perplexitybot

**Bot documentation** **Meta** \- https://developers.facebook.com/docs/sharing/bot  
  
**Bot documentation** **ImageSift** \- https://imagesift.com/about  
  
CCBOT - [https://commoncrawl.org/ccbot](https://commoncrawl.org/ccbot?ref=nordisk-ai.ghost.io)