Google Launches a New Web Crawler for Vertex AI
Google has quietly launched Google-CloudVertexBot, a new bot for Vertex AI customers that crawls website content at the request of site owners. Learn how it works and whether you need to protect your site.
Google has recently introduced a new bot, Google-CloudVertexBot, which is part of its Vertex AI product suite. The new bot is designed to crawl website content for commercial customers and is different from the other bots listed in Google's Search Central documentation, which are tied to Google Search or advertising. According to official Google Cloud documentation, this new bot can crawl websites at the request of site owners.
Google-CloudVertexBot and Vertex AI Agents
Google-CloudVertexBot is used in connection with Vertex AI Agent Builder, which handles various types of data stores, including public website data. The documentation lists two types of website indexing the bot can perform:
Basic website indexing: Uses existing indexed content from Google.
Advanced website indexing: Requires domain verification from the site owner and is tied to specific indexing quotas.
Advanced indexing uses Google-CloudVertexBot, and unlike Google's traditional bots that crawl public pages, this bot works only on behalf of site owners, meaning it doesn't automatically crawl public websites.
Key Features and Use Cases
Google Cloud's documentation describes that data stores with website data use indexed data from public websites, including text and images with metadata. Site owners can provide a list of domains and set up search or recommendation systems based on crawled data from those domains.
The changelog states:
"Introducing the Google-CloudVertexBot crawler: A new bot that crawls websites at the request of site owners when building Vertex AI Agents. The new crawler was introduced to help site owners identify traffic from the new crawler."
How Does Google-CloudVertexBot Work?
User Agent: Google-CloudVertexBot uses "Google-CloudVertexBot" as its User Agent token.
Public Access: This crawler doesn't index public websites and works only on behalf of site owners.
Robots.txt: It doesn't appear necessary to block Google-CloudVertexBot via robots.txt, since it only acts at the request of site owners.
For more information and details on how to handle this new crawler, it's recommended to read the latest Google documentation on Google-CloudVertexBot.