Google Warns Against Blocking the GoogleOther Bot
The bot visiting your site called "GoogleOther Bot" - is it safe, or should you block it from crawling your data?
GoogleOther is a generic crawler created by Google for various purposes not covered by the specialized bots for Search, Ads, Video, Images, News, Desktop, and Mobile. It's used by Google's internal teams for research and development related to various products.
The official description of GoogleOther is:
"GoogleOther is the generic crawler that may be used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and development."
Three Types of GoogleOther Crawlers
There are three types of GoogleOther crawlers:
GoogleOther: Generic crawler for public URLs
GoogleOther-Image: Optimized for crawling public image URLs
GoogleOther-Video: Optimized for crawling public video URLs
All three types of GoogleOther can be used for research and development purposes.
Non-Search-Specific Functions That GoogleOther Supports
Google doesn't specify particular non-search-specific functions that GoogleOther supports. The crawler exists for research and development and may support new products or improvements to existing products.
Gary Illyes Warns Against Blocking GoogleOther
Gary Illyes, a prominent figure at Google, warns against blocking GoogleOther. He clarifies that this doesn't affect Google Search since Googlebot is used to index content. However, blocking GoogleOther could affect other products and services within Google, although he didn't specify which products might be involved.
Pros and Cons of Blocking GoogleOther
Pros
Inclusion in Research: Sites that allow GoogleOther may be included in research for future Google products that could be related to search, maps, shopping, or images. This can lead to positive tests that increase revenue for the site.
Server Resources: Saving server resources by blocking GoogleOther isn't necessary since it doesn't crawl often enough to be noticeable.
Cons
Participation in Market Research: Sites that don't want to participate in market research or machine learning models may choose to block GoogleOther.
Unknown Purposes: Allowing GoogleOther to crawl a site for unknown purposes gives Google a "blank check" to use your site data in whatever way they want, except for training public language models.
Conclusion
Should you block GoogleOther? It's an uncertain question. There are potential benefits, but there isn't enough information to make a well-informed decision.