# List of search engine crawlers excluded by the app

By default, the app automatically excludes **Google bots** and several other bots from the redirection. This feature ensures optimal website indexing and allows Google to accurately crawl and index all versions of your store.

Here are the list of bots available in the app's script.

<table><thead><tr><th width="154">Category</th><th width="612">Bot Names</th></tr></thead><tbody><tr><td>Google Bots</td><td>googlebot, www.google.com, googlebot-mobile, googlebot-image, googlebot-news, googlebot-video, google-adwords-instant, appengine-google, google-search-console, google-shopping-quality, google web preview analytics, storebot-google, googleweblight, google favicon, duplexweb-google, google-read-aloud, adsbot-google-mobile-apps, apis-google, google-inspectiontool, google-cloudvertexbot, google-extended, googleproducer, google-speakr, google search console, google-safety, google-cws, google-agent, googlemessages, google-notebooklm, google-pinpoint, google-producer, google-amp-cache-request, feedfetcher, googlebot-discovery</td></tr><tr><td>Google Ads Bots</td><td>adsbot-google, adsbot-google-mobile, mediapartners-google</td></tr><tr><td>Google Feeds &#x26; Preview</td><td>feedfetcher-google, google web preview, googleother-image, googleother-video, googleother</td></tr><tr><td>AI Crawlers</td><td>GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, MistralAI-User, Cohere-AI, meta-externalagent, meta-webindexer, meta-externalfetcher, meta-externalads, Applebot-Extended, Bytespider, Diffbot, ICC-Crawler, YouBot</td></tr><tr><td>AI User Fetchers</td><td>ChatGPT-User, Claude-User, Perplexity-User</td></tr><tr><td>Search Engine Crawlers</td><td>Amazonbot, DuckAssistBot, PetalBot</td></tr><tr><td>SEO / Analytics Crawlers</td><td>AhrefsBot, SemrushBot, MJ12bot, CCBot, SerpstatBot, BLEXBot, DotBot</td></tr><tr><td>Gemini Deep Research</td><td>gemini-deep-research</td></tr><tr><td>Social Media Bots</td><td>facebookexternalhit, Twitterbot, LinkedInBot, Pinterestbot, Slackbot, Discordbot, SkypeUriPreview</td></tr><tr><td>Apple Bot</td><td>applebot</td></tr><tr><td>Bing Bot</td><td>bingbot</td></tr><tr><td>Yandex Bot</td><td>yandexbot</td></tr><tr><td>DuckDuckGo Bot</td><td>duckduckbot</td></tr><tr><td>Baidu Bot</td><td>baiduspider</td></tr><tr><td>Yahoo Bot</td><td>slurp</td></tr><tr><td>Google Inspection Tools</td><td>google-inspectiontool, google-site-verification</td></tr><tr><td>Google Automation Tools</td><td>chrome-lighthouse</td></tr><tr><td>MSN Bot</td><td>msnbot</td></tr><tr><td>Open-source Crawlers</td><td>Nutch</td></tr></tbody></table>

If you are using a service for your website that uses a specific bot, please let us know by dropping us an email at <help@spicegems.com>.&#x20;

Here is a quick video of Bots testing:

{% embed url="<https://vimeo.com/1149446675?fe=ci&fl=sv&share=copy>" %}


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://help.spicegems.com/geoip-country-redirect/country-redirect/faqs/list-of-search-engine-crawlers-excluded-by-the-app.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.