> For the complete documentation index, see [llms.txt](https://help.spicegems.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://help.spicegems.com/geoip-country-redirect/country-redirect/faqs/list-of-search-engine-crawlers-excluded-by-the-app.md).

# List of search engine crawlers excluded by the app

By default, the app automatically excludes **Google bots** and several other bots from the redirection. This feature ensures optimal website indexing and allows Google to accurately crawl and index all versions of your store.

Here are the list of bots available in the app's script.

<table><thead><tr><th width="154">Category</th><th width="612">Bot Names</th></tr></thead><tbody><tr><td>Google Bots</td><td>googlebot, www.google.com, googlebot-mobile, googlebot-image, googlebot-news, googlebot-video, google-adwords-instant, appengine-google, google-search-console, google-shopping-quality, google web preview analytics, storebot-google, googleweblight, google favicon, duplexweb-google, google-read-aloud, adsbot-google-mobile-apps, apis-google, google-inspectiontool, google-cloudvertexbot, google-extended, googleproducer, google-speakr, google search console, google-safety, google-cws, google-agent, googlemessages, google-notebooklm, google-pinpoint, google-producer, google-amp-cache-request, feedfetcher, googlebot-discovery</td></tr><tr><td>Google Ads Bots</td><td>adsbot-google, adsbot-google-mobile, mediapartners-google</td></tr><tr><td>Google Feeds &#x26; Preview</td><td>feedfetcher-google, google web preview, googleother-image, googleother-video, googleother</td></tr><tr><td>AI Crawlers</td><td>GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, MistralAI-User, Cohere-AI, meta-externalagent, meta-webindexer, meta-externalfetcher, meta-externalads, Applebot-Extended, Bytespider, Diffbot, ICC-Crawler, YouBot</td></tr><tr><td>AI User Fetchers</td><td>ChatGPT-User, Claude-User, Perplexity-User</td></tr><tr><td>Search Engine Crawlers</td><td>Amazonbot, DuckAssistBot, PetalBot</td></tr><tr><td>SEO / Analytics Crawlers</td><td>AhrefsBot, SemrushBot, MJ12bot, CCBot, SerpstatBot, BLEXBot, DotBot</td></tr><tr><td>Gemini Deep Research</td><td>gemini-deep-research</td></tr><tr><td>Social Media Bots</td><td>facebookexternalhit, Twitterbot, LinkedInBot, Pinterestbot, Slackbot, Discordbot, SkypeUriPreview</td></tr><tr><td>Apple Bot</td><td>applebot</td></tr><tr><td>Bing Bot</td><td>bingbot</td></tr><tr><td>Yandex Bot</td><td>yandexbot</td></tr><tr><td>DuckDuckGo Bot</td><td>duckduckbot</td></tr><tr><td>Baidu Bot</td><td>baiduspider</td></tr><tr><td>Yahoo Bot</td><td>slurp</td></tr><tr><td>Google Inspection Tools</td><td>google-inspectiontool, google-site-verification</td></tr><tr><td>Google Automation Tools</td><td>chrome-lighthouse</td></tr><tr><td>MSN Bot</td><td>msnbot</td></tr><tr><td>Open-source Crawlers</td><td>Nutch</td></tr></tbody></table>

If you are using a service for your website that uses a specific bot, please let us know by dropping us an email at <help@spicegems.com>.&#x20;

Here is a quick video of Bots testing:

{% embed url="<https://vimeo.com/1149446675?fe=ci&fl=sv&share=copy>" %}