Should You Allow ChatGPT and Perplexity to Crawl Your Website?

The short answer

For most Australian businesses, yes. Allow OAI-SearchBot and PerplexityBot if you want your public pages to have a chance of being cited in ChatGPT and Perplexity. GPTBot needs its own decision because it relates to AI training. Private content belongs behind a login, as robots.txt only tells compliant crawlers what you prefer.

Crawler access can’t guarantee a citation. Our guide to getting cited by AI tools covers the content and reputation work that comes next.

Which AI crawlers do what?

Each crawler has its own job, so a blanket block on “AI bots” can remove you from AI search without solving the problem you had in mind.

OpenAI’s crawler documentation and Perplexity’s crawler guide are the sources for this table, and both can change. Perplexity says neither of its crawlers collects content for AI foundation models.

Should you block GPTBot?

Start with the value of your content and how you want it used.

OpenAI treats its settings independently, so you can allow OAI-SearchBot and disallow GPTBot. The first setting relates to ChatGPT search, while the second tells OpenAI not to use crawled content for model training.

Allowing GPTBot can suit businesses whose public pages exist to explain their services, locations and expertise. Blocking it may suit businesses that sell research, courses or premium articles.

Copyright is a separate question. Australia has declined to introduce a text and data mining exception for AI training, so ask a lawyer how your robots.txt setting fits with your wider rights if your content is licensed or commercially valuable.

A shared option is also on the way. Google plans to support a shared AI training control in robots.txt, with support targeted for 2027. Search Engine Journal reports that Google may treat this opt-out like a Google-Extended block if a site doesn’t name Google-Extended, and that this control also covers grounding in Gemini Apps. For now, keep using each company’s current settings.

What about Gemini and Google AI Overviews?

Google uses a different control. Google-Extended is a robots.txt token that manages whether Google-crawled content can be used for Gemini training and grounding in Gemini Apps. Google says it doesn’t affect inclusion or ranking in Google Search.

The same token covers training and grounding, so blocking it can limit your visibility in the Gemini app. For AI Overviews and AI Mode, Google’s guidance on generative AI features requires a page to be indexed and eligible to show with a snippet.

What should your business allow?

We use this table as a starting point, then adjust it for each site’s content, legal obligations, and risk.

Regulated businesses have more to consider, and our guide to GEO in regulated industries sets out those checks.

How do you set this up in robots.txt?

This example allows ChatGPT search and Perplexity search while blocking OpenAI’s training crawler:

txt

User-agent: OAI-SearchBot  Allow: /  User-agent: PerplexityBot  Allow: /  User-agent: GPTBot  Disallow: /

If your file already names one of these crawlers, edit that group instead of adding a second one.

Review your existing file before you add these lines. Under the Robots Exclusion Protocol, a crawler uses the * group when no group matches its name. A named crawler follows its own group, so any private folder blocked under * needs repeating in each named group.

OpenAI says search changes can take about 24 hours, and Perplexity allows up to 24 hours.

Why isn’t robots.txt enough?

The same protocol states that its rules aren’t a form of access authorisation. Google also warns that robots.txt can’t enforce crawler behaviour, which is why it recommends password protection for private files.

Later use is another limit. Google’s Gary Illyes said in October 2026 that robots.txt governs crawling and doesn’t control what happens to content afterwards.

Other systems can overrule your file too. Cloudflare announced in July 2025 that it would block AI crawlers by default on new domains, so check your own dashboard along with any firewall or security plugin. Perplexity notes that Cloudflare’s Bot Fight Mode can still challenge its crawlers when a custom rule is in place.

How can you check your settings?

1. Open your robots.txt file: Visit yourdomain.com.au/robots.txt and look for named AI crawlers and any Disallow: / rule.
2. Review your firewall and CDN: Look for bot protection rules, then compare any allow rules with the IP ranges OpenAI and Perplexity publish.
3. Read your server logs: Search for the user agents above to confirm requests are reaching your site.
4. Test important prompts: See whether ChatGPT and Perplexity can find and link to your key pages.
5. Watch your referrals: Look for chatgpt.com and perplexity.ai in GA4.

Our guide on what to do when ChatGPT gets your business wrong can help if crawler access isn’t the problem.

Frequently asked questions

Will blocking GPTBot remove me from ChatGPT search?

No. Your ChatGPT search setting sits with OAI-SearchBot.

Can I block Perplexity completely?

You can block PerplexityBot in robots.txt, but Perplexity-User generally ignores those rules because a person requested the page. Stopping those visits would take a server or firewall rule.

Will allowing AI crawlers get my business cited?

There’s no guarantee. AI tools still need clear, useful and trustworthy content to work with.

Does blocking AI crawlers affect my Google rankings?

Rules for OAI-SearchBot, GPTBot and PerplexityBot don’t change your Googlebot settings.

Check what your website is allowing

If you’re unsure which crawlers can reach your site, book a free consultation and our team can look at your settings as part of your wider search strategy.

This article is general information, not legal advice. Crawler documentation and platform settings can change, so check the latest guidance before you make changes.