Free 2-min Growth Check

AI search

AI training opt-out: a robots.txt guide for Irish sites

How Irish businesses can refuse AI training without leaving Google or ChatGPT search: what each robots.txt control does and the 2027 shared opt-out.

Banner for a guide to opting out of AI training in robots.txt, with two glossy switches on a dark desk by a rainy window, one glowing electric blue.

Many Irish business owners assume they face a hard choice: let AI companies use their content, or vanish from ChatGPT and Google’s AI answers. In fact these are two separate switches. A site can say no to AI training and still stay visible in search and AI answers, as long as it blocks the right crawler and leaves the others alone.

Key takeaways

  • AI training and AI search are controlled separately, so a business can refuse training without leaving Google Search.
  • Google says blocking Google-Extended has no effect on inclusion or ranking in Google Search.
  • Blocking Googlebot or OpenAI’s search crawler removes a site from AI Overviews or ChatGPT search answers, which is rarely what a business wants.
  • Google plans to support a shared AI training opt-out in robots.txt, together with other AI providers, from early 2027.
  • Until then, the working controls are the company-specific lines such as Google-Extended and GPTBot.
On this page
Comparison infographic: training crawlers Google-Extended and GPTBot can be blocked, search crawlers Googlebot and OAI-SearchBot should stay open.

Bríd runs a two-person accountancy practice in Kenmare and publishes plain guides on tax deadlines. A client tells her that AI companies copy websites to train their models, and she wonders whether she should block them all. Before she touches anything, she needs to know which crawler does what, because the wrong line could remove her from the AI answers her new clients now read. This is the core of GEO services in Ireland: making sure a business is easy for AI search to find while it keeps control of its content.

Every switch lives in robots.txt (A small text file that tells crawlers which pages they may visit.), where each AI company reads its own name. Some names cover training, some cover search, and the table below sets them side by side.

Control in robots.txtCompanyWhat it controlsDoes blocking it remove you from search?
GooglebotGoogleGoogle Search, including AI Overviews and AI ModeYes
Google-ExtendedGoogleTraining Gemini models and grounding in Gemini AppsNo, not from Google Search
OAI-SearchBotOpenAIChatGPT search resultsYes, from ChatGPT search answers
GPTBotOpenAITraining OpenAI’s modelsNo

The table shows the pattern Bríd was looking for. Training lines can be closed without leaving search; search lines should stay open. A rule that names a crawler only affects that crawler, so one careless line does not switch everything off, but a broad rule for all crawlers can.

Figure infographic of one shared opt-out line in robots.txt reaching many AI providers from 2027.

What Google announced on 30 October 2026#

Want this done for your business? Talk to us about GEO / AI search optimisation

Bríd’s question became easier to answer at the end of October. On the Google Search Central blog, Google announced plans to support a shared AI training control in robots.txt, based on work by the AIPREF group at the Internet Engineering Task Force. A site will be able to use one shared policy token to state its training preference across participating AI providers, instead of writing separate lines for each one.

The same post explains how the two kinds of control will fit together. If a site opts out of training with the shared token and does not name Google-Extended, Google plans to treat that as a block on Google-Extended. Company-specific controls will keep working alongside it, and sites that allow Google-Extended stay eligible to appear in Gemini Apps responses and receive clicks from them. Google aims to support the control together with other AI providers in 2027, and links the plan to the EU Code of Practice for general-purpose AI models under the EU AI Act, which recognises robots.txt as a main way for publishers to state their preferences.

For Bríd, the news means one line could soon do the work of many. Instead of tracking the crawler name of every AI developer, she will be able to state her training choice once and let each participating provider read it. It also means the choice will matter more, because a single opt-out line may reach several companies at the same time.

Note: Google’s crawler documentation says it intends to support this shared training signal, written as user-agent: __usage_ai-training, in early 2027; until then it has no effect.

What each control does today#

Until the shared token arrives, Bríd has to use the controls that already work. Google’s crawler documentation describes Google-Extended as the token that decides whether crawled content may train future Gemini models and be used for grounding in Gemini Apps. It states that Google-Extended does not affect a site’s inclusion in Google Search and is not used as a ranking signal.

AI features inside Google Search work differently. Google’s guide to AI features and your website says a page must be indexed and eligible for a snippet to appear as a link in AI Overviews or AI Mode, and that robots.txt rules for Googlebot are the control. To limit what is shown, Google points to snippet controls such as nosnippet, max-snippet or noindex.

OpenAI splits its crawlers the same way. Its crawler documentation says GPTBot collects content that may be used to train its models, while OAI-SearchBot finds pages for ChatGPT search, and each setting is independent. Sites that block OAI-SearchBot will not be shown in ChatGPT search answers, and changes to robots.txt can take about 24 hours to apply.

Watch out: A copied “block all AI bots” list often includes Googlebot or OAI-SearchBot, which quietly removes a site from the AI answers it wanted to appear in.

Checklist infographic for robots.txt: Googlebot allowed, OAI-SearchBot allowed, training is your choice, review in 2027.

A safe robots.txt plan for an Irish business#

With the facts in place, Bríd can decide in a few steps. The same order works for most small businesses:

  1. Open your current robots.txt by adding /robots.txt to your web address, and note every user-agent line.
  2. Keep Googlebot and OAI-SearchBot allowed, so the site stays in Google Search, AI Overviews and ChatGPT search.
  3. Decide on training: to opt out, add a group for Google-Extended and one for GPTBot with disallow: / under each.
  4. Weigh Gemini Apps separately, because blocking Google-Extended also stops grounding there.
  5. Check the file again a day later, and review it in 2027 when the shared training token goes live.

Bríd chooses to block GPTBot but keep Google-Extended allowed, because she wants her guides to be cited in Gemini Apps as well. Another owner with paid courses online might choose the opposite, and both choices are valid. Other AI companies have their own crawler names too; Baikou’s guide to the Meta AI crawler shows what to check for Meta.

Tip: On WordPress, an SEO plugin usually manages robots.txt, so edit the rules there rather than uploading a file that the plugin may overwrite.

How Baikou checks AI crawler access#

When Baikou’s team reviews a site for AI search, it checks that AI crawlers and Bing can reach and index the pages, since several assistants search through them. It reads the robots.txt rules line by line and flags any rule that blocks a search crawler by mistake. The team also runs a free AI visibility check, which asks AI assistants real buyer questions and shows whether a business is named, cited or missing in the answers.

Where to start this month#

Open your robots.txt today and check that no rule blocks Googlebot or OAI-SearchBot. Then make a calm decision about training, using the table above, and put a reminder in your calendar for early 2027. Baikou helps Irish businesses stay visible in AI search while keeping control of their content; more is on the Baikou home page.

Want this done for your business?

Talk to us about GEO / AI search optimisation: get recommended by ChatGPT, Perplexity, Google AI.

See GEO / AI search optimisation

FAQ#

Will blocking AI training hurt my Google rankings?

Not through Google-Extended. Google states that this control is not used as a ranking signal and does not affect inclusion in Google Search. Rankings depend on Googlebot being able to crawl and index your pages.

Do I need to do anything about the shared token now?

No. Google plans to support it from early 2027, so it has no effect yet. Use the company-specific lines today and revisit the file once the providers confirm support.

Can I stop AI Overviews from quoting my page without leaving Google?

Partly. Google’s snippet controls, such as nosnippet or max-snippet, limit how much text can be shown from a page. They also limit normal search snippets, so test them on one page first.

Does robots.txt stop every AI visit?

Not every one. OpenAI notes that ChatGPT-User visits a page when a person asks ChatGPT about it, and robots.txt rules may not apply to those visits. For questions about how your content may be used legally, speak to a solicitor.

Free AI audit

Find out what is holding your site back

We look at your website and ads and send you a short written report.

  • What worksthe pages and searches already bringing you customers
  • What does notwhat is costing you visits and enquiries
  • What to do firstthree fixes, in order
Get my free AI auditReport within 2 business days. No calls, no sales pitch.

Services for this

More on ai search