What changed
Google has announced plans to support a shared AI training control that websites can place in their robots.txt file. The idea comes from the AIPREF working group at the Internet Engineering Task Force (IETF). Website owners would no longer need a separate instruction for each AI developer. Instead, they could use one shared policy token to state a broad training preference that applies across participating AI providers.
If a site uses this token to opt out of training and does not specifically mention Google-Extended, Google plans to treat that as a disallow of Google-Extended. Google-Extended is the AI training control Google has offered since 2023. It will keep working, along with other company-specific controls. Google describes this as a draft proposal shaped by feedback from publishers, civil society groups and regulators. It aims to support the control alongside other AI providers in 2027.
Why it matters
For a business owner, this could make AI training choices much simpler to manage. One line could cover many AI companies, while you still keep the option to give specific instructions to individual providers. Google also points out that the choice carries a visibility tradeoff. The Google Search Central Blog notes:
Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps.
In practice, a blanket opt-out could affect how your site shows up in Google’s AI experiences. Google also says the proposal is consistent with the EU Code of Practice for General Purpose AI model providers, which recognizes robots.txt as a primary way for publishers to express machine-readable preferences.
What to do
- Review your robots.txt now. Check whether you already allow or block Google-Extended or other AI crawlers, and confirm that the current setup reflects your goals.
- Decide your AI stance. Weigh protecting your content from training against being eligible to appear and earn clicks in Gemini Apps.
- Plan for the interaction. If you want a broad opt-out but still want visibility in Gemini Apps, you will likely need to explicitly allow Google-Extended once the shared token is supported.
- Wait for the final details. This is still a draft. Hold off on major changes until the token format is standardized and providers confirm their support.
- Watch for updates. Google says it will share progress as work continues through the IETF.
Source: Supporting a shared AI training control for websites, Google Search Central Blog. This brief sums up their reporting; all credit for the original goes to them.
