# OpenAI Will Honor Shared Robots.txt Signal for AI Training

By Simon Yoon

Canonical URL: https://www.tokenpost.com/news/technology/45087
Published: 2026-10-30T20:09:48.000Z
Updated: 2026-10-30T20:09:48.000Z
Section: Technology

> The file communicates crawler preferences, but its rules do not block access. OpenAI separates its training and search crawlers, and changes do not establish that older material was removed.

OpenAI said Friday, Oct. 30, that it will honor a shared robots.txt signal through which website operators can express AI training preferences to participating providers.

The publicly accessible file at a website’s /robots.txt address uses directives such as User-agent, Allow and Disallow to communicate which pages cooperating crawlers should access or avoid. Those rules are preferences, not access authorization. A crawler that does not cooperate can ignore them.

OpenAI’s crawler setup separates training from search. A site operator can disallow GPTBot for training crawls while allowing OAI-SearchBot; allowing the search crawler can preserve a page’s eligibility for ChatGPT search.

Changing the instructions does not establish that material collected earlier has been removed from training datasets or existing models.