Add as a preferred source on GoogleOpenAI Will Honor Shared Robots.txt Signal for AI Training
The file communicates crawler preferences, but its rules do not block access. OpenAI separates its training and search crawlers, and changes do not establish that older material was removed.
OpenAI said Friday, Oct. 30, that it will honor a shared robots.txt signal through which website operators can express AI training preferences to participating providers.
The publicly accessible file at a website’s /robots.txt address uses directives such as User-agent, Allow and Disallow to communicate which pages cooperating crawlers should access or avoid. Those rules are preferences, not access authorization. A crawler that does not cooperate can ignore them.
OpenAI’s crawler setup separates training from search. A site operator can disallow GPTBot for training crawls while allowing OAI-SearchBot; allowing the search crawler can preserve a page’s eligibility for ChatGPT search.
Changing the instructions does not establish that material collected earlier has been removed from training datasets or existing models.
Simon Yoon
ReporterSimon Yoon reports on blockchain technology for TokenPost. Send corrections or tips to info@tokenpost.com.