OpenAI has launched web crawler GPTBot to improve artificial intelligence models.
"Web pages crawled with the GPTBot user agent may potentially be used to improve future models and are filtered to remove sources that require paywall access, are known to gather personally identifiable information (PII) or have text that violates our policies," the company said in a post on its website.
"Allowing GPTBot to access your site can help AI models become more accurate and improve their general capabilities and safety," OpenAI wrote.
A web crawler is a type of bot.
It is usually operated by search engines that index the content of websites for the sites to appear in search results, according to internet company Cloudflare.
They are called "web crawlers" because crawling is the term for automatically accessing a website and obtaining data using software.
OpenAI also provided instructions on disallowing the GPTBot from accessing a website – either partially or fully.
Websites can block the crawler's IP address or add the GPTBot to the site’s robots.txt file. The file essentially instructs web crawlers on what is accessible from a site.
"To allow GPTBot to access your only parts of your site you can add the GPTBot token to your site’s robots.txt," it explained.
"For OpenAI's crawler, calls to websites will be made from the IP address block documented on the OpenAI website," OpenAI concluded.
Notably, AI companies, including OpenAI, previously signed an agreement with the White House to develop a watermarking system to let internet users know if something was generated by AI. However, the organizations have not pledged to stop using internet data for training.
