Yesterday we’ve all received the news that OpenAI has just released the GPTBot that’s going to start crawling our websites to feed content back to the language model. In this article I’ll dive into this in more detail and share the most important resources you need to know and I’ll also share with you how to block the GPTBot from crawling your website.

Read More: If you still don’t know what ChatGPT is then I have an article here for you on how you can use ChatGPT in your business.

So, what is this GPTbot thing?

OpenAI have announced in the following article that the GPTBot is going to be crawling websites to get more data to feed the language model. You can find their announcement here. Don’t worry about the part regarding blocking the bot. I will give you more details below on how to do that, so that you can actually implement this if you’d like to.

Here is the important quote about what this bot will be doing:

Web pages crawled with the GPTBot user agent may potentially be used to improve future models and are filtered to remove sources that require paywall access, are known to gather personally identifiable information (PII), or have text that violates our policies. Allowing GPTBot to access your site can help AI models become more accurate and improve their general capabilities and safety.

My main concern here is that nowhere it is mentioned that our content being scraped is going to function in any way like a search engine would – and that is to give credit and bring traffic back to our websites – and that is the part that irks me.

So, it’s up to you to look into this and to make up your mind on what you’re going to do about it. I am just here to give you the tools to act if you want to so let’s move on to telling you how to block the GPTBot from crawling your website.

You have the option to block the GPTBot from crawling your website

So, there is an option to block the GPTBot from crawling your website.

You have 2 options here. You can either block the bot from your entire website. Or from only specific parts.

To be able to do this you need to edit a file on your website called robots.txt. This is the file that tells all bots visiting your website how to behave. And don’t worry this is not going to impact the google bot for example. 

How to find your robots.txt file to edit

I work mostly with established business owners and most of them are running on WordPress, so I’m going to start here.

If you’re using Yoast SEO

If you’re using YoastSEO as your SEO plugin of choice, then you can follow the detailed instructions here to find your robots.txt file.

Then scroll down to find out what to add to the file.

If you’re using RankMath SEO

If you’re using RankMath SEO, then these instructions here are the ones for you.

If you have FTP Access to your website

If you have a custom site or you simply feel comfortable to download, edit and upload the file via FTP then you can do so following these instructions. Each hosting platform will look a little bit different but the general path is the same.

What if you’re on Squarespace

There are many reasons why I dislike Squarespace for serious businesses, but basically it doesn’t seem as though you have the option to edit this file on Squarespace, going by this chat here.

How to Block the GPTBot from Crawling your Website

There are 2 ways to block the GPTBot. Either you block it from the whole site OR from specific pages only.

Blocking the bot from the whole site

To block the bot from your entire website you need to simply add the following code to your robots.txt file and then save it or re-upload it depending on which method you’ve used above.

User-agent: GPTBot
Disallow: /

Blocking the bot from crawling parts of your website

To block it from parts of your site, copy the code below and amend it so that you exlude the directories you do not want crawled.

User-agent: GPTBot
Allow: /directory-1/
Disallow: /directory-2/

Doing this will stop the GPTBot from crawling your website.

If you are struggling reach out to me and I can do it for you.