Robots.txt Tester

robots.txt is a text file at the root of your website that tells search engine crawlers which pages they can or cannot visit. It follows the Robots Exclusion Protocol. Misconfigured robots.txt can accidentally block Googlebot from indexing your entire site — which is why testing is critical before deploying changes.

The asterisk (*) matches any sequence of characters. For example, Disallow: /*.pdf$ blocks all URLs ending in .pdf. The dollar sign ($) anchors the match to the end of the URL. The most specific (longest) matching rule wins. If an Allow and Disallow rule are equal length, Allow takes precedence.

"Disallow:" with no value means "allow everything" — it's effectively the same as no rule. This is often used as a shorthand to explicitly permit all crawling: User-agent: * / Disallow:

GPTBot is OpenAI's crawler used to train AI models like ChatGPT. You can block it with: User-agent: GPTBot / Disallow: /. Similarly, CCBot (Common Crawl) and other AI crawlers can be blocked this way. This does not affect your Google rankings.

Blocking a page in robots.txt prevents Googlebot from crawling it, but Google can still index the URL if other pages link to it — it just won't see the content. To fully remove a page from search results, use a noindex meta tag or the URL Removal Tool in Google Search Console.