Tools · Generators · SEO Last update: Nov 16, 2026

Robots.txt Generator

Create a robots.txt file – presets for WordPress and online shops, your own rules, the sitemap line and optional blocking of AI crawlers.

Preset
Block AI crawlers

What goes into robots.txt

  • User-agent – which crawler a group of rules is for; * means all.
  • Disallow / Allow – paths the crawler may not or may fetch; * matches any characters and $ the end of the URL.
  • Sitemap – the absolute URL of your XML sitemap.

Presets

  • WordPress – blocks the admin area (but keeps admin-ajax.php reachable) and internal search results.
  • Online shop – blocks cart, checkout and account pages and sorting and filter parameters that create endless URL variants.
  • Allow all and Block all – for open sites and for staging copies.

Example: the WordPress preset with two extra paths and five AI crawlers blocked gives a file with 11 rules and the sitemap line.

Blocking AI crawlers

GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl) and Bytespider (ByteDance) collect pages for training AI models. OAI-SearchBot and PerplexityBot crawl for AI search answers – blocking them can remove your site from those answers. Google-Extended and Applebot-Extended are not separate crawlers but tokens: blocking them stops use of your content for Gemini and Vertex AI or Apple’s models without affecting Google or Apple search. Crawlers decide themselves whether to respect robots.txt.

Crawling is not indexing

A page blocked in robots.txt can still appear in search results if other sites link to it. To keep a page out of the index, allow crawling and add a noindex tag.

Frequently asked questions

Where do I put robots.txt?

In the root of the domain, e.g. https://www.example.com/robots.txt; each subdomain needs its own file.

Does robots.txt stop a page from being indexed?

No – it only stops crawling. Use a noindex tag to keep a page out of search results.

How do I block ChatGPT and other AI crawlers?

Add a group for each crawler, such as User-agent: GPTBot followed by Disallow: /.