Robots.txt Generator
Build a valid robots.txt with allow/disallow rules, AI crawler opt-outs, per-bot groups, crawl-delay and sitemap lines.
This guide is not yet available in your language and is shown in English.
Tool guide
What this tool does
Robots.txt Generator creates a standards-compliant robots.txt file from presets and simple rule lists. It validates paths and sitemap URLs, groups rules per user agent, adds opt-outs for documented AI crawlers and explains what each choice does to search crawling.
robots.txt is a plain text file at the root of a host that tells crawlers which paths they may request. A single misplaced slash can hide an entire site from search engines, and a forgotten rule can let crawlers waste time on endless filter and session URLs. Since 2023 site owners also use it to opt out of AI training crawlers, each of which uses its own user-agent token.
This generator follows RFC 9309, the Robots Exclusion Protocol standard published in 2022, and Google’s documented parsing behaviour. You start from a preset, add disallow and allow paths, tick the AI crawlers you want to exclude and list your sitemaps. The tool validates every path and URL, merges duplicates and adds optional comments that explain the file to colleagues.
How to use Robots.txt Generator
- Pick a preset under “Start from a preset”, or keep Custom rules.
- Choose the Default for all crawlers: allow with exceptions, or block everything.
- List Disallow paths and any Allow exceptions, one per line, starting with /.
- Tick the AI crawlers to block and add Extra bot rules in the format “BotName: /path”.
- Enter your Sitemap URLs, then download robots.txt and upload it to the root of the host.
Tips for better results
- Use * as a wildcard and $ to anchor the end of a URL, for example Disallow: /*.pdf$.
- When rules conflict, Google applies the most specific (longest) matching rule, and Allow wins a tie.
- Each subdomain and protocol needs its own file: https://shop.example.com/robots.txt is separate from www.
- Google-Extended controls use of your content for Gemini training only; it does not change how Googlebot indexes your pages.
Common use cases
- Keeping crawlers out of carts, checkouts and faceted filter URLs.
- Blocking a staging server completely while it is publicly reachable.
- Opting out of AI training crawlers such as GPTBot, ClaudeBot and CCBot.
- Advertising sitemap locations to every search engine in one place.
Common mistakes to avoid
- Using robots.txt to remove pages from search: blocked URLs can still be indexed from links. Use noindex on a crawlable page instead.
- Blocking CSS and JavaScript folders, which stops Google rendering the page properly.
- Leaving “Disallow: /” in place after launching a site that was built on staging.
- Assuming robots.txt is security: it is public and only advisory, so never list secret paths in it.
Frequently asked questions
Where do I put robots.txt?
At the root of the host, for example https://www.example.com/robots.txt. Crawlers do not look for it in subfolders, and each subdomain needs its own file.
Does Google respect Crawl-delay?
No. Googlebot ignores Crawl-delay and adjusts its crawl rate automatically. Bing, Yandex and some other crawlers do read it.
Is there a size limit?
Google processes the first 500 KiB of a robots.txt file and ignores the rest. A normal file is a few hundred bytes.
Will blocking AI crawlers affect my Google rankings?
Blocking GPTBot, ClaudeBot, CCBot or Google-Extended does not affect Google Search. Blocking Googlebot itself would remove your pages from Search, so the generator never does that unless you block everything.
Feedback & suggestions
Help us improve Robots.txt Generator. We read every message.