Multiple crawler groups
Use wildcard, Googlebot, Bingbot, GPTBot or a custom user agent.
Robots.txt Generator
Robots.txt Generator
Configure crawler groups, path rules and sitemap declarations.
robots.txt output
User-agent: *
- No sitemap URL has been declared.
Crawler access control
What is robots.txt?
Robots.txt is a public file at the site root that tells compliant crawlers which paths may or may not be fetched. Rules are grouped by User-agent and may declare sitemap locations.
It is not access control and does not reliably prevent indexing. Protect sensitive content with authentication and use noindex or removal responses for search-result control.
Configurable rules
Allow and Disallow
Maintain rules per group and flag invalid paths or full-site blocking.
Sitemap declarations
Add and deduplicate one or more absolute sitemap URLs.
Why generate robots.txt carefully?
One broad Disallow can block an entire site or critical resources, while invalid syntax may simply be ignored.
Centralize crawl scope
Set clear boundaries for admin, search parameters and low-value paths.
Catch dangerous rules
Flag full-site blocks, invalid paths, duplicate groups and unsupported directives.
Declare sitemap locations
Publish the canonical sitemap endpoints in the same root file.
How does the robots.txt generator work?
All configuration is processed locally in the browser and is never stored.
- 01
Create crawler groups
Choose a common user agent or enter the exact crawler token.
- 02
Add path rules
Maintain Allow and Disallow paths for each group.
- 03
Declare sitemaps
Add complete HTTPS sitemap URLs and remove duplicates.
- 04
Review and publish
Copy or download the file and serve it from /robots.txt.
Robots.txt generator FAQ
Does Disallow prevent indexing?
Not always. A linked URL may still appear without a snippet. Use noindex on a crawlable page or remove the page when indexing must stop.
Can robots.txt protect an admin area?
No. It is public. Protect private areas with authentication, authorization and network controls.
Does Google support Crawl-delay?
Googlebot does not support Crawl-delay in robots.txt. Manage crawl behavior through site performance, response status and Search Console.
What happens when Allow and Disallow conflict?
Major crawlers generally use the longest matching rule. Avoid relying on ambiguous equal-length conflicts.
Where must robots.txt be published?
At the protocol and host root, for example https://example.com/robots.txt.
Related SEO tools
After publishing, check the real response, group parsing and path behavior.
XML Sitemap GeneratorCrawl same-site pages and build a valid XML sitemap.Meta Tag GeneratorCreate search and social sharing metadata.Schema JSON-LD GeneratorBuild structured data for common entity types.SEO Slug GeneratorTurn Chinese titles into short readable URL paths.Robots.txt checkerCheck the live file, crawler groups and path access.