Web & Dev

robots.txt Generator

Build a robots.txt with per-user-agent allow/disallow rules, crawl-delay and a sitemap URL.

User-agent groups

Group 1

User-agent

Crawl-delay (seconds, optional)

Allow paths (one per line)

Disallow paths (one per line)

Sitemap URL (optional)

User-agent: *
Disallow: /admin
Disallow: /private

Sitemap: https://example.com/sitemap.xml