Live
Sitemap Generator
Generate a sitemap.xml from your page paths.
Web · Free tool
robots.txt Generator
Write crawl rules the correct way — user agents, allow/disallow paths, and a Sitemap: line — without memorizing the syntax.
Build robots.txt
Used in the header comment only.
robots.txt
How it works
How robots.txt works
Groups of rules
Each user agent gets its own group of Allow/Disallow lines. The * group applies to everyone; a group for a named bot overrides it for that bot.
Paths, not pages
Rules match path prefixes — /admin/ covers the whole folder. Matching is case-sensitive and relative to the root, so absolute URLs belong only on the Sitemap: line.
It is a suggestion
robots.txt is honored by compliant crawlers (Google, Bing, and friends). It is not access control — for truly private content, require login on the server.
Questions
robots.txt FAQ
What does User-agent: * mean?
The asterisk matches every crawler. Rules under it apply to all bots unless a specific bot has its own group — specific groups win, so Googlebot can get different rules than everyone else.
Should I disallow or noindex?
Disallow blocks crawling — the page may still show in results via links, but without its content. For pages you want gone from search, a noindex meta tag on the page is the modern, reliable signal.
Where do I put the file?
At the root: https://yourdomain.example/robots.txt. Nothing else is served from that path, and the filename is case-sensitive on most servers.