robots.txt basics
robots.txt sits at the site origin (/robots.txt) and lists crawl preferences by User-agent.
- User-agent: which crawler the following rules apply to (
*= default / all). - Disallow: path prefixes the crawler is asked not to fetch.
- Allow: path prefixes permitted (useful when a broader Disallow exists).
- Sitemap: discovery hints; does not override Disallow preferences.
Advisory, not enforcement
Well-behaved crawlers may honor the file. Malicious scrapers and some products ignore it. Prefer authentication, network controls, and terms for access you actually need to enforce.
robots.txt vs llms.txt
robots.txt is a long-standing crawl preference convention. llms.txt is a separate curation file for humans and cooperative agents — it does not replace robots rules.