Robots.txt Guide
The robots.txt file tells compliant crawlers which URL paths they should avoid requesting. It is useful for reducing crawl waste and keeping bots away from low-value technical areas. It is not a privacy system, an authentication layer, or a reliable method for removing a URL from search.
The file must be available at the root of the hostname:
https://example.com/robots.txt
A file at /folder/robots.txt does not control the whole site.
Basic syntax
User-agent: *
Disallow: /admin/
Disallow: /internal-search/
Allow: /assets/
Sitemap: https://example.com/sitemap.xml
User-agent selects a crawler. Disallow identifies a path the crawler should not request. Allow can create a more specific exception. Sitemap points to an XML sitemap.
What robots.txt should block
Good candidates include crawl-heavy areas that provide little public value:
- internal search result pages;
- faceted combinations with no search demand;
- calendar traps;
- staging paths that are protected by authentication anyway;
- generated session URLs; and
- administrative interfaces.
What it should not block
Do not block CSS, JavaScript, or image assets that are needed to render important pages. Do not block a page merely because you want it removed from search; if the crawler cannot fetch the page, it cannot see a noindex directive on that page.
Sensitive content must be protected with authentication or server authorization. A robots file is public and can reveal the paths you attempted to hide.
Rules are hostname-specific
www.example.com, example.com, and shop.example.com each require their own file. The same is true for HTTP and HTTPS environments in crawler processing, although production sites should redirect to one HTTPS host.
AI crawler policy
Create rules based on business intent. You may choose to allow search and answer retrieval while limiting model-training bots, but bot names and policies vary by provider and can change. Keep a short internal document that records why each bot-specific rule exists.
Example structure:
User-agent: ExampleTrainingBot
Disallow: /
User-agent: *
Disallow: /account/
Disallow: /billing/
Sitemap: https://example.com/sitemap.xml
Use real, current user-agent names only after checking the provider’s documentation.
Common failure modes
Accidental sitewide block
User-agent: *
Disallow: /
This is correct for some private environments but disastrous on a public production site.
Wrong capitalization or path assumptions
Path matching can be case-sensitive depending on the URL. /Private/ and /private/ may be different resources.
Staging rules copied to production
Deployment pipelines should test the production robots file automatically. A simple smoke test can fail the release if the production host contains a sitewide block.
Blocking parameter URLs without understanding internal links
It is better to fix the links that generate useless URLs than to rely only on blocking the crawler after the URLs exist.
Testing process
- Request
/robots.txtfrom the production hostname. - Confirm a 200 response and plain-text content type.
- Review each group from top to bottom.
- Test representative allowed and blocked URLs.
- Check the sitemap line.
- Inspect server logs after deployment.
- Recheck after CMS, CDN, or migration changes.
Checklist
- [ ] File is located at the hostname root.
- [ ] Production is not blocked sitewide.
- [ ] Important assets remain crawlable.
- [ ] Sensitive paths use authentication, not robots.txt.
- [ ] Internal search and crawl traps have a documented policy.
- [ ] Sitemap URL is correct and returns 200.
- [ ] Bot-specific rules reflect current business intent.
- [ ] Deployment tests detect accidental changes.
Frequently asked questions
Does robots.txt remove a page from search?
Not reliably. A blocked URL may remain known through external links. Use appropriate removal and indexation controls instead.
Can malicious bots ignore robots.txt?
Yes. The standard depends on voluntary compliance.
Can I have more than one sitemap line?
Yes. Multiple sitemap declarations are valid when the site uses several sitemap files or indexes.
Turn the checklist into an action plan
A useful guide should end with a decision, not another browser tab. Run the site through Rofix, review the highest-impact findings first, and assign each fix to an owner. Re-run the audit after deployment so the team can verify that the issue is actually resolved rather than merely marked complete.