XML Sitemaps: Build a Clean Discovery Feed
An XML sitemap is a machine-readable list of URLs you want crawlers to discover. It is especially useful for large sites, new sites, media-heavy sites, and pages that are not linked frequently. A sitemap does not force indexing. It is a discovery and monitoring aid.
What belongs in a sitemap
Include canonical URLs that return 200 and are intended for indexing. Exclude redirects, error pages, blocked URLs, duplicate parameter versions, and pages marked noindex. The sitemap should describe the preferred site, not every URL the server can generate.
Sitemap indexes
Large sites can split URLs by content type, locale, or update workflow and list those files in a sitemap index. Logical grouping makes monitoring easier. Do not create hundreds of tiny files without an operational reason.
Last modification dates
Use modification timestamps only when they reflect meaningful content changes. Updating every timestamp on every deployment teaches crawlers that the field is unreliable.
Images, video, and news
Specialized sitemap extensions can provide additional metadata for eligible content. Use them when the site has a real need and can keep the information accurate.
Submission and monitoring
Reference the sitemap in robots.txt and submit it through relevant webmaster tools. Compare submitted URLs with indexed and excluded URLs. A growing gap can reveal canonical conflicts, thin pages, crawl failures, or accidental noindex rules.
Implementation checklist
- [ ] Include only canonical indexable URLs.
- [ ] Remove redirecting and error URLs.
- [ ] Use HTTPS and the preferred hostname.
- [ ] Keep modification dates accurate.
- [ ] Split large files logically.
- [ ] Reference the sitemap in robots.txt.
- [ ] Submit it in webmaster tools.
- [ ] Monitor submitted versus indexed counts.
- [ ] Regenerate after URL migrations.
- [ ] Validate XML after deployment.
Frequently asked questions
Does every website need a sitemap?
Small, well-linked sites may be discovered without one, but a clean sitemap is still useful for monitoring and migrations.
Can a URL be indexed if it is not in the sitemap?
Yes. Crawlers discover URLs through links and other sources.
Should noindex pages appear in the sitemap?
No. Including them sends conflicting intent.
Turn the checklist into an action plan
A useful guide should end with a decision, not another browser tab. Run the site through Rofix, review the highest-impact findings first, and assign each fix to an owner. Re-run the audit after deployment so the team can verify that the issue is actually resolved rather than merely marked complete.