Build a correct sitemap.xml, keep lastmod honest, submit it in Google Search Console, and pair it with robots.txt so crawlers discover the pages you actually want indexed.
By codefunc
A sitemap does not force Google to index your site, and it does not override noindex or robots rules. It is a discovery hint: a machine-readable list of URLs you consider important, with optional metadata like lastmod. Done well, it speeds up discovery of new tool pages and blog posts. Done poorly — broken XML, blocked URLs, fake dates — it wastes crawl budget and confuses Search Console reports.
Bottom line: Ship a valid XML sitemap of canonical, indexable URLs. Keep lastmod truthful. Reference the sitemap from robots.txt, submit it in Search Console, and re-validate after every deploy that changes routes.
Developers and technical SEO owners of marketing sites, docs, and utility hubs (including SSG sites like codefunc) who need reliable indexing without a CMS plugin. If you generate sitemap.xml at build time, this workflow fits how modern Next.js and static sites ship.
Search engines discover URLs through links and through sitemaps. Sitemaps help when:
Sitemaps do not:
noindexThe common format is Sitemap Protocol 0.9 as XML:
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://example.com/</loc>
<lastmod>2026-07-17</lastmod>
</url>
<url>
<loc>https://example.com/tools/json-formatter</loc>
<lastmod>2026-07-10</lastmod>
</url>
</urlset>
| Element | Required | Notes |
|---|---|---|
urlset / sitemapindex |
Yes | Root element |
url → loc |
Yes | Absolute URL; prefer canonical https |
lastmod |
No | W3C datetime; be honest |
changefreq |
No | Often ignored by Google; optional |
priority |
No | Often ignored by Google; optional |
For most sites, loc + accurate lastmod is enough. Do not spend time fine-tuning priority values that crawlers may ignore.
If you exceed 50,000 URLs or 50 MB uncompressed per file, split into multiple sitemaps and point to them from a sitemap index:
<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap>
<loc>https://example.com/sitemaps/tools.xml</loc>
<lastmod>2026-07-17</lastmod>
</sitemap>
<sitemap>
<loc>https://example.com/sitemaps/blog.xml</loc>
<lastmod>2026-07-17</lastmod>
</sitemap>
</sitemapindex>
Validate both the index and child files with the Sitemap.xml Validator.
lastmod should reflect when the page content meaningfully changed, not when the CI job ran. Inflating lastmod on every deploy trains crawlers to distrust your dates.
Good practice:
Acceptable formats include 2026-07-17 and full ISO timestamps like 2026-07-17T14:00:00+00:00. Stay consistent across the file.
Include:
https:// URLs matching your preferred host (www vs apex)Exclude:
noindex pages?utm_…, sort variants)Cross-check titles and canonicals with the Meta Tags Generator when you add new templates — a sitemap entry cannot fix a missing or conflicting canonical.
Your robots file should allow crawling of sitemap URLs and should advertise the sitemap location:
User-agent: *
Allow: /
Disallow: /api/
Sitemap: https://example.com/sitemap.xml
Rules of thumb:
Disallows a path, remove it from the sitemap (or fix the disallow). Listing blocked URLs creates Search Console noise.Sitemap: URL.Disallow: /) and should not be submitted to production Search Console properties.Validate syntax with the Robots.txt Checker before deploy. For a deeper robots walkthrough, see the robots.txt checker guide.
sitemap.xml (or the index path), and submit.lastmod + crawl — resubmitting is optional if the same URL still hosts a fresh file.Common Search Console failures:
| Symptom | Likely cause | Fix |
|---|---|---|
| Couldn't fetch | 404, 5xx, or blocked by robots | Serve 200 at the sitemap URL; allow crawlers |
| Sitemap is HTML | Soft 404 or SPA fallback | Return application/xml (or correct XML content type) |
| Invalid URL | Relative loc, wrong host |
Absolute canonical URLs only |
| Submitted URL not indexed | Quality, duplication, or noindex |
Fix page-level indexing signals |
On SSG sites, generate the sitemap at build time from the same route registry that powers navigation. That prevents drift where a new /tools/{slug} page ships without a sitemap entry.
Checklist per release:
loc returns 200 and is canonicalnoindex URLs listedrobots.txt allows those paths and references the sitemapGoogle supports extensions (image, video, news) and hreflang annotations for multilingual sites. Add them only when you have a concrete need — empty image entries and incorrect hreflang pairs create more problems than they solve. For a single-language developer tools site, a clean urlset of page URLs is usually enough.
Treat the sitemap as an inventory of canonical, crawlable, indexable URLs — not a ranking lever. Keep XML valid, lastmod honest, and robots.txt aligned so you never advertise blocked paths. Submit once in Search Console, then validate on every release that adds or removes routes. Use codefunc's Sitemap.xml Validator, Robots.txt Checker, and Meta Tags Generator to catch structural and indexing mismatches before crawlers do.
A practical guide to search indexing in the AI era — what crawlers still need, how LLMs surface content, and where AI tooling actually speeds up discoverability without spam penalties.
How to count words and characters the way products and platforms do — limits, diffs, Markdown pitfalls, and a practical editing workflow for developers who ship copy.
How to format and inspect XML without breaking namespaces, attributes, or mixed content — plus when to beautify vs minify and how XPath fits the workflow.
Find a developer tool