Web & API
URL Anatomy and Percent-Encoding: Query Strings Without the Bugs
Break down URL parts, encode query values correctly, and know when to use encodeURIComponent vs encodeURI — plus reserved characters that break APIs.
Jul 22, 20266 minRead
A broken robots.txt can block your entire site from Google, leave staging paths crawlable, or silently fail because of a typo in a directive name. The file lives at /robots.txt on every domain, but most teams only look at it when Search Console shows a crawl anomaly — or when nothing gets indexed at all.
Bottom line: Build a draft with the Robots.txt Generator, then paste into a checker before every deploy. Confirm User-agent groups are valid, Disallow paths match intent, and Sitemap URLs are absolute. Remember: robots.txt controls crawling, not indexing — use noindex meta tags when you need pages out of search results.
What robots.txt does (and does not do)
| robots.txt | Does | Does not |
|---|---|---|
| Crawling | Tells compliant bots which paths to skip | Remove pages already indexed |
| Indexing | Indirect — uncrawled pages may not be found | Replace <meta name="robots" content="noindex"> |
| Security | Nothing — the file is public | Hide admin panels or API keys |
| Performance | Can reduce crawl budget waste | Fix slow pages or Core Web Vitals |
Google, Bing, and other major crawlers voluntarily follow robots rules. Malicious bots ignore them. Treat the file as a polite request to search engines, not access control.
File location and format
Every site serves the file at the root:
https://example.com/robots.txt
Rules:
- Plain text, UTF-8
- One directive per line:
Directive: value - Lines starting with
#are comments - Directives are case-insensitive; values are usually case-sensitive for paths
- Blank lines are ignored
Minimal valid example:
User-agent: *
Disallow: /admin/
Sitemap: https://example.com/sitemap.xml
Core directives
| Directive | Purpose | Example |
|---|---|---|
User-agent |
Which bot the following rules apply to | User-agent: Googlebot |
Disallow |
Paths the bot should not crawl | Disallow: /private/ |
Allow |
Override a broader Disallow (Google/Bing) | Allow: /public/ |
Sitemap |
Absolute URL to your XML sitemap | Sitemap: https://example.com/sitemap.xml |
Crawl-delay |
Seconds between requests (non-Google) | Crawl-delay: 10 |
Host |
Preferred mirror domain (Yandex) | Host: example.com |
User-agent groups
Rules apply to the most recent User-agent line until the next one:
User-agent: Googlebot
Disallow: /search/
User-agent: *
Disallow: /admin/
Allow: /
Sitemap: https://example.com/sitemap.xml
User-agent: * matches all bots not covered by a more specific group.
Disallow patterns
| Pattern | Meaning |
|---|---|
Disallow: / |
Block entire site for that user-agent |
Disallow: (empty) |
Allow everything (no restrictions) |
Disallow: /tmp/ |
Block paths starting with /tmp/ |
Disallow: /*.pdf$ |
Block URLs ending in .pdf (Google supports * and $) |
Common mistake: Disallow: / on User-agent: * after a deploy typo blocks all crawling. Search Console may show "Blocked by robots.txt" for every URL.
Allow overrides Disallow
When both apply to the same URL, the more specific rule wins (Google). Useful for opening one public page under a blocked directory:
User-agent: *
Disallow: /docs/
Allow: /docs/public/
How to check robots.txt
1. Fetch the live file
Open https://yoursite.com/robots.txt in a browser or run:
curl -s https://yoursite.com/robots.txt
Confirm HTTP 200, Content-Type: text/plain, and no HTML error page masquerading as robots.txt.
2. Validate syntax
Paste the content into the Robots.txt Checker. It flags:
- Missing
User-agentdirective - Malformed lines (missing colon or value)
- Unknown directive names
Syntax validation catches typos like User-agent * (missing colon) or Disalllow: / before they reach production.
3. Test in Google Search Console
Search Console → Settings → robots.txt (or the URL Inspection tool) shows whether Google can fetch and parse your file. Use URL Inspection on a specific page to see if it is "Blocked by robots.txt."
4. Cross-check with sitemap
List sitemap URLs in robots.txt and validate the XML separately with the Sitemap.xml Validator. A sitemap pointing to URLs that robots blocks sends mixed signals — usually unintentional.
Common mistakes
| Mistake | Symptom | Fix |
|---|---|---|
Disallow: / on * |
Zero pages crawled | Remove or narrow the rule |
No User-agent line |
Invalid file | Add User-agent: * before rules |
| Relative sitemap URL | Sitemap ignored | Use absolute URL with https:// |
Blocking / in staging robots copied to prod |
Site de-indexed over time | Environment-specific robots |
Using robots for noindex intent |
Page still indexed if linked externally | Add noindex meta or X-Robots-Tag |
| Typo in directive name | Rule silently ignored | Validate syntax before deploy |
| Disallowing CSS/JS | Rendering issues in search | Allow assets Google needs to render |
robots.txt vs meta robots vs X-Robots-Tag
Use the right lever:
| Goal | Mechanism |
|---|---|
| Stop crawling a path | Disallow in robots.txt |
| Keep out of search results but allow crawl | <meta name="robots" content="noindex"> |
| HTTP-level control | X-Robots-Tag: noindex header |
| Block images from image search | Disallow in robots or noindex on image URL |
For developer tool sites like codefunc: index public tool pages, allow / in robots.txt, and disallow only internal paths like /api/ if they exist.
Example robots.txt files
Standard marketing site
User-agent: *
Allow: /
Disallow: /api/
Disallow: /admin/
Sitemap: https://example.com/sitemap.xml
Block staging entirely
User-agent: *
Disallow: /
Use only on non-production hosts — never copy this to production.
Allow Googlebot full access, restrict others
User-agent: Googlebot
Allow: /
User-agent: *
Disallow: /
Sitemap: https://example.com/sitemap.xml
Pre-deploy checklist
- File returns HTTP 200 at
/robots.txt - At least one
User-agentgroup exists - No accidental
Disallow: /on production -
SitemapURLs are absolute and reachable - Syntax validated with a robots.txt checker
- Sitemap XML validated separately
- Search Console shows successful fetch after deploy
Practical takeaway
Check robots.txt every time you ship — not only at launch. Paste the file into a validator, fix syntax errors, and read each Disallow line as "this path will not be crawled." Pair robots checks with sitemap validation and correct noindex tags when you need pages out of search results. For deeper SEO hygiene, see the meta tags guide and AI SEO indexing guide.
Related on codefunc
- Robots.txt Checker — validate syntax locally
- Sitemap.xml Validator — check XML structure
- Meta Tags SEO Guide — titles, canonicals, Open Graph
- Meta Tags Generator — copy-paste head tags