JPKCom DB

robots.txt & Sitemap — Tips & Tricks

Insider moves for robots.txt & Sitemap: common pitfalls, AI-crawler strategy, how the two files work together, and combining it with the SEO & GEO Analyzer.

Back to the overview: robots.txt & Sitemap · Open the tool: www.jpkc.com/tools/robots-sitemap/

The manual explains every function; the examples show the workflows. This page covers what both assume: where the typical mistakes hide, how to treat AI crawlers strategically, and how to combine the tool sensibly with others. The interface is in English, so the real tab and button names appear as you'll see them.

robots.txt: the most dangerous pitfalls

  • Disallow: / blocks your entire website. A single line in the wrong block locks out all crawlers completely — and with it your site from Google. Build the file in the generator, check it with the live test below the preview, and after uploading verify in the Check robots.txt tab that Googlebot and co. really are Allowed. If Disallow: / sits in the * block, the notes list warns you explicitly.
  • Rules are prefixes. Disallow: /admin also blocks /administrator and /admin-guide/; Disallow: /admin/ blocks only the directory. For patterns there's * and a trailing $ (/*.pdf$) — and because crawlers compare path plus query string, /*?s= also catches search queries on subpages. When in doubt, type the path into the live test.
  • A group of its own overrides *. Once a bot has its own group, only that group applies to it — every block from User-agent: * is gone for that bot. If you want to allow GPTBot something specific, repeat the important blocks in its group.
  • Blank lines don't separate groups. Two User-agent lines with a blank line between them still share the rules below. Only a User-agent line after a rule starts a new group. The checker points out such blank lines; the generator sidesteps the problem by giving a block without rules an Allow: /.
  • A server error blocks everything. If /robots.txt returns a 5xx status or 429, crawlers treat the whole site as blocked — whereas a 404 allows everything. After server moves or CDN changes, check in the Check robots.txt tab which status the file really returns, and whether an HTML page comes back instead of the file.
  • robots.txt is not security. The file is publicly readable and obeyed by crawlers only voluntarily. Never write secret paths into Disallow rules — you'd reveal them to anyone who opens your-domain.com/robots.txt. Whatever truly needs protecting belongs behind server-side authentication.
  • Blocking is not the same as de-indexing. A page blocked in robots.txt can still show up in Google's results (without a snippet) when other pages link to it. To keep a page out of the index, use a noindex meta tag or the X-Robots-Tag HTTP header — and then do not block it via robots.txt, or Google never sees the noindex.
  • Never block CSS and JavaScript. Google renders your pages to understand them. Block /wp-content/ or an asset folder wholesale and your pages look broken — bad for ranking. Be specific (see the WordPress template in the Examples tab).
  • Google ignores Crawl-delay. Most bots support the directive, but not Googlebot — there you control crawl frequency via Search Console. Bing and Yandex honor it, as the tool's notes list also tells you. For those bots a moderate value (e.g. 5–10 seconds) makes sense; don't overdo it or you needlessly slow down indexing.
  • Bot names without a version. Crawlers compare only the product name: User-agent: Googlebot/2.1 acts like Googlebot, and a shortened User-agent: google doesn't match Googlebot at all. Take the names from the autocomplete and they'll be right.

Steer AI crawlers deliberately

The tool's real edge over simple generators is its view of AI crawlers — both when building (40 bot suggestions in the autocomplete field, two AI templates) and when checking (the per-bot table with type and applicable group).

  • Decide per purpose, not wholesale. The bot table in the Reference tab distinguishes three purposes: AI training (e.g. GPTBot, ClaudeBot, Google-Extended, CCBot), AI search (e.g. OAI-SearchBot, Claude-SearchBot, PerplexityBot), and AI user request (ChatGPT-User, Claude-User, Perplexity-User — fetches a person triggers in a chat). If you want to be cited in AI answers but stay out of training, take the Block AI Training, Allow AI Search template; if you want no AI access at all, Block All AI Crawlers.
  • Bundle the bots in one group. A block with GPTBot, ClaudeBot, CCBot is easier to read than three identical blocks — the generator turns it into one group with three User-agent lines. Crawlers merge several groups for the same bot anyway; the checker points that out.
  • Google-Extended ≠ Googlebot. Google-Extended blocks only Gemini training, not classic Google search. Anyone who blocks everything Google-like out of fear of "AI" accidentally throws away their normal visibility.
  • Re-check after editing. Even before uploading, the generator's live test shows whether a bot is blocked. After uploading, check the file in the Check robots.txt tab: the per-bot table shows you in black and white which AI crawler is now Allowed and which Blocked — including the group and the rule that actually applies.
  • Mind the GEO consequences. Blocked AI crawlers cost points in the SEO & GEO Analyzer's GEO score: its AI Crawlers Allowed check tests all 16 AI bots from the same bot list and deducts in steps — of 3 points, 2 remain with 1 to 3 blocked bots, 1 with 4 to 6, none from 7 on. So even a single blocked bot costs a point. Blocking is a deliberate decision against AI visibility — not an accident that should happen in passing.

Sitemap: clean over complete

  • Only canonical URLs that return HTTP 200. No duplicates with ?utm_* or session parameters, no redirects, no 404s. Every dead or duplicate URL wastes crawl budget.
  • No blocked or noindex pages in the sitemap. This is the golden rule: a URL never belongs in both a Disallow of the robots.txt and the sitemap — that contradiction confuses crawlers. The sitemap says "please index", the robots.txt says "don't fetch". The Check Sitemap tab cross-checks both files automatically and names the URLs that are blocked for Googlebot.
  • Only target URLs, no redirects. The Do the URLs work? sample in the checker finds redirects and broken pages. For large sitemaps it checks 20 URLs spread evenly across the list — if hits pile up there, it pays to look at the whole list, which you take into your own tooling via Copy all URLs.
  • lastmod only with a real date. A faked modification date, or one set wholesale to "today", undermines crawlers' trust in your signal. Better to leave the field empty than to lie — and use the Empty lastmod → today button only when the pages really changed today. The Metadata table in the checker shows how consistently you use lastmod.
  • You can skip changefreq and priority. Google ignores both fields; that's why the tool leaves them empty by default and marks them "ignored by Google" in the checker. What matters is an accurate lastmod.
  • Leave special characters to the generator. An & in a query string must appear as & in XML, and non-ASCII characters and spaces belong percent-encoded. The generator takes care of both; a hand-built sitemap with a raw & isn't well-formed — and the checker reports it as an error, because crawlers discard the whole file.
  • Split large sites. 50,000 URLs or 50 MB is the limit per file — beyond that, use several sitemaps plus an index file, which you build in the generator via File type Sitemap index. The checker handles gzip-compressed .xml.gz files too. The Check Sitemap tab reports more than 50,000 URLs and can take an index file apart child by child. It checks the 50 MB limit only in Expert Mode: through the server proxy a file that large never arrives in full — it is truncated first and reported as incomplete.

Privacy and operation

  • Generators stay local. The two generator tabs produce and check everything in the browser; your working state is stored only in your LocalStorage, nothing is uploaded. Reset (with a confirmation) clears it.
  • The checkers run via a proxy — by design. A browser can't load a foreign file directly because of CORS. The server-side proxy fetches it; the checked domain therefore sees a request from the JPKCom server, not your IP. Handy when you don't want to show up in a foreign site's crawler log.
  • localhost and intranets won't work. For SSRF protection the proxy blocks private and internal addresses. You check a local dev instance either via a public staging domain or via Expert Mode with a local proxy.
  • Sit out the rate limit. In standard mode at least 3 seconds must pass between two checks, and server-side 120 fetches per minute are possible. If a wait note appears, just wait a moment instead of firing again.
  • Very large sitemaps may get truncated. The proxy loads up to 5 MB and decompresses .xml.gz files to at most 10 MB; beyond that the checker reports that the results are incomplete. For complete fetches there's Expert Mode.

Combining with other JPKCom tools

  • Build here, check in context. You build the robots.txt and the sitemap in this tool — the effect on a concrete page you see in the SEO & GEO Analyzer: its Robots Analysis tab and the checks Allowed by robots.txt, Sitemap in robots.txt, and AI Crawlers Allowed work on exactly what you produce here — and use the same parser, so both tools reach the same verdict. Sequence: build here → upload → check the same URL in the analyzer → read it green.
  • Complement the GEO side. The sitemap is the roadmap for classic search engines; the machine-readable counterpart for LLMs you create with the llms.txt Generator.
  • Optimize the listed pages. Every URL in your sitemap should have clean meta data — which you build with the Meta Tags Generator.

More context: the overview for the big picture, the manual for every option, and the examples for the step-by-step workflows. You can try all of it directly in the tool.