JPKCom DB

robots.txt & Sitemap — Examples

Hands-on walkthroughs with robots.txt & Sitemap: build a WordPress robots.txt, block AI crawlers, generate a sitemap, and check a foreign file live.

Back to the overview: robots.txt & Sitemap · Open the tool: www.jpkc.com/tools/robots-sitemap/

The manual explains every tab and option in detail. This page adds concrete workflows: typical tasks, played through step by step. The tool's interface is in English, so tab and button names appear exactly as you'll see them.

Example 1: Build a robots.txt for WordPress

The classic — a clean robots.txt for a typical WordPress site.

  1. Open the tool, go to the Examples tab, and click Load into Generator on WordPress. Once you confirm that the form may be replaced, the tool jumps to the robots.txt tab and fills it with the WordPress template.
  2. Look at the User-agent: * block: it blocks /wp-admin/ but explicitly frees /wp-admin/admin-ajax.php via Allow (WordPress needs it for front-end features; being the longer rule, the Allow wins), and the wildcard rules /*?s= and /*?p= block internal search result pages and addresses with the ?p= parameter. /wp-includes/ and /wp-content/ stay open on purpose — that's where CSS, JavaScript, and images live.
  3. Adjust the sitemap URL. The template enters https://example.com/wp-sitemap.xml. Replace it with the real address of your sitemap (an SEO plugin often uses a different one) — the Sitemaps field takes several, one per line.
  4. Check the preview on the right and the notes below it, then Copy or Download. Upload the file to your web root so it's reachable at https://your-domain.com/robots.txt.

Important: never block /wp-content/ or /wp-includes/ wholesale — they hold your images plus the CSS and JavaScript Google needs to render your pages.

Example 2: Block AI crawlers, allow search engines

You want to stay visible in classic search engines, but you don't want AI crawlers scraping your content for training. Whether AI search services may still find and cite you is a separate decision.

  1. In the Examples tab, pick one of the two AI templates and click Load into Generator:
    • Block AI Training, Allow AI Search blocks only the training crawlers. AI search and fetches a person triggers in a chat (ChatGPT, Claude, Perplexity) stay allowed.
    • Block All AI Crawlers also blocks the search and user-request crawlers; classic search engines may still fetch everything.
  2. Both templates create an open User-agent: * block with an empty Disallow: (everyone allowed) and below it one block with Disallow: / that bundles the AI crawlers, comma-separated. For Block AI Training those are GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, CCBot, and Bytespider; Block All AI Crawlers adds OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, and Perplexity-User. In the output each bot gets its own User-agent: line above the shared rule.
  3. Add missing bots as needed. Just type them comma-separated into the AI block's User-agent field — the autocomplete suggests Amazonbot, DuckAssistBot, and MistralAI-User, among others.
  4. Verify the result with the live test below the preview: Googlebot must be Allowed for /, GPTBot Blocked.
  5. Copy/Download, upload, done.

Two honest notes: first, not every bot respects robots.txt — it's a voluntary policy, not a technical barrier. Second, blocking Google-Extended (Gemini training) does not exclude Google's normal search; that keeps running via Googlebot. Which bot serves which purpose (AI training, AI search, AI user request) is in the bot table in the Reference tab.

Example 3: Build an XML sitemap by hand

For a smaller site without a sitemap plugin, you build the file right in the tool.

  1. Switch to the Sitemap tab. Leave File type on Sitemap (pages) and enter your domain in Base URL (e.g. https://your-domain.com).
  2. Add the pages. The fastest way is Add many at once: paste a list, one path per line (/, /services/, /contact/ …), then click Add lines. Single rows are added with Add entry.
  3. Only set lastmod when you actually know the date. Enter it as 2026-10-15, with time and time zone if needed (2026-10-15T09:30:00+02:00). A made-up modification date does more harm than good — otherwise leave the field empty. The Empty lastmod → today button fills every empty field with today's date; use it only when the pages really changed today, for instance at launch.
  4. You can leave changefreq and priority empty. Google ignores both fields, and the tool says so itself.
  5. Read the notes below the preview. That's where the tool reports an invalid date, duplicate entries, or paths that can't be resolved without a base URL.
  6. Put the rows in the order you want with the arrow buttons. Then Copy or Download and place the file as sitemap.xml in your web root.

Tip: declare the sitemap in your robots.txt (the Sitemaps field in the robots.txt tab) and submit it in Google Search Console (opens in a new tab) and Bing Webmaster Tools (opens in a new tab) — the note right below the editor reminds you of this too.

Example 4: Check a foreign domain's robots.txt

You want to know how an existing site (your own or a competitor's) steers its crawlers.

  1. Go to the Check robots.txt tab, enter a domain (e.g. example.com — /robots.txt is appended automatically), and click Check.
  2. Read the summary and the checks. They show the HTTP status and every error and warning with its line number — a rule before the first User-agent line, say, or an HTML error page instead of the file. If the server answers with an error (5xx), crawlers treat the whole site as blocked; the tool flags that prominently.
  3. Then the per-bot table. For 40 known bots it shows at a glance whether the home page is Allowed or Blocked. The Group column reveals where the decision comes from: the name of a group dedicated to that bot, * (the wildcard group), or none (no group applies — so allowed); Matched rule names the rule.
  4. Watch the AI rows (type AI): does the site lock out GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot, and co.? That's exactly the information you need for your own GEO strategy as well.
  5. Test a specific URL. In the Test a URL against these rules card, enter a path with a query string (e.g. /blog/my-article?ref=newsletter) and a user-agent (e.g. Googlebot) and click Test — the tool tells you whether the file allows or blocks access, which rule on which line applies, and which group is in effect.
  6. Under Sitemaps declared you find the sitemaps linked in the file. The check button next to each takes you straight to the sitemap check (see the next example).

Example 5: Validate a sitemap and check spec limits

You want to make sure your sitemap.xml is clean and complete.

  1. In the Check Sitemap tab, enter your domain (e.g. your-domain.com) and click Check. The tool then takes the sitemap your robots.txt declares, falling back to /sitemap.xml. Alternatively, enter the full sitemap URL or arrive here directly via the check button from the robots.txt check.
  2. Read the checks. At the top are errors that make crawlers discard the file — above all XML that isn't well-formed (e.g. an unescaped & in a URL). Below that: relative or duplicate addresses, invalid lastmod values, and more than 50,000 URLs — beyond that you have to split into several files plus an index file. The same applies from 50 MB uncompressed; the checker reports that limit only in Expert Mode, though — through the server proxy a file that large is truncated first and reported as incomplete.
  3. Look at the robots.txt card. It reports whether your robots.txt blocks listed URLs for Googlebot (a contradiction you should resolve) and whether the sitemap is declared there.
  4. Read the Metadata table. Its progress bars show the percentage of URLs with a lastmod — the field that counts. changefreq and priority sit next to it, marked "ignored by Google".
  5. Run the sample. In the Do the URLs work? card, click Check N URLs: the tool requests up to 20 evenly spread URLs and flags redirects and broken pages. Neither belongs in a sitemap.
  6. The URLs table shows the first 100 entries. Copy all URLs puts the complete list (beyond 100 as well) on your clipboard — handy for reconciling with your CMS.

Example 6: Break down a sitemap index file

Large sites spread their URLs across several sitemaps bundled by an index file.

  1. In the Check Sitemap tab, enter the URL of the index file (often …/sitemap_index.xml) and click Check. If the index is declared in the robots.txt, the bare domain is enough.
  2. The tool detects automatically that this is a sitemap index file and lists the child sitemaps with their optional lastmod under Child sitemaps.
  3. Drill into individual child sitemaps. Next to each is a Check button that loads that file and analyzes it in detail (checks, metadata, robots.txt cross-check, sample). Compressed child sitemaps (.xml.gz) are decompressed along the way.

That way you work from the index down to the individual URL — without opening the files yourself. To edit an index, read it into the sitemap generator via Load File; the tool then switches to Sitemap index on its own (see Example 8).

Example 7: Test a robots.txt before uploading

You want to be sure a new rule blocks exactly what it should — and nothing more. You check that right in the generator, before the file goes live.

  1. In the Examples tab, load the Standard Website template into the generator. Among other things it blocks /admin/, the internal search via /*?s=, and all PDFs via /*.pdf$.
  2. In the Test a URL against this robots.txt card below the preview, enter a path in Path or URL, say /downloads/price-list.pdf. The result appears instantly: Blocked — rule "Disallow: /*.pdf$", group "*".
  3. Try the edge cases: /downloads/price-list.pdf?v=2 is Allowed, because the $ requires the end of the URL and the query string comes after it. /administrator/ is allowed and /admin/users blocked, because the rule with the trailing slash matches only the directory.
  4. Test individual bots. Set Path or URL back to / and enter GPTBot under User-agent. As long as there's no group for GPTBot, the * group applies to it and the result reads Allowed. Add a GPTBot group with Disallow: / via Add User-agent Block, and the result flips to Blocked — from then on GPTBot ignores every rule in the * group.
  5. Need an exception? Add an Allow rule with /downloads/terms.pdf to the * block, switch User-agent back to Googlebot (the * block no longer applies to GPTBot), and enter /downloads/terms.pdf in Path or URL. The test shows Allowed for exactly that file: the longer rule wins.

Meanwhile, the notes below the preview warn you about typical mistakes, such as a path without a leading slash.

Example 8: Build a sitemap index for a shop

A shop with many products spreads its URLs across several sitemaps — the index bundles them into one address that you declare in your robots.txt.

  1. In the Examples tab, load the Shop: Sitemap Index template into the generator. The Sitemap tab is then set to File type Sitemap index (sitemaps), the list is labeled Sitemaps, and changefreq/priority are hidden.
  2. Enter your shop domain in Base URL and adjust the entries — one per sitemap file, e.g. /sitemap-pages.xml, /sitemap-products-1.xml, /sitemap-products-2.xml. If a sitemap lives on another host, enter it as a full URL; the tool keeps it as is and points it out.
  3. Set each sitemap's lastmod to the date it last changed.
  4. Download saves the file as sitemap_index.xml. Put it in your web root and add it under Sitemaps in the robots.txt tab.
  5. After uploading, check the index in the Check Sitemap tab as in Example 6 and work your way through the child sitemaps.

There's more depth: the overview for the big picture, the manual for every option, and the tips & tricks for strategy and pitfalls. You can try all of it directly in the tool.