robots.txt & Sitemap — Manual

Back to the overview: robots.txt & Sitemap · Open the tool: www.jpkc.com/tools/robots-sitemap/

This manual describes robots.txt & Sitemap in full: each of the seven tabs, every option of the two generators, the behavior of the two live checkers, and the technical limits underneath. The tool's interface is in English, so the tab and button names appear here exactly as you'll see them.

Layout: seven tabs, four functions

The tool is split into seven tabs: robots.txt (generator), Sitemap (generator), Check robots.txt (live checker), Check Sitemap (live checker), Examples, Tips, and Reference. The two generators produce files; the two checkers fetch and analyze existing files from a foreign URL. Examples, Tips, and Reference are reference tabs that work without any input. On narrow screens, form and output stack vertically.

robots.txt generator

The first tab builds a robots.txt with a live preview: every change in the form on the left (section Configure robots.txt) updates the output editor on the right after a short delay.

User-agent blocks

The core, found in the form under Groups. Each block corresponds to one group in the robots.txt. Via Add User-agent Block you add as many as you need; a new block starts with an empty Disallow: line. Each block has:

• a User-agent field with autocomplete (a datalist). Enter several bots comma-separated (GPTBot, ClaudeBot) — each gets its own User-agent: line in the output, and the rules below apply to all of them. The suggestions are the 40 bot names from the tool's bot list — search engines (Googlebot, Bingbot, DuckDuckBot, Applebot, YandexBot …), archives (archive.org_bot), social bots (facebookexternalhit, Twitterbot, LinkedInBot), AI crawlers for training (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, CCBot …), for AI search and user-triggered fetches (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot …), and SEO-tool crawlers (AhrefsBot, SemrushBot, MJ12bot …). You can type any other name too; * stands for "every bot without a group of its own".
• rule rows. Add Rule adds one, each with a type select (Disallow or Allow) and a path field (e.g. /admin/). Paths are prefixes and may contain * (any characters) and a trailing $ (end of the URL), for instance /*.pdf$ or /*?. An empty value produces Disallow: — "nothing blocked", so everything is allowed.
• a Crawl-delay field (seconds, 0 to 3600, optional). It produces a Crawl-delay: line in the block; the notes list below the preview adds that Google ignores the directive while Bing and Yandex honor it.

You move blocks and rule rows with the arrow buttons (Move up/Move down) and remove them with the trash or X button. Blocks can also be reordered with the mouse by dragging the grip handle.

The generator handles two special cases on its own: a block without any rule row automatically gets Allow: / — without a rule, RFC 9309 would merge it with the next block and it would inherit that block's rules; a note below the preview tells you so. A block without a user-agent but with rules is left out and flagged as well.

Sitemaps

Below the blocks sits the Sitemaps field: one absolute URL per line. Each line is written at the end of the file as Sitemap: URL; these lines apply to all crawlers, independent of the groups. The generator no longer offers a Host: directive — only Yandex ever read it, and it has had no effect since 2018.

Output, copy, download, import

On the right is the output editor (read-only). Every generated file starts with the comment line # robots.txt — generated with https://www.jpkc.com/tools/robots-sitemap/. Above the editor:

• Load File — reads an existing robots.txt (up to 20 MB) and fills the form from it. Each group in the file becomes one block; several user-agents of a group appear comma-separated in the same block, and sitemaps land in the Sitemaps field. A report above the form names what gets lost or stands out: comments are not kept, an obsolete Host line is dropped, and the warnings and errors from the lint (see below) are listed. If the file contains neither groups nor sitemaps, the tool says so and leaves the form untouched.
• Reset — after a confirmation, resets the form to the default: one User-agent: * block with an empty Disallow: rule.
• Copy — copies the robots.txt to the clipboard.
• Download — saves it as robots.txt (text/plain).

Below the editor, a note reminds you that the file belongs in the root of every host and must be served as text/plain — subdomains need their own file.

The working state is saved automatically in the browser (LocalStorage) and restored on your next visit; form states from older versions of the tool are carried over. On the very first load you see the default User-agent: * block.

Notes below the preview

Right below the editor, a list shows everything that stands out in the file you're building, sorted into errors, warnings, and info. The notes come from the same checks as in the Check robots.txt tab (listed there under What you see in the result), plus the form's own notes — about the automatically added Allow: /, for example. With nothing to report, the list stays hidden.

URL test in the generator

The Test a URL against this robots.txt card tests live against the file you're building: in Path or URL you enter a path or a full URL (default /), in User-agent a bot (default Googlebot, with autocomplete). The result updates with every keystroke and every change to the form: Allowed or Blocked, plus the matching rule and the group that applies to the bot — or a note that no rule matches or no group applies. How the tool decides is explained under How the tool reads a robots.txt.

Sitemap generator

The second tab builds an XML sitemap or a sitemap index, also with a live preview (section Configure sitemap.xml).

File type: sitemap or sitemap index

The File type select determines what you get:

• Sitemap (pages) — a regular sitemap with <urlset>, one entry per page.
• Sitemap index (sitemaps) — an index with <sitemapindex>, one entry per sitemap file. The list is then labeled Sitemaps, the changefreq and priority fields disappear (an index entry knows only <loc> and <lastmod>), and Download saves the file as sitemap_index.xml.

Base URL and URL rows

• Base URL — the address relative paths are joined with (e.g. https://example.com). Full URLs in the rows stay as they are. Without a base URL a relative path can't be resolved: the entry is missing from the output, and an error note names it.
• URLs — Add entry adds rows. Each row has:
  • Path or URL — a path relative to the base URL (e.g. /about/; a missing leading / is added) or a full URL. The tool percent-encodes the address (non-ASCII characters, spaces) and escapes it for XML — an & in the query string becomes &.
  • Last modified — a text field for <lastmod> in W3C Datetime format: the date alone (2026-10-15) or with time and time zone (2026-10-15T09:30:00+02:00).
  • changefreq — a select for <changefreq>: always, hourly, daily, weekly, monthly, yearly, never (or empty).
  • priority — a select for <priority>: 1.0 down to 0.0 in tenths (or empty).

Optional fields are written only when set; both selects default to empty. The form itself points out that Google ignores changefreq and priority and that an accurate lastmod is what counts. You move rows with the arrow buttons and remove them with the X button.

The Empty lastmod → today button fills today's date into every row without a lastmod; rows that already have a date stay unchanged.

Adding many entries at once

In the Add many at once field you paste a list — one path or URL per line — and turn it into entries with Add lines. If the form holds nothing but a single empty starting row, the list replaces that row. Blank lines are skipped.

Output and import

The editor on the right shows the finished file with the XML declaration and the namespace http://www.sitemaps.org/schemas/sitemap/0.9. Below it the tool lists the validation findings as you go — a lastmod that isn't a valid W3C date, duplicate addresses, entries on a host other than the base URL, or more than 50,000 entries (then: split up and use an index). A note under the editor reminds you to declare the sitemap in your robots.txt and submit it in Search Console and Bing Webmaster Tools. Buttons:

• Load File — reads an existing file (.xml or .txt, up to 20 MB). The tool understands regular sitemaps, sitemap indexes (it then switches to Sitemap index), text sitemaps with one URL per line, and RSS/Atom feeds, whose links it imports as a regular sitemap. The base URL is derived from the first entry; entries on that host become paths (query string and fragment are kept), entries on other hosts keep their full URL. lastmod is imported as text, so a time of day is not lost. If the XML isn't well-formed, the tool rescues the addresses and reports that the other fields stay empty; further findings appear in the report above the form.
• Reset — after a confirmation, resets the form to a single row / without a base URL.
• Copy / Download — copies or saves the file as sitemap.xml (sitemap_index.xml in index mode, type application/xml).

On first load, three example rows (/, /about/, /contact/) are pre-filled. The working state is stored locally here too.

Live checker: Check robots.txt

This tab (Check a live robots.txt) fetches an existing robots.txt from a foreign domain and dissects it. You either enter just a domain (e.g. example.com; /robots.txt is appended automatically) or a full URL. If the protocol is missing, https:// is prepended. Clicking Check starts the fetch.

What you see in the result

• A summary with HTTP status, file size, number of groups, and the count of errors (or, if there are none, of warnings) — or "No problems found.". An empty file is classified as "Empty file — everything is allowed.".
• Checks — the findings, sorted into error, warning, and info, with line numbers. About the server's response: a redirect (crawlers follow up to five), an HTML page instead of a text file (usually the file doesn't exist and the server returns an error or home page), a content type other than text/plain, gzip compression, invalid UTF-8, a file truncated by the fetch limit. About the content: rules before the first User-agent line (which no crawler applies), lines without field: value, directives without a colon or with a typo (Google tolerates Dissallow, for instance; other crawlers don't), empty user-agents, user-agent values with a version (Googlebot/2.1 only counts as googlebot), blank lines between User-agent lines, paths without a leading / or *, relative sitemap URLs, a non-numeric Crawl-delay (every Crawl-delay also gets the info that Google ignores it while Bing and Yandex honor it), unsupported directives such as Host, Noindex, Nofollow, Clean-param, Request-rate, or Visit-time, unknown directives, several groups for the same bot (crawlers merge them), a file without any group, User-agent: * with Disallow: /, and files over 500 KiB (Google reads only the first 500 KiB). A byte order mark at the start of the file is noted as info.
• Per-bot access to / — a table that checks for 40 known bots whether they may fetch the home page. Per bot: name, operator (Owner), Type (Search, Social, Archive, AI, SEO Tool), Access (Allowed/Blocked), Group (the bot name of the group that applies, * for the wildcard group, or none), Crawl-delay, and the rule that actually matches (Matched rule). This is the fastest way to see whether you're accidentally locking out AI crawlers.
• Groups — each group in the file individually, with its user-agents, its line number, a crawl-delay badge, and its allow/disallow rules. A group without a restricting rule is marked "No restricting rule — full access.".
• Sitemaps declared — every Sitemap: line in the file. Next to each absolute URL is a check button that jumps straight to the Check Sitemap tab and checks that sitemap; relative entries are marked "not absolute". If there's no such line at all, the tool recommends adding one.
• Test a URL against these rules — a small form: enter a path or URL under Path or URL (default /) and a bot under User-agent (default Googlebot), then click Test. You get back Allowed or Blocked, the tested path in the form the tool compares it in (percent-encoded), the matching rule with its line number, the group, and any crawl-delay. The User-agent field also accepts a complete user-agent string; the tool extracts the product names from it.
• Raw content — the raw file contents, collapsible and copyable via Copy (the display stops after 50,000 characters, Copy delivers everything).

Edge cases

The HTTP status decides how crawlers treat the file — and that's how the tool classifies it:

• 2xx — the file is evaluated as described above.
• 4xx (e.g. 404) — there is no robots.txt; crawlers may fetch everything. The tool shows a warning instead of an evaluation.
• 5xx or 429 — the server answers with an error; crawlers must treat the whole site as blocked until the file is reachable again. The tool shows this as an error.
• Any other status is reported as "Unexpected status".

If the file can't be fetched at all (DNS error, timeout, blocked address), an error message gives the reason.

Live checker: Check Sitemap

The same flow for sitemaps (Check a live sitemap.xml): enter a domain or full URL, click Check.

Which file gets checked

• If you enter just a domain (example.com or https://example.com/), the tool first fetches the robots.txt and checks the first sitemap declared there. If several are declared, a note offers the others as buttons so you can check them one by one. If none is declared, the tool takes /sitemap.xml.
• If you enter a full URL, exactly that file is checked.

The tool understands regular sitemaps (<urlset>), sitemap indexes (<sitemapindex>), text sitemaps with one URL per line, and RSS/Atom feeds, which Google also accepts as sitemaps. Gzip-compressed files (.xml.gz) are decompressed before the check.

Regular sitemap (urlset)

For regular sitemaps, text sitemaps, and feeds the tool shows:

• A summary with the file type (sitemap, text sitemap, feed …), the number of URLs, the file size, and the count of errors (or, if there are none, of warnings) — or "No problems found.".
• Checks — findings about the response (redirect: submit the final URL; HTML page instead of a sitemap; gzip; invalid UTF-8; truncated file) and about the content: XML that isn't well-formed (crawlers then discard the whole file), a missing or wrong sitemap namespace, <url> without or with several <loc>, mixed <url> and <sitemap> entries, relative addresses, whitespace or non-ASCII characters in addresses, addresses over 2,048 characters, addresses on a different host or protocol than the sitemap, duplicates, a lastmod that isn't a valid W3C date or lies in the future, invalid changefreq and priority values, more than 50,000 URLs, or more than 50 MB uncompressed. The 50 MB check only applies in Expert Mode: through the server proxy a file that large is already truncated when fetched (see fetch limits), and the checker then reports that the results are incomplete. As info: Google ignores changefreq and priority; a sitemap without any lastmod; text sitemap or feed detected.
• Metadata — a table with the number of URLs and progress bars showing the percentage of URLs with lastmod, changefreq, and priority set (the latter two marked "ignored by Google"). If the sitemap uses extensions, an Extensions row counts images, videos, news entries, and hreflang alternates.
• Do the URLs work? — the sample, see below.
• URLs — a table of the entries (the first 100 of N) with URL and lastmod. Copy all URLs copies the complete list, not just the 100 shown.
• Raw content — the raw contents, collapsible and copyable (the display stops after 50,000 characters, Copy delivers everything).

Sitemap index

If the checker detects an index file (<sitemapindex>), it lists the child sitemaps under Child sitemaps instead, with their optional lastmod. Each child sitemap has a Check button to examine it individually. Here too the checks report invalid addresses and dates, <url> entries inside the index, and more than 50,000 listed sitemaps.

Cross-check against the robots.txt

As soon as the sitemap has entries, the tool fetches the robots.txt of the same host and compares the two files. The robots.txt card reports:

• how many listed URLs the robots.txt blocks for Googlebot, with up to three examples — a sitemap should contain only fetchable pages;
• whether the robots.txt blocks the sitemap itself;
• whether the sitemap is declared in the robots.txt, and if not, the matching Sitemap: line to add;
• that the whole site counts as blocked if the robots.txt answers with a server error — or that nothing is blocked when there's no robots.txt.

Sample: do the URLs work?

The Do the URLs work? card checks on demand (Check N URLs) whether the listed pages are actually reachable. With up to 20 URLs it checks every one; with more, a sample of 20 spread evenly across the list (the button then reads Check 20 URLs (sample)). Each URL is requested via HEAD; if a server rejects HEAD, the server-side proxy follows up with a small GET. For each URL you see the status and a verdict: OK, a redirect ("list the final URL" — the sitemap should contain the target), Broken from status 400 up ("remove it from the sitemap"), or, if no answer comes back, the fetch error. Only absolute http(s) addresses are checked. A closing line sums up how many URLs are fine, redirected, or broken.

Edge cases

If the server doesn't answer with 2xx, the tool reports "No sitemap at …" with the status (and, if you entered only a domain and the robots.txt names no sitemap, it says that too). An empty file, an HTML page, or an unknown root element (neither <urlset> nor <sitemapindex>) is reported as an error; a well-formed file without a single entry appears in the summary as "file without entries". If the file was cut off by the proxy's size limit, a note says the results are incomplete.

How the tool reads a robots.txt

The generator test, the per-bot table, the URL test, and the sitemap cross-check all use the same evaluation per RFC 9309, as Google reads it:

• Groups: consecutive User-agent lines form one group and share the rules below them. Only a User-agent line after a rule starts a new group — a blank line does not end a group. Several groups for the same bot are merged.
• Which group applies: a bot follows the group carrying its name and then ignores User-agent: * entirely; only without a group of its own does * apply. The product name is compared case-insensitively and without a version (Googlebot/2.1 → googlebot) — and exactly: google does not match Googlebot. Without a group of their own, Googlebot-Image, Googlebot-News, and Googlebot-Video fall back to the Googlebot group.
• Which rule applies: the comparison covers path plus query string. Rules are prefixes (/admin also matches /administrator), * stands for any characters, a trailing $ for the end of the URL. The longest matching rule wins; on equal length, Allow wins. Paths and rules are compared in a uniform percent-encoding, so /ä and /%C3%A4 count as equal.
• /robots.txt itself is always allowed.

Examples, Tips, Reference

• Examples — ready-made templates with a preview. For the robots.txt: Allow All Bots, Standard Website (admin, login, and private areas, internal search, and PDFs blocked via wildcard), WordPress, Block AI Training, Allow AI Search, and Block All AI Crawlers. For the sitemap: Simple Website (four pages with a real lastmod), Blog (dates with time and time zone, one URL with a query string), and Shop: Sitemap Index. The Load into Generator button replaces the respective generator with the template after a confirmation and switches to its tab.
• Tips — compact best-practice cards ("Do This" / "Avoid This") for both files plus an explanation of how robots.txt and the sitemap work together.
• Reference — the format specification: a table of robots.txt directives with an example and support notes, the bot list with operator and type, a table of sitemap XML elements, and links to RFC 9309, Google's guides for robots.txt and sitemaps, the Sitemaps protocol, and Search Console.

robots.txt directives (Reference)

Directive	Support	Meaning
User-agent	all	Starts a group (product name, case-insensitive). Consecutive lines share one group; groups for the same bot are merged.
Disallow	all	Blocks paths starting with the value (path + query). * = any characters, $ at the end = end of the URL. An empty value blocks nothing.
Allow	all	Allows a path. The longest matching rule wins; on equal length, Allow.
Crawl-delay	most	Seconds between requests. Not supported by Googlebot (use Search Console there).
Sitemap	all	Absolute URL of a sitemap. May appear several times, independent of the groups.
Host	obsolete	Was only read by Yandex and has been ignored since 2018 — redirect to the preferred host instead.
Noindex	none	Never standard; Google stopped honoring it in 2019 — use meta robots or X-Robots-Tag instead.

Bot list (Reference)

The Common Bot Names table lists the 40 bots the tool also uses for autocomplete and the per-bot table — with operator or purpose and type. For AI crawlers the type column shows the purpose instead of "AI": AI training (e.g. GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, CCBot, Bytespider, Amazonbot), AI search (OAI-SearchBot, Claude-SearchBot, PerplexityBot, DuckAssistBot), or AI user request (ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User — fetches a person triggers in a chat).

Sitemap XML elements (Reference)

Element	Required	Values / note
<urlset>	yes	Root element with namespace http://www.sitemaps.org/schemas/sitemap/0.9.
<url>	yes	Container per entry.
<loc>	yes	Fully qualified URL including protocol, max 2048 characters, URL-encoded.
<lastmod>	optional	Last significant change in W3C Datetime format (2026-10-15 or 2026-10-15T09:30:00+02:00). Google uses it when it is consistently accurate.
<changefreq>	optional	always/hourly/daily/weekly/monthly/yearly/never — ignored by Google.
<priority>	optional	0.0–1.0 — ignored by Google.
<sitemapindex> / <sitemap>	index only	Lists up to 50,000 sitemaps (each with <loc>, optional <lastmod>). An index must not list other indexes.
Limits	—	50,000 URLs and 50 MB per file, uncompressed. Text sitemaps (one URL per line) and RSS/Atom feeds are accepted too.

Architecture, limits, and privacy

• Generators and evaluation entirely client-side. The two generator tabs produce their files fully in the browser, and the parser, checks, and URL tests run there too. Nothing is uploaded; the working state lives only in your LocalStorage. The tool keeps no history of your checks.
• Checkers via a server-side proxy. A browser can't load a foreign file directly because of CORS. So a server-side proxy on the JPKCom server fetches the file via cURL; the analysis then runs locally in the browser again. The checked domain sees a request from the JPKCom server (with a browser user agent), not your IP address.
• No public API. The two server-side endpoints (a fetch proxy and a token-based helper endpoint) are not a publicly callable API — they're used exclusively by the tool's JavaScript and hardened against abuse (token authentication, referer checks).
• SSRF protection: internal, local, and private IP addresses are blocked, and every redirect hop is re-validated.
• Fetch limits: at most 5 MB of body (larger files are truncated — a note appears), gzip files are decompressed to at most 10 MB, a 15 s timeout per request, 30 s for all redirects combined, at most 5 redirects. The sample requests each URL with a 10 s timeout.
• Rate limit: in standard proxy mode at least 3 seconds must pass between two checks; click sooner and a note tells you how long to wait. Server-side there's a limit of 120 fetches per minute per IP address — beyond that you get "Too many requests — please wait a minute".
• Expert Mode (optional). A toggle in the header enables a local proxy (http://127.0.0.1:<port>) that fetches the files directly — without the wait and without the server proxy's size cap; .xml.gz files are then decompressed by the browser. Setup is advanced and not needed for normal use.

For the intro, the audiences, and the big picture, see the overview. Concrete walkthroughs are in the examples. You can try all of it directly in the tool.