robots.txt & Sitemap

Build and check crawler control — in one tool

Two files decide how search engines and AI systems see your website: the robots.txt (what may a crawler fetch?) and the sitemap.xml (which pages should be discovered?). robots.txt & Sitemap helps you with both — and goes a step beyond plain generators: it builds the files and also checks existing files live from any domain.

The tool bundles four functions across seven tabs: a robots.txt generator, a sitemap generator, and two live checkers that fetch and analyze a robots.txt or a sitemap.xml from any URL. Add to that tabs with ready-made examples and a format reference. Everything runs in the browser — no account, no installation.

It's built for everyone working on a site's technical discoverability: developers who need a correct robots.txt or sitemap fast; SEO and content people who want to verify that search engines and AI crawlers see the right pages; and agencies that want to take apart an existing configuration on someone else's domain in seconds.

The four functions at a glance

robots.txt generator

A form that assembles a complete robots.txt with live preview. You add as many user-agent blocks as you like, give each block Allow and Disallow rules, optionally set a Crawl-delay, and finish with a Sitemap: line plus an optional Host: directive (Yandex). Blocks can be reordered by drag and drop. An autocomplete field suggests 40-plus common bot names — from Googlebot and Bingbot to AI crawlers like GPTBot, ClaudeBot, or PerplexityBot. Copy the result, download it as a file, or read an existing robots.txt via Load File to edit it.

Sitemap generator

Here you build an XML sitemap per the Sitemaps protocol. You set a base URL and add URL rows, each with a path and optional lastmod (date), changefreq (always to never), and priority (0.0–1.0). Reorder by drag and drop, copy or download the result — and import an existing sitemap.xml.

Live checker: robots.txt

Enter a domain (e.g. example.com) or a full URL, and the tool fetches the robots.txt and dissects it: a Per-Bot Access table shows, for 40-plus known bots, whether each is allowed or blocked and which rule applies; plus all user-agent blocks, the declared sitemaps, and an interactive test of whether a given path is reachable for a given bot.

Live checker: sitemap

The same flow for the sitemap.xml: the tool counts the URLs, checks coverage of lastmod, changefreq, and priority, warns when the spec limits are exceeded (50,000 URLs / 50 MB per file), and detects sitemap index files whose child sitemaps you can drill into directly.

Generators local, checkers via a proxy

Important for understanding and for privacy: the two generators run entirely in your browser — nothing leaves your machine, the working state is only stored locally. The two checkers, by contrast, have to fetch a foreign file, which a browser can't do directly because of CORS. So a server-side proxy on the JPKCom server fetches the file; the analysis then runs locally in your browser again. The checked domain therefore sees a request from the JPKCom server, not your IP address. Internal and private addresses are blocked server-side. How the proxy works and what limits apply (size, timeout, rate limit) is covered in the manual.

Try it now

→ Open robots.txt & Sitemap — build a file or check a foreign domain, right in the browser, no account. In the Examples tab, ready-made templates (standard website, WordPress, block AI crawlers, blog, shop) load into the generators as a starting point.

SEO & GEO Analyzer — inspects a whole URL including its own Robots Analysis tab; here you build the rules, there you see them in the context of the entire page.
Meta Tags Generator — clean titles, descriptions, and Open Graph/Twitter data for the pages your sitemap lists.
llms.txt Generator — the AI counterpart to the sitemap: a structured content overview specifically for LLMs.

There's more on the subpages: the manual with every function, option, and limit in detail, hands-on examples, and a collection of tips & tricks.