robots.txt & Sitemap

Build and check crawler control — in one tool

Two files decide how search engines and AI systems see your website: the robots.txt (what may a crawler fetch?) and the sitemap.xml (which pages should be discovered?). robots.txt & Sitemap helps you with both — and goes a step beyond plain generators: it builds the files and also checks existing files live from any domain.

The tool bundles four functions across seven tabs: a robots.txt generator, a sitemap generator, and two live checkers that fetch and analyze a robots.txt or a sitemap.xml from any URL. On top of that come tabs with ready-made examples, best practices, and a format reference. The foundation is the robots standard RFC 9309 (opens in a new tab), read the way Google interprets it: the tool evaluates wildcards, groups, and precedence exactly as a crawler would. No account, no installation.

It's built for everyone working on a site's technical discoverability: developers who need a correct robots.txt or sitemap fast; SEO and content people who want to verify that search engines and AI crawlers see the right pages; and agencies that want to take apart an existing configuration on someone else's domain in seconds.

The four functions at a glance

robots.txt generator

A form that assembles a complete robots.txt with a live preview. You add as many user-agent blocks as you need — one block can serve several bots, comma-separated (GPTBot, ClaudeBot) —, give each block Allow and Disallow rules with wildcards (*, a trailing $), optionally set a Crawl-delay, and list as many Sitemap: lines as you like. An autocomplete field suggests 40 common bot names — from Googlebot and Bingbot to AI crawlers such as GPTBot, ClaudeBot, OAI-SearchBot, or PerplexityBot. Below the preview the tool lints the file as you type, and a live test tells you whether a given path is allowed or blocked for a given bot — before the file ever goes online. Copy the result, download it, or read an existing robots.txt via Load File to edit it.

Sitemap generator

Here you build an XML sitemap per the Sitemaps protocol (opens in a new tab) — or, with one switch, a sitemap index that bundles several sitemaps. You set a base URL and add entries, one at a time or as a list with one path per line, each with an optional lastmod (W3C date, with or without time) plus changefreq and priority. The tool encodes and escapes the URLs correctly and validates the output continuously. An existing sitemap, sitemap index, text sitemap, or RSS/Atom feed comes in via Load File — without hosts, query strings, or times getting mangled.

Live checker: robots.txt

Enter a domain (e.g. example.com) or a full URL, and the tool fetches the robots.txt and dissects it. First it classifies the HTTP status (a 404 allows everything, a server error blocks everything), then it lists syntax errors and pitfalls with line numbers, and a per-bot table shows for 40 known bots whether they may fetch the home page and which group and rule apply. Add to that every group in the file, the declared sitemaps, and an interactive test for any path and bot.

Live checker: sitemap

The same flow for the sitemap: give it just a domain and the tool takes the sitemaps declared in the robots.txt, falling back to /sitemap.xml. It checks whether the XML is well-formed, validates every address and every lastmod, flags more than 50,000 URLs per file as an error (the 50 MB limit is checked in Expert Mode), and also understands sitemap indexes, text sitemaps, RSS/Atom feeds, and gzip-compressed .xml.gz files. A cross-check against the robots.txt finds URLs the sitemap offers but Googlebot may not fetch, and a sample of up to 20 URLs shows whether the listed pages really answer with 200.

Generators local, checkers via a proxy

Important for understanding and for privacy: the two generators run entirely in your browser — nothing leaves your machine, and the working state is stored only locally. The evaluation (parser, linting, URL tests) runs in the browser too. The two checkers, by contrast, have to fetch a foreign file, which a browser can't do directly because of CORS. So a server-side proxy on the JPKCom server fetches the file; the analysis then runs locally again. The checked domain therefore sees a request from the JPKCom server, not your IP address. Internal and private addresses are blocked server-side. How the proxy works and which limits apply (size, timeout, rate limit) is covered in the manual.

Try it now

→ Open robots.txt & Sitemap — build a file or check a foreign domain, right in the browser, no account. In the Examples tab, ready-made templates (standard website, WordPress, block AI training, block all AI crawlers, blog, shop with a sitemap index) load into the generators as a starting point.

SEO & GEO Analyzer — inspects a whole URL including its own Robots Analysis tab and uses the same robots.txt and sitemap parser, bot list included; here you build the rules, there you see them in the context of the entire page.
Meta Tags Generator — clean titles, descriptions, and Open Graph/Twitter data for the pages your sitemap lists.
llms.txt Generator — the AI counterpart to the sitemap: a structured content overview specifically for LLMs.

There's more on the subpages: the manual with every function, option, and limit in detail, hands-on examples, and a collection of tips & tricks.