Skip to content
XUtilHub

XML Sitemap Generator – Crawl Your Site Free

Generate an XML sitemap by crawling your website (up to 500 pages) or from a list of URLs. Respects robots.txt and noindex. Download sitemap.xml instantly.

  • Free, no sign-up
  • Fetches public pages only, nothing stored

About the XML Sitemap Generator

Enter your website address and the generator follows your internal links to list every indexable page, skipping URLs blocked by robots.txt or marked noindex, then builds a valid sitemap.xml. You can also paste a list of URLs instead.

Key features

  • Crawls your site by following internal links (up to 500 pages)
  • Respects robots.txt and skips noindex pages
  • Uses canonical URLs and strips tracking parameters
  • Optional lastmod, changefreq and priority
  • Or paste a list of URLs instead of crawling
  • Valid, escaped sitemap.xml ready to download

Generate an XML sitemap in one click

An XML sitemap lists the pages you want search engines to find. It's especially helpful for new sites, large sites and pages that aren't well linked. Enter your homepage address and the generator crawls your site like a search engine: it follows internal links, respects robots.txt, skips noindex pages and builds a valid sitemap.xml.

What's included

PageIncluded?
Internal pages returning 200✅
Pages with a canonical to another URLThe canonical URL instead
Pages marked noindex❌
URLs blocked by robots.txt❌
Redirects, errors (4xx/5xx)❌
Images, PDFs and other files❌
Links marked rel="nofollow"Not followed

Example output

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2026-10-21</lastmod>
  </url>
</urlset>

After you generate it

  1. Upload sitemap.xml to the root of your site.
  2. Reference it in robots.txt: Sitemap: https://example.com/sitemap.xml.
  3. Submit it in Google Search Console (Sitemaps report) and Bing Webmaster Tools.
  4. Regenerate when you add or remove pages, or use a CMS plugin that keeps it updated automatically.

Sitemap best practices

  • List only canonical, indexable URLs that return 200.
  • Use absolute URLs with the correct protocol and host.
  • Keep lastmod accurate; don't set it to today for every page.
  • Split very large sites into several sitemaps of under 50,000 URLs each.

Reference your sitemap from robots.txt with the Robots.txt Generator, and check individual pages with the Meta Tag Checker.

How to use the XML Sitemap Generator

  1. Enter the address of the page or website you want to check.
  2. Click the button. Our server fetches the public page (it can't reach private or local addresses).
  3. Review the results, starting with anything marked in red or amber.
  4. Fix the issues on your site, then run the check again to confirm.

Frequently asked questions

Why weren't all my pages found?

The crawler only finds pages linked from other pages, in the HTML the server sends. Pages linked only through JavaScript, forms or search, orphan pages, and pages blocked by robots.txt or marked noindex won't appear. You can add missing URLs using the 'Paste URLs' option.

Should I include changefreq and priority?

Google ignores both, so they're left out by default. An accurate lastmod is the only optional field Google uses, and only when it's reliable.

How many URLs can a sitemap contain?

Up to 50,000 URLs and 50 MB uncompressed. Larger sites split their URLs across several sitemaps listed in a sitemap index file.

Does the crawler slow down my site?

It fetches at most four pages at a time, stops after 500 pages or about 45 seconds, and identifies itself as XUtilHubBot, so its load is similar to a single visitor browsing quickly.

Do these tools store the pages I check?

No. Pages and robots.txt files are fetched on demand to analyze them and are not saved. The robots.txt generator, tester and meta tag generator run entirely in your browser.

Can I check pages on localhost or my intranet?

No. For security, our server only fetches public websites. For a private page, use the 'Paste HTML' option of the meta tag checker.

Do the tools respect robots.txt?

The sitemap crawler follows your robots.txt rules and skips pages marked noindex, just like a search engine would.

Last updated:

Related SEO tools