How to Get Every URL From a Website's Sitemap

Find a site's sitemap, follow sitemap index and .xml.gz files, and export every page URL with its lastmod date, by hand or in bulk.

A sitemap is the list of pages a site wants search engines to know about. Most sites publish one, and reading it is usually the quickest way to get a full page list for an SEO audit, a content inventory, a migration check, or a crawler you are building. It is also much lighter on the site than crawling every link.

Here is how to do it by hand, with a short script, and with a tool when you need many sites at once.

Step 1: Find the sitemap

Open https://example.com/robots.txt in a browser. Look for lines that start with Sitemap:. A site can list several.
If robots.txt lists none, try https://example.com/sitemap.xml, then https://example.com/sitemap_index.xml.
If neither exists, the site may have no sitemap at all. Some CMS plugins use other paths, so a search such as site:example.com sitemap can help, but there is no guarantee one exists.

Step 2: Work out what kind of file it is

Open the sitemap URL and look at the first lines:

A <urlset> file is a list of pages. Each page is in a <loc> tag, often with a <lastmod> date.
A <sitemapindex> file is a list of other sitemaps. Big sites split their pages across many files, and an index can point to further indexes.
A file ending in .gz is gzip compressed. Your browser will download it rather than show it.
Some sites use a plain text sitemap with one URL per line.

The sitemap protocol limits one file to 50,000 URLs, so a large site almost always uses an index.

Step 3: Pull the URLs out

For a single small <urlset> file, a one-line command works on macOS, Linux, or Git Bash on Windows:

curl -s https://example.com/sitemap.xml | grep -o '<loc>[^<]*</loc>' | sed 's/<[^>]*>//g' > urls.txt

For a gzipped file, add a decompression step:

curl -s https://example.com/sitemap-0.xml.gz | gunzip | grep -o '<loc>[^<]*</loc>' | sed 's/<[^>]*>//g'

For an index, run the same command on the index first. The <loc> values you get are sitemap files, so run the command again on each one and join the results. In Python, the getsitemap library on PyPI does this recursion for you, starting from robots.txt and the usual paths.

Step 4: Clean the list

Remove duplicates. The same page often appears in more than one sitemap.
Filter to the section you care about, such as URLs containing /blog/.
If you only want recent changes, keep rows whose lastmod is after your cutoff. Treat lastmod with care: some sites stamp every URL with the time of their last build, so it is not always a true edit date.

Where it gets slow

The manual method is fine for one site with a few files. It gets tedious when an index points to dozens of gzipped files, when you have a list of domains, or when you need the source file and lastmod for every URL in one table. Be polite as well: if you script this, pause between requests and respect robots.txt.

Sitemap URL Extractor, an Apify actor, does these steps for a list of domains. It reads robots.txt, falls back to /sitemap.xml and /sitemap_index.xml, follows index files up to 5 levels deep, detects gzip from the file content, and returns one row per unique URL with lastmod, changefreq, priority and the sitemap file it came from. You can filter with a regular expression (urlPattern), keep only URLs changed since a date or age such as 30 days (modifiedSince), and cap each run with maxItems and maxItemsPerDomain. It costs $0.0005 per returned URL, so the default cap of 1,000 URLs is about $0.50.

Its limits, so you know what it will not do: it respects robots.txt and skips a site whose robots.txt cannot be reached, it does not crawl pages when a site has no sitemap, it does not return image, video, news or hreflang entries, and it does not read RSS or Atom feeds used as sitemaps.

FAQ

Does every website have a sitemap?

No. Many do, because CMS platforms and SEO plugins create one, but it is optional. If there is none, the only way to list pages is to crawl links from the home page.

Is a sitemap a complete list of pages?

Not always. It lists the pages the site chose to include. Old, private or badly configured pages may be missing, and some listed URLs may redirect or return errors.

How do I get the page content, not just the URLs?

Fetch each URL. If you want clean text for a language model or search index, pass the list to URL to Markdown Converter, and see the guide on converting a web page to Markdown.

Can I use this on a competitor's site?

A public sitemap is published for crawlers to read. Respect the site's robots.txt and terms, and keep request rates low.

This guide was written with AI (Claude) and checked against the product's documentation.

All guides All products