Sitemap URL Extractor
Enter a domain, get every page URL from its sitemaps with lastmod dates. Follows nested and gzipped sitemaps. Pay per URL.
Get it on Apify, $0.0005 per URLGive it one or more domains and get back one row per page URL listed in the site's sitemaps, with the lastmod date the site reports. It reads robots.txt for Sitemap lines, tries /sitemap.xml and /sitemap_index.xml if there are none, follows sitemap index files down to the nested ones, and decompresses gzipped files. You can also paste a direct sitemap URL.
Optional filters: a URL pattern (regular expression, for example /blog/) and a "modified since" setting that takes a date or an age like 30 days. Per-domain and total caps keep one large site from using up a run.
Each result has the URL, lastmod, changefreq, priority, the sitemap file it came from, and a fetch time. Skipped files and a per-site summary are saved separately and are not billed.
Use it for SEO audits, change monitoring, and seeding a crawler.
Limits: a single sitemap file can list up to 50,000 URLs, and the default maxItems of 1000 caps a run at about $0.50. Set maxItems lower for a quick look. It respects robots.txt, does not crawl links, and ignores image, video, and news extensions.
How-to guide: How to Get Every URL From a Website's Sitemap
Get it on ApifyRelated
URL to Markdown Converter
Turn public web pages into clean Markdown with title, links, and heading sections. You pay only for pages that convert.
PDF to Text and JSON
Give it PDF links, get the text of every page plus title, author, dates and page count as JSON. No OCR. $0.005 per PDF.