Sitemap Extractor: URLs from sitemap.xml (nested, gzip)
Sitemap URL inventory with an optional status check per URL (404, redirect, noindex), from sitemaps that never fail on a bad file: nested indexes, .gz, .txt, lastmod, priority, images, hreflang. Depth limit, new-since-last-run monitor. Partial results, not errors.
Who it is for
SEO specialists, developers, and technical marketers auditing sites, domains and software ecosystems.
Pricing
| Event | Price | Per 1,000 |
|---|---|---|
| Sitemap fileCharged once per sitemap file (xml, .gz or txt) that was fetched and from which at least one URL was delivered. Sitemap index files, missing files, HTML pages and files with no URLs are never charged. | - | - |
| LinkCharged once per URL that the optional check mode actually requested (HEAD, then GET fallback, including redirects and timeouts). URLs skipped because robots.txt disallows them or the host is non-public are never charged. Off by default. | - | - |
| Actor StartCharged when the Actor starts running. Number of events charged depends on Actor memory (one event per GB, minimum one event). | $0.00005 | - |
Prices are read live from the Apify listing at build time. Apify platform usage is included in event prices.
Sample output
| recordType | site | url | changefreq | priority |
|---|---|---|---|---|
| url | crawlee.dev | https://crawlee.dev/blog | weekly | 0.5 |
| url | crawlee.dev | https://crawlee.dev/blog/archive | weekly | 0.5 |
| site-summary | crawlee.dev |
First rows of a real run, selected columns. Emails, phone numbers and personal names are masked. Full output has more fields.
Input
startUrlsREQUIRED · Domains or sitemap URLs: One per row. A bare domain or site URL (https:// is added if missing) triggers discovery: robots.txt Sitemap lines first, then common paths such as /sitemap.xml. A URL that looks like a sitemap (.xml, .xml.gz, .txt, or 'maxUrlsPerSite· Max URLs per site: Hard cap on unique URLs returned per site. Reading stops (and the download is aborted) once it is reached, so a 2-million-URL sitemap does not stall the run. The site summary says when the cap truncated results.checkMode· Link check mode: off: inventory only. sitemapUrls: request the sitemap URLs (HEAD, GET fallback) and flag 404, 5xx, unreachable, redirects and noindex. pageLinks: request the internal links found on the pages listed below (or the home pacheckPageStatus· Check page status (HTTP 200/404/redirect): Simple on/off switch: request every sitemap URL (evenly spaced sample, capped by Max links to check) and add statusCode, redirects and flags such as not-found, server-error and noindex to each row. Same as Link check modmaxSitemapFiles· Max sitemap files per site: Cap on sitemap files fetched per site, counting index files. Nested indexes are followed up to 6 levels deep.maxCrawlingDepth· Max crawling depth (nested sitemap levels): How many levels of nested sitemap indexes to follow below the first sitemap file. 0 reads only the sitemaps listed in robots.txt or given by you; 6 (default and maximum) follows the whole tree. Lower it to skip deep archincludeDuplicates· Emit duplicate URL rows: Duplicates are always counted and listed in the site summary. Turn this on to also emit a row (isDuplicate = true) for every repeat occurrence.pageUrls· Pages to scan (pageLinks mode): Pages whose internal links are checked when Link check mode is 'pageLinks'. Leave empty to use the home page of each start URL. The page fetch itself is not billed; each link checked is.maxLinksToCheck· Max links to check: sitemapUrls mode: per site, an evenly spaced sample across the whole inventory (not just the first rows). pageLinks mode: per page. This is the cost ceiling for check mode.checkIndexability· Read noindex signals: Uses a GET for each checked URL (first 256 KB) to read meta robots, plus the X-Robots-Tag header, so noindex pages in the sitemap are flagged. Turn off for HEAD-only, lighter checks.
Use it from an AI agent (MCP)
Add this server URL to Claude, Cursor, VS Code or any MCP client. Authenticate with your Apify account.
https://mcp.apify.com/?tools=transparent_meteorite/sitemap-link-inventory
FAQ
How do I run Sitemap Extractor: URLs from sitemap.xml (nested, gzip)?
Open it on Apify, fill in the input form, and press Start. You can also call it from the Apify API, a schedule, or an AI agent via MCP. Results land in a dataset you can download as JSON, CSV or Excel.
How is it billed?
Pay-per-event on Apify: you are charged only for the events listed in the price table above, plus nothing else from this listing. There is no subscription.
Where does the data come from?
Only publicly available sources. The Actor does not log in to any account or bypass access controls.
Can I use it from an AI assistant?
Yes. Add the MCP URL shown above to any MCP-capable client (Claude, Cursor, VS Code and others) and the Actor becomes a callable tool.