Sitemap Extractor and Website URL Finder (works without a sitemap)

Give it a domain, a homepage or a sitemap link, and it returns every page URL it can find, one row per unique URL. For each URL you get lastmod, changefreq and priority when the sitemap has them, plus where the URL came from.

It starts from robots.txt, follows nested sitemap indexes, reads .xml, .xml.gz and plain-text sitemaps, and falls back to RSS/Atom feeds. If a site has no sitemap and no feed, it follows internal links for a small number of pages. It opens no browser, needs no login and reads no personal data.

Who uses it

Use	How
SEO and content audits	Pull the full URL list with lastmod, filter to /blog/, compare against your analytics or Search Console
Site migrations	Export all URLs of the old site before you redirect them
Competitor research	List every product or category URL of a store you want to study
AI and RAG pipelines	Get the list of pages to feed into a scraper, embedder or crawler
Change monitoring	Schedule it with a "Modified after: 7 days" filter to see what was updated this week
Broken-link work	Use the URL list as the input for a link checker

How it works

For each website, in this order:

1. Reads robots.txt and takes every Sitemap: line. It also tries http:// and the www. variant if the first host does not answer.
2. Downloads each sitemap. Indexes are followed down to the depth you set (default 5), and loops are detected. Gzipped files are unpacked. Text sitemaps (one URL per line) work too.
3. If robots.txt lists nothing usable, it tries the usual places: /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml, /sitemap.xml.gz, /sitemap.txt and a few more. It also checks the homepage for <link rel="sitemap">.
4. If there is still nothing, it reads RSS/Atom feeds (the ones the homepage points to, or common paths like /feed). This is limited by the feeds setting.
5. If there is no feed either, it crawls. It opens up to maxCrawlPages pages (default 50) of the same domain and returns every internal link it finds on them. 50 pages usually give a few hundred URLs, not the whole site.

Each URL is returned once per run, even if it shows up in two sitemaps or two of your websites. A start URL with a path, such as https://example.com/blog/, returns only URLs under that path. Turn that off with limitToStartPath if you want the whole site.

Websites run in parallel (10 by default). A site that answers every request with 403, 429 or a bot-challenge page is dropped early instead of being retried for minutes, and it is not charged.

Input

{
    "startUrls": ["nodejs.org", "https://www.sitemaps.org/sitemap.xml"],
    "includeUrlPatterns": ["/blog/"],
    "lastmodAfter": "90 days",
    "maxUrlsPerSite": 1000
}

Field	Type	Default	What it does
startUrls	list	required	Domains, homepages or direct sitemap/feed URLs. Duplicates are removed
includeUrlPatterns	list	empty	Keep only URLs matching at least one pattern. Regex (case-insensitive) or simple wildcards, e.g. /products/.*, \.pdf$
excludeUrlPatterns	list	empty	Drop URLs matching any pattern, e.g. /tag/, \?
lastmodAfter	text	empty	Keep URLs modified on or after this date. 2025-01-31 or 30 days, 2 weeks, 6 months
lastmodBefore	text	empty	Keep URLs modified on or before this date. Same formats
includeUrlsWithoutLastmod	boolean	false	With a date filter on, also keep URLs that have no lastmod
maxUrlsPerSite	integer	0	Stop a site after this many URLs (after filters). 0 means no limit
limitToStartPath	boolean	true	Only URLs under the path of the start URL
feeds	fallback / always / never	fallback	fallback reads feeds only when no sitemap URLs were found. always adds feed URLs too
crawlFallback	boolean	true	Follow internal links when there is no sitemap and no feed
maxCrawlPages	integer	50	Pages opened by the crawl fallback, per site. Max 5000
includeExtras	boolean	false	Add images and hreflang alternates from the sitemap
respectRobotsTxt	boolean	true	Skip disallowed paths when probing and crawling. Sitemaps listed in robots.txt are always read
maxConcurrency	integer	10	Websites processed at once
sitemapConcurrency	integer	5	Sitemap files downloaded at once from one site
maxSitemapDepth	integer	5	Levels of nested indexes to follow
requestTimeoutSecs	integer	60	Time allowed to download one file
siteTimeoutSecs	integer	900	Time limit per site. When reached, the site keeps what it already collected
maxRetries	integer	2	Retries on network errors, 429 and 5xx. Each retry switches HTTP version or client
ignoreSslErrors	boolean	true	Read sitemaps from sites with expired or broken certificates
proxyConfiguration	object	no proxy	Only needed when a site blocks the requests

Output

One row per URL. This is a real row from a run on allbirds.com with includeExtras on, shortened:

{
    "inputUrl": "https://www.allbirds.com",
    "site": "allbirds.com",
    "url": "https://www.allbirds.com/products/mens-wool-runners",
    "lastmod": "2026-10-05T03:55:27.000Z",
    "changefreq": "daily",
    "priority": null,
    "source": "sitemap",
    "sourceUrl": "https://www.allbirds.com/sitemap_products_1.xml?from=1878193471557&to=7369944137808",
    "status": "ok",
    "error": null,
    "extractedAt": "2026-10-05T03:55:26.049Z",
    "images": ["https://cdn.shopify.com/s/files/1/1104/4168/files/Allbirds_WL_RN_SF_PDP_Natural_Grey_LAT.png?v=1751143404"],
    "alternates": []
}

source is sitemap, rss, atom or crawl. URLs from a crawl have no lastmod, changefreq or priority. lastmod is converted to ISO 8601; if the sitemap holds a value that cannot be parsed, the raw text is kept. Fragments (#...) are removed.

A website that returns no URLs gets one free status row instead, so you always know why:

status	Meaning
ok	A URL row. This is the only status that is charged
failed	The site could not be reached, the sitemap could not be read, or the site blocks automated requests
no-urls	No sitemap, no feed and no crawlable links
no-match	URLs were found, but none passed your filters or the start path
skipped	robots.txt disallows the site and lists no sitemap
invalid-input	The line was not a valid domain or URL
not-processed	The run hit the cost limit or the memory limit before this site

The dataset has two views: the URL table, and "Websites without URLs". A run summary (sitemaps found per site, sitemap errors, methods used, duration) is saved under SUMMARY in the key-value store.

Pricing

You pay per extracted URL (event url-extracted), plus a tiny start fee per run.

Apify plan	Price per 1,000 URLs
Free	$0.80
Starter / Bronze	$0.40
Scale / Silver	$0.30
Business / Gold	$0.20

Actor Start: $0.00005 per run.

What counts as a charged URL: a row with status: ok that was saved to the dataset. Not charged: status rows, invalid input, sites that fail, duplicates, URLs removed by your filters, and URLs cut by maxUrlsPerSite. Charges are sent in batches of 500 rows, so the bill follows the rows in your dataset.

Your spending cap works. The Actor reads your maximum cost per run and stops cleanly when the next URL would go over it. It keeps everything saved so far and marks the unvisited sites as not-processed. The summary says stoppedByCostLimit: true.

Example costs, on the Free plan and on Business:

URLs extracted	Free	Business
10,000	$8.00	$2.00
1,000,000	$800	$200

Add $0.00005 per run to each. These are the Actor's own prices. Apify platform usage (compute) is billed separately according to your plan, and sitemap reading is light, so it is usually small next to the per-URL price.

Tip for a first run: set maxUrlsPerSite to 1000 and a low maximum cost per run, check the rows, then raise both.

Limits

• Very large sites. Each sitemap file is downloaded in one piece, with a cap of 100 MB per file. Sites with millions of URLs have many files, which is fine, but each file takes memory while it is parsed. A million-URL run needs several GB of RAM, so raise the run memory (the Actor allows up to 4 GB). If memory reaches about 90%, the Actor stops on purpose, keeps the saved URLs and sets stoppedByMemory. Splitting the job with includeUrlPatterns or by sub-sitemap URL is safer than one giant run.
• Time. Each site has a time limit (900 seconds by default). A big site that hits it returns partial results and says so in the summary notes. Raise siteTimeoutSecs if you need the rest.
• Sites that block bots. Some sites answer 403, 429 or a challenge page. The Actor gives up on them after three bad answers in a row and reports failed. A proxy can help, but there is no promise. It does not solve captchas or log in.
• Incomplete sitemaps. A sitemap lists what the site owner chose to list. Pages missing from it are not returned. Many sitemaps are out of date, and lastmod is often missing or set to the build date for every page. If you need to know whether the sitemap is complete, compare against a crawl. The crawl fallback only runs when no sitemap was found, so it will not fill gaps in a sitemap that exists.
• The crawl fallback is light on purpose. It reads the HTML the server sends, not JavaScript-rendered links, and stops after maxCrawlPages pages. It is for sites with nothing else to read, not a full crawler.
• Duplicates are counted once per run, across all your start URLs. The same URL listed by two of your websites appears once.
• A URL is a listing, not a check. The Actor does not request each page. A URL in a sitemap can return 404 today.
• Order. Rows are not sorted. Sort in the dataset or after export.

Responsible use

The Actor reads public files: robots.txt, sitemaps, feeds and, in the fallback, public HTML pages. It does not log in, does not use cookies from a session and does not store personal data, only URLs and their metadata. With respectRobotsTxt on (the default), it skips paths that robots.txt disallows when it probes common sitemap locations and when it crawls. Sitemaps that robots.txt itself lists are always read, because the site publishes them for crawlers. Requests per site are limited by sitemapConcurrency (5 by default); lower it for small sites. You are responsible for the terms of the sites you point it at and for what you do with the URLs.

FAQ

Does it work on a site with no sitemap? Yes, in a limited way. It tries feeds, then follows internal links for up to maxCrawlPages pages. The result is the part of the site reachable from those pages, not a guaranteed full list.

Why did I get fewer URLs than the site has? Usually the sitemap is incomplete, or you hit maxUrlsPerSite, the site time limit, or your cost cap. Check the SUMMARY record for notes and stoppedByCostLimit.

What do I get for a site that fails? One row with status and a plain error text, at no cost.

Can I give it a sitemap URL directly? Yes. A link ending in .xml, .xml.gz or a feed path is read as is, without looking for robots.txt or other sitemaps.

Can I get only the blog, or only pages changed recently? Yes. Use a start URL with a path (https://example.com/blog/) or includeUrlPatterns, and lastmodAfter such as 30 days. Many sitemaps have no lastmod; set includeUrlsWithoutLastmod to keep those URLs.

Do I need a proxy? Almost never. Sitemaps are public. Add one only when a site blocks the requests.

Can I call it from the API or an AI agent? Yes. Run it through the Apify API, a webhook, a schedule or the Apify MCP server, and read the dataset items.

Feedback

Found a site that does not work, or want a feature? Open an issue in the Issues tab of this Actor and include the start URL and the run link. Bug reports with a run link get fixed fastest.

Not affiliated with any of the websites you extract from.