Sitemap & llms.txt URL Extractor

Get every URL a website publishes, as clean structured data. Give it a domain; it reads robots.txt, follows every sitemap it names, walks sitemap indexes to the end, unpacks gzipped sitemaps, probes the usual sitemap paths when robots.txt declares none, and reads llms.txt and llms-full.txt. No browser, no crawling, no login.

Typical uses: SEO audits and content inventories, site migrations and redirect maps, change monitoring with the lastmod date, feeding a URL list to a scraper or a crawler, and AI agents building a corpus of a site's pages.

Why this one

Sitemaps are a small format with a lot of edge cases, and most extractors fall over on them. This Actor handles the ones that actually break in production:

• Sitemap indexes, followed recursively, not just the first file.
• Gzipped sitemaps (.xml.gz), including servers that send them as application/octet-stream with no transparent decoding.
• Plain-text sitemaps, one URL per line, which the specification allows and parsers forget.
• Servers that answer 200 with an HTML error page where a sitemap should be. A missing llms.txt on a large site often returns a 60 KB HTML page with a success status; this Actor detects that and reports "not found" rather than returning nonsense.
• Several Sitemap: directives in one robots.txt, and relative paths in them.
• Duplicate URLs across sitemaps, removed per site.

What you get for every URL

Field	Example
url	https://stripe.com/payments
site	https://stripe.com
lastmod	2026-09-18T09:11:59.922Z (when the sitemap publishes it)
changefreq, priority	weekly, 0.8
source	robots, probe, sitemap-index, llms.txt or llms-full.txt
foundIn	the exact sitemap or text file the URL came from
title	the link label, for llms.txt entries
scrapedAt	ISO date

The run's key-value store also gets a SUMMARY record with, per website: the sitemaps found in robots.txt, the paths that were probed, how many sitemap files were read, any sitemap that failed and why, whether llms.txt exists, and how many URLs were found and returned.

Input

{
    "websites": ["stripe.com", "https://vercel.com"],
    "includeLlmsTxt": true,
    "excludePatterns": ["/tag/", "\\?"],
    "lastmodSince": "2026-01-01",
    "maxUrlsPerSite": 5000
}

• websites: domains or URLs. Anything you pass is reduced to its origin, so a deep link works.
• includeLlmsTxt: also read /llms.txt and /llms-full.txt and return the links they list, with their labels. On by default.
• probeCommonPaths: when robots.txt names no sitemap, try /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml and a few more. On by default.
• includePatterns / excludePatterns: regular expressions applied to the URL. Filtered-out URLs are never charged.
• lastmodSince: keep only URLs changed on or after this date.
• maxUrlsPerSite, maxUrls, maxSitemapsPerSite: caps.

Output example

{
    "site": "https://stripe.com",
    "url": "https://stripe.com/payments",
    "lastmod": null,
    "changefreq": null,
    "priority": null,
    "source": "llms.txt",
    "foundIn": "https://stripe.com/llms.txt",
    "title": "Stripe Payments",
    "scrapedAt": "2026-09-21T20:05:00.000Z"
}

What is llms.txt

llms.txt is a Markdown file at the root of a site that lists the pages worth reading, written for AI agents rather than search engines. Stripe publishes 307 links in theirs, Apify 72. It is the fastest way to get a site's own view of what matters, and almost no extractor reads it. This one does, and it tells you in SUMMARY whether the file exists at all.

Pricing

Pay per event: $1.01 per 1,000 URLs returned, and platform usage is on us. Filters run before charging, so an exclude pattern or a lastmodSince only bills the URLs you keep. Use maxUrls or the run's maximum total charge to cap spend.

Using it from an AI agent or via API

Input is a short JSON and output is a uniform list of URLs, which makes this a natural first step for an agent that has to work through a site: get the URL list here, then fetch the pages it actually needs. Call it through the Apify API, the Apify MCP server or any Apify client, and read the default dataset.

Legal and data policy

This Actor reads only files a site publishes for machines to read: robots.txt, its declared sitemaps, and llms.txt. It sends one plain HTTP request per file, follows redirects, and never renders or crawls pages. It collects no personal data.

More tools

Same approach, same output discipline:

• ATS Jobs Scraper — every open role from any company’s Greenhouse, Lever or Ashby job board.
• Website Screenshot & PDF API — any public page rendered into an image or an A4 PDF.

FAQ

A website returns no URLs. Check SUMMARY: it names the sitemaps found in robots.txt, the paths probed, and the reason each sitemap failed. Some sites publish no sitemap at all; this Actor will not crawl to invent one.

Can it handle sites with hundreds of thousands of URLs? Yes. Raise maxSitemapsPerSite and set maxUrls to the number you actually want, so you stay in control of the bill.

How do I track changes over time? Schedule the Actor and use lastmodSince with the date of your last run to get only what changed, which costs a fraction of a full pass.

Which ATS, shop or CMS does it support? All of them. It reads the standard files, so a WordPress, Shopify, Webflow, Next.js or hand-written site is the same to it.