Sitemap Extractor + robots.txt & llms.txt AI Crawler Checker

Get every URL from a website's sitemaps (nested indexes and .gz files included) together with its robots.txt rules, AI-crawler policy (GPTBot, ClaudeBot, PerplexityBot ...) and llms.txt - as one clean, stable JSON dataset. No page scraping, no browser, no proxies: only the public files that websites publish for crawlers. Built to be called by humans, workflows and AI agents.

What it does

For each website you provide, the Actor:

1. Fetches robots.txt, parses all rules and the declared Sitemap: lines.
2. Discovers and crawls all sitemaps - sitemap index files, .xml.gz, plain-text sitemaps - and outputs one record per URL (lastmod, changefreq, priority, image count, source sitemap).
3. If no sitemap is declared, probes the common locations (/sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml ...), honouring robots.txt.
4. Reads /llms.txt and /llms-full.txt (the emerging standard that tells AI models what a site is about) and parses title, summary, sections and links.
5. Reports, for 14 well-known AI crawlers, whether the site blocks or allows them.

Use cases

• RAG / LLM ingestion: get the list of all pages (filtered by regex or lastModifiedAfter) before you fetch and embed them.
• SEO and site audits: sitemap coverage, stale lastmod dates, missing sitemaps or robots.txt, accidental Disallow: /.
• AI-visibility / GEO audits: which sites publish llms.txt, which block GPTBot or ClaudeBot.
• Competitor and content monitoring: schedule it daily with lastModifiedAfter to detect new pages.
• AI agents: a one-call tool that answers "what pages does this site have and may I crawl them?"

Input

Field	Type	Default	Description
urls	string[]	required	Domains or URLs. Only the site root is used.
includeSitemapUrls	boolean	true	Output sitemap_url records.
includeSiteSummary	boolean	true	Output one site_summary record per site (robots + llms.txt + stats).
maxUrlsPerSite	integer	1000	Cap on URLs per site (max 5,000,000).
maxItems	integer	1000	Hard cap on billed results for the whole run.
maxSitemapsPerSite	integer	50	Cap on sitemap files fetched per site.
urlIncludePattern / urlExcludePattern	regex string	-	Filter URLs, e.g. /blog/.
lastModifiedAfter	ISO date	-	Only URLs with lastmod on or after this date.
llmsTxtMaxChars	integer	100000	Truncate stored llms.txt text.
requestTimeoutSecs	integer	20	Per-request timeout.

Example:

{
  "urls": ["example.com", "https://www.anthropic.com"],
  "maxUrlsPerSite": 500,
  "urlIncludePattern": "/blog/",
  "lastModifiedAfter": "2025-01-01"
}

Output

Every record has recordType, url and scrapedAt. Three record types share one dataset (use the dataset views Sitemap URLs, Site summaries, Errors).

sitemap_url:

{
  "recordType": "sitemap_url",
  "url": "https://www.sitemaps.org/protocol.html",
  "domain": "www.sitemaps.org",
  "lastmod": "2022-12-15",
  "changefreq": null,
  "priority": null,
  "imageCount": 0,
  "sourceSitemap": "https://www.sitemaps.org/sitemap.xml",
  "scrapedAt": "2026-10-01T19:15:02.353Z"
}

site_summary (abridged):

{
  "recordType": "site_summary",
  "url": "https://llmstxt.org",
  "domain": "llmstxt.org",
  "robots": { "found": true, "blocksEntireSiteForAll": false, "declaredSitemaps": ["..."],
              "aiCrawlers": { "GPTBot": "default_allowed", "ClaudeBot": "blocked" } },
  "llmsTxt": { "found": true, "title": "llms.txt", "summary": "...", "linkCount": 3, "sections": [ ... ] },
  "llmsFullTxt": { "found": false },
  "sitemaps": { "fetched": 1, "failed": 0, "urlsExtracted": 2, "truncated": true },
  "scrapedAt": "2026-10-01T19:15:02.353Z"
}

AI-crawler values: blocked, allowed (explicit rules for that bot), default_allowed, default_blocked (inherited from User-agent: *).

error records (free): invalid input, unreachable sites, blocked targets - one bad site never stops the run.

How much does it cost?

Pay per event: 0.001 USD per result (about 1.00 USD per 1,000) plus a tiny start fee of 0.00005 USD per run (per GB of memory, this Actor uses 1 GB at most). One sitemap_url record or one site_summary record = 1 result. error records are free. Platform usage is included in the event price.

Scenario	Results	Approx. cost
Audit 100 sites, summary only (includeSitemapUrls: false)	100	0.10 USD
5 sites with the default maxUrlsPerSite of 1,000 (+ 5 summaries)	up to 5,005	about 5.01 USD
One large site, 10,000 URLs + 1 summary	10,001	about 10.00 USD
One run with default settings (maxItems 1,000)	max 1,000	max about 1.00 USD

Cost control: maxItems is a hard cap on billed results for the whole run (default 1,000), and the Actor also stops when your run's maximum cost per run setting is reached. Use maxUrlsPerSite, urlIncludePattern and lastModifiedAfter to pay only for the URLs you need.

Why this Actor (vs. alternatives)

The Store has single-purpose tools: sitemap URL extractors, robots.txt checkers and llms.txt/AI-crawler auditors. This Actor puts the three in one run and one dataset:

	This Actor	Sitemap-only extractors (e.g. apify/sitemap-extractor)	AI-crawler / llms.txt audit Actors
All URLs from nested / gzipped sitemaps	yes	yes	not their focus
robots.txt rules + AI-crawler block/allow for 14 bots	yes	no	yes
llms.txt / llms-full.txt parsed	yes	no	yes
HTTP status check of every URL	no	yes (apify/sitemap-extractor)	no
Page content / scores / fix suggestions	no	no	varies

Pick this one when you need the URL list and the crawler/AI policy of the same sites in a single call. If you only need cheap bulk URL lists with status codes, a sitemap-only Actor can be the better fit. (Comparison based on the public Store listings, 2026-10-02.)

Use it from n8n, Make, code and AI agents

n8n / Make: use the Apify node/module "Run Actor" (or "Run Actor and get dataset items") with the input above and read the dataset items.

API (curl):

curl -X POST "https://api.apify.com/v2/acts/ai_solution~sitemap-robots-llms-extractor/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"urls":["example.com"],"maxUrlsPerSite":100}'

MCP / AI agents: add this Actor in the Apify MCP server (https://mcp.apify.com, tool ai_solution/sitemap-robots-llms-extractor). The input schema is fully described so an agent can call it without extra instructions.

Scheduling and webhooks

1. In Apify Console open Schedules, create a schedule (for example 0 6 * * 1 = every Monday 06:00 UTC) and add this Actor with your input.
2. In the Actor's Integrations tab (or the schedule's run) add a webhook for the event ACTOR.RUN.SUCCEEDED pointing to your endpoint or an n8n/Make Webhook node. The default payload contains resource.defaultDatasetId.
3. Fetch the results: GET https://api.apify.com/v2/datasets/<defaultDatasetId>/items?token=...

To detect new pages, pass a fresh lastModifiedAfter date on each run (for example with an n8n expression and the Run Actor node, or via the API).

FAQ and limitations

• Does it scrape pages? No. It only reads robots.txt, sitemaps and llms.txt.
• Is robots.txt respected? Yes: the Actor identifies itself with its own User-Agent, throttles requests per host (about 3/s max), and does not probe guessed sitemap locations that robots.txt disallows. Sitemaps explicitly declared in robots.txt are fetched, as search engines do.
• JavaScript-generated or login-protected sitemaps are not supported. Sites behind bot protection (Cloudflare challenge etc.) may return errors - the Actor does not try to bypass them.
• Private/localhost addresses are refused. Very large sitemaps are capped at 60 MB per file.
• lastmod values are returned as published by the site; they are often inaccurate.

Changelog

• 1.0.4 - Lower price: 0.001 USD per result (was 0.0015 USD). No other changes.
• 1.0.3 - Clearer README (cost examples, comparison, scheduling), new Store title and description. No changes to input, output fields, behaviour or pricing.
• 1.0.2 - Output schema (dataset views in the Output tab).
• 1.0.0 - First release.

Support / Issues

Found a bug or need a field added? Open a ticket in this Actor's Issues tab in Apify Console.

Legal

Only publicly available files that sites publish for crawlers are fetched. No personal data is collected. You are responsible for using the output in line with the target sites' terms and applicable law.