# Sitemap URL Extractor + Robots.txt & llms.txt Validator (`far003/sitemap-url-extractor-validator`) Actor Extract every URL from a site's sitemaps and check the sitemap files, robots.txt (rules for AI bots) and the structure of llms.txt. Streaming parser, flat memory; sites that cannot be read are free. - **URL**: https://apify.com/far003/sitemap-url-extractor-validator.md - **Developed by:** [Francesco Antonio Russo](https://apify.com/far003) (community) - **Categories:** SEO tools, Developer tools, AI - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing $0.40 / 1,000 results This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## Sitemap URL Extractor + Robots.txt & llms.txt Validator Reads the sitemaps of a website and hands you every URL in them, one record each, with dates, images, videos, news and language alternates. In the same pass it checks the sitemap files themselves (format, limits, errors), `robots.txt` (RFC 9309, with a table of which AI bots are allowed or blocked) and the structure of `llms.txt`. The sitemap is read as a stream, so memory stays flat whether the file has 1,000 or 1,000,000 URLs, and a site that fails never stops the others. Data source: only files that the site itself publishes for robots (`robots.txt`, sitemaps, `llms.txt`, `llms-full.txt`). Nothing else on the site is fetched, no JavaScript is run, and the URLs found are **not** requested to see whether they answer. ### What you get Two kinds of records in the same dataset, told apart by the field `type`: - `url`: one per URL found in a sitemap: `url`, normalized `lastmod`, `changefreq`, `priority`, the sitemap file it came from, images, videos, news, `hreflang` alternates, whether `robots.txt` allows the path (`robotsAllowed`), which AI bots it bars from it (`aiBotsBlocked`), what is wrong with it (`issues`) and, with *Only new or changed URLs*, whether it is `new` or `updated`. - `site`: exactly one per entry of your input, always delivered, even when everything fails: status, the list of sitemap files read (type, HTTP status, compressed or not, size, number of URLs, problems), the `robots.txt` report with the AI-bot table, the `llms.txt` report, totals, a 0-100 score with its breakdown, warnings. In the Output tab you find two views: **URLs** and **Site reports**. The views do not filter rows by `type`: the URLs view shows the `site` rows with empty cells and the other way round. To get only one kind, filter on `type` after reading (in n8n an *IF*/*Filter* node on `{{ $json.type }}`; in Python `item["type"]`), and through the API pass `fields=` with the columns you want (for example `fields=type,site,status,score,urlsFound,aiBotsBlockedList`). The `site` record has flat fields for a spreadsheet (`urlsFound`, `urlsDelivered`, `robotsTxtFound`, `robotsTxtIssueCount`, `aiBotsBlockedCount`, `aiBotsBlockedList`, `aiBotsRestrictedList`, `aiBotsNamedList`, `llmsTxtFound`, `llmsTxtValid`, `sitemapIssueCount`, `urlIssueCount`) next to the detailed objects, and the Site reports view shows them first. #### Example `url` record (real run, `llmstxt.org`) ```json { "type": "url", "site": "https://llmstxt.org", "url": "https://llmstxt.org/core.html", "lastmod": "2026-09-24T18:20:59.055Z", "lastmodRaw": "2026-09-24T18:20:59.055Z", "changefreq": null, "priority": null, "sitemapUrl": "https://llmstxt.org/sitemap.xml", "sourceFormat": "xml", "images": [], "imageCount": 0, "videos": [], "videoCount": 0, "news": null, "alternates": [], "robotsAllowed": true, "aiBotsBlocked": [], "change": "new", "issues": [], "checkedAt": "2026-10-13T10:11:15.640Z" } ``` #### Example `site` record (real run, `llmstxt.org`) The same run; only the `aiBots` table is shortened here (the real record lists 24 bots). ```json { "type": "site", "site": "https://llmstxt.org", "input": "llmstxt.org", "status": "ok", "error": null, "robotsTxt": { "url": "https://llmstxt.org/robots.txt", "httpStatus": 200, "found": true, "botWall": false, "sizeBytes": 41, "interpretation": "parsed", "note": null, "groups": 0, "sitemaps": [ "https://llmstxt.org/sitemap.xml" ], "crawlDelay": null, "issues": [], "syntaxErrorLines": [], "unknownDirectives": [], "aiBots": { "GPTBot": { "rule": "allowed", "matchedGroup": "none", "explicit": false, "blockedPathCount": 0 }, "ClaudeBot": { "rule": "allowed", "matchedGroup": "none", "explicit": false, "blockedPathCount": 0 }, "Googlebot": { "rule": "allowed", "matchedGroup": "none", "explicit": false, "blockedPathCount": 0 } } }, "llmsTxt": { "url": "https://llmstxt.org/llms.txt", "httpStatus": 200, "found": true, "botWall": false, "sizeBytes": 637, "valid": true, "linksWellFormed": true, "title": "llms.txt", "summary": "A proposal that those interested in providing LLM-friendly content add a /llms.txt file to their site. This is a markdown file that provides brief background information and guidance, along with links to markdown files providing more detailed information.", "sections": [ { "title": "Docs", "linkCount": 3 } ], "sectionCount": 1, "linkCount": 3, "hasOptionalSection": false, "llmsFullTxt": { "found": false, "sizeBytes": 0 }, "issues": [], "malformedLines": [], "note": null }, "sitemaps": [ { "url": "https://llmstxt.org/sitemap.xml", "type": "urlset", "httpStatus": 200, "compressed": true, "sizeBytes": 880, "urlCount": 7, "lastmod": null, "crossHost": false, "issues": [], "detail": null, "errorLine": null } ], "sitemapCount": 1, "totals": { "urlsFound": 7, "urlsDelivered": 7, "duplicates": 0, "filteredOut": 0, "heldByLimit": 0, "unchanged": 0, "resumedUrls": 0, "newUrls": 7, "updatedUrls": 0, "removedSinceLastRun": 0, "urlsWithIssues": 0, "urlsWithRemarks": 0, "issuesByCode": {}, "sitemapIssuesByCode": {} }, "complete": true, "score": 100, "scoreIsPartial": false, "scoreChecks": [ "robots", "sitemaps", "llms" ], "scoreBreakdown": [ { "component": "robots_txt_present_and_clean", "points": 20, "max": 20, "note": "robots.txt found, no problem" }, { "component": "robots_txt_declares_sitemaps", "points": 10, "max": 10, "note": "every declared sitemap was read" }, { "component": "sitemap_found_and_parsed", "points": 15, "max": 15, "note": "at least one sitemap file was parsed" }, { "component": "sitemap_files_clean", "points": 20, "max": 20, "note": "1 of 1 sitemap files without problems" }, { "component": "urls_without_issues", "points": 20, "max": 20, "note": "7 of 7 URLs without problems" }, { "component": "urls_with_valid_lastmod", "points": 5, "max": 5, "note": "7 of 7 URLs have a valid lastmod" }, { "component": "llms_txt", "points": 10, "max": 10, "note": "valid llms.txt, no remark" } ], "warnings": [], "charged": true, "checkedAt": "2026-10-13T10:11:13.629Z", "urlsFound": 7, "urlsDelivered": 7, "robotsTxtFound": true, "robotsTxtIssueCount": 0, "aiBotsBlockedCount": 0, "aiBotsBlockedList": [], "aiBotsRestrictedList": [], "aiBotsNamedList": [], "llmsTxtFound": true, "llmsTxtValid": true, "sitemapIssueCount": 0, "urlIssueCount": 0 } ``` ### Input | Field | Default | What it does | |---|---|---| | `urls` | required | Domains (`example.com`), site URLs (`https://example.com/`) or the URL of a sitemap file. One site report per entry. Private, local or malformed entries come back as `invalid` (free). | | `extractUrls` | `true` | Deliver the URL records. Off = site reports only (a sample of up to 50 sitemap files per site is read and checked, see `maxSitemapsPerSite`; the URLs are not delivered or charged). | | `maxUrlsPerSite` | `0` (no limit) | Stop a site after this many URL records. | | `maxSitemapsPerSite` | `0` (automatic, max 5,000) | How many sitemap files of one site are read. Automatic = 5,000 when the URLs are extracted and **50** when `extractUrls` is off (the site-report mode reads a declared sample: the report says "N of M sitemap files checked (audit sample)", `complete` is `false`, no score points are lost). Set a number to choose your own limit. | | `includePatterns` / `excludePatterns` | empty | Glob patterns on the URL: `*` stays inside one path segment, `**` crosses segments. A pattern starting with `/` is matched on path and query. Exclusion wins. | | `modifiedAfter` | none | A date (`2026-01-31`) or `30 days`. URLs older than that, or without a valid `lastmod`, are not delivered. | | `onlyNew` | `false` | Deliver only URLs that are new or whose `lastmod` changed since the previous run with this option (see below). | | `checkRobotsTxt` | `true` | Read and validate `robots.txt`, build the bot table, fill `robotsAllowed` / `aiBotsBlocked`. | | `checkLlmsTxt` | `true` | Read and validate `/llms.txt`, look for `/llms-full.txt`. | | `extraUserAgents` | empty | Bot names to add to the table (for example your own crawler). | | `respectRobotsTxt` | `true` | Do not fetch a sitemap or `llms.txt` that `robots.txt` forbids to this Actor's user-agent (`SitemapRobotsLlmsBot`). If `robots.txt` answers with a server error, RFC 9309 says to assume everything is disallowed. | | `requestsPerSecond` | `2` (1-5) | Per host. A `Crawl-delay` in `robots.txt` that is slower wins, capped at 10 seconds. | | `maxConcurrency` | `5` | Sites read at the same time. | | `timeoutSecs` | `30` | Maximum wait for the response headers and maximum silence while a file downloads. | | `maxSitemapMegabytes` | `50` | Uncompressed size read from one sitemap; a bigger file is cut there and flagged `truncated_at_limit`. | A wrong option (a word where a number is expected, `requestsPerSecond: 9`, a bad date) stops the run **before** any request or charge, with the reason in the error message. #### How sitemaps are found 1. `Sitemap:` lines of `robots.txt`. A line may point to another host (`robots.txt` of `www.example.com` declaring `http://www.example.cn/sitemap.xml`): it is read, with `crossHost: true` and the informational remark `cross_host_declared_in_robots` (not a failure, no points lost); the `robots.txt` of that host is read first and obeyed (when `respectRobotsTxt` is on), with its `Crawl-delay`; if that file is an index, its children are followed while they stay on that host. URLs inside it are not flagged `different_host` when they are on the site's host or on the host of that sitemap. 2. If there are none: `/sitemap.xml`, `/sitemap_index.xml`, `/sitemap-index.xml`, `/wp-sitemap.xml`, `/sitemap.xml.gz`, `/sitemap.txt`, in this order, stopping at the first one that is a sitemap. 3. If your input is already the URL of a sitemap, the read starts there (`robots.txt` and `llms.txt` are still read from its origin). Sitemap indexes are followed up to 5 levels deep, with loop protection. Formats: `urlset`, `sitemapindex`, RSS 2.0, Atom, plain text (one URL per line); gzip is recognised from the first bytes, not from the file name. Extensions: `image:image`, `video:video`, `news:news`, `xhtml:link` (hreflang). The sitemap elements are recognised by their name, in any namespace (or none), and the extensions with tolerance (http or https, trailing slash, any version, or the conventional prefix `image:` `video:` `news:` `xhtml:` whatever the URI is): a file whose namespace is not the standard one is still read, and says so with `missing_namespace`, `legacy_namespace` or `nonstandard_namespace` (a remark that costs a few score points, not a failure). If a domain redirects to another host (for example `docs.example.com` to `platform.example.com`), the host that `robots.txt` finally comes from is treated as part of the site. Likewise, when a sitemap of the site redirects to another host (for example `webflow.com/sitemap.xml` to a CDN), the children of that index are followed on the host it was served from. **Every request obeys the `robots.txt` of the host it goes to** (when `respectRobotsTxt` is on): the bare domain and `www.`, `http://` and `https://`, a sitemap or `llms.txt` on another host, the host a redirect leads to. Each of those `robots.txt` files is read once, before the first request to that host; if one cannot be fetched (server error, no answer), RFC 9309 says everything there is disallowed. The fields `robotsAllowed` and `aiBotsBlocked` of a URL use the `robots.txt` of the URL's own host when that file was loaded in this run (because something was read from that host), otherwise the `robots.txt` of the site. ### What you pay and what you don't Pay-per-event, one event: **Result**. **$0.40 per 1,000 results** (that is $0.0004 per result). Platform usage is included in the price. | Delivered record | Charged? | |---|---| | every `url` record | yes, 1 result | | `site` record, status `ok`, `partial` or `no_sitemap`, when at least one of `robots.txt`, `llms.txt` or a sitemap was actually read | yes, 1 result | | `site` record with status `blocked` (HTTP 401/403/429/503, an empty 202 or a challenge page, or `robots.txt` forbids it), `unreachable`, `invalid` | free | | `site` record where nothing was read | free | | `site` record delivered after the spending limit was reached | free, with a warning | | URLs already delivered before a restart of the run (migration) | not delivered or charged again | | URLs left out by filters, duplicates, unchanged URLs with `onlyNew` | not delivered, not charged | Example: one site with 2,000 URLs in its sitemaps = 2,000 URL results + 1 site report = 2,001 results = **$0.8004**. The same site with `extractUrls` off = 1 result = **$0.0004**. A domain that does not exist = $0. The spending limit you set on the run is respected to the single result: when it is reached the Actor stops cleanly (`spending limit reached`), the URLs found but not delivered are counted in `totals.heldByLimit`, and they are **not** remembered by `onlyNew`, so the next run delivers them. The Actor does not try to get around blocks: one honest user-agent that names this bot, no proxies, no disguise. ### Only new or changed URLs With `onlyNew`, the Actor keeps in a named key-value store (`sitemap-robots-llms-state`), for each site, a compact map from a 52-bit hash of the URL to its `lastmod`. The first run delivers everything as `new`. Later runs deliver a URL only if it is not in the map (`new`) or its `lastmod` differs (`updated`). - **Only what was delivered is remembered.** Filters, `maxUrlsPerSite`, `modifiedAfter` and the spending limit never write into the memory, so a URL you filtered out is simply `new` the day you stop filtering. - `removedSinceLastRun` (URLs remembered but no longer in the sitemaps) is computed only when the whole site was read, with no filter and no limit; otherwise it is `null` and a warning says why. A site that could not be read keeps its memory untouched. - Limit: at most 2,000,000 URLs are remembered per site. ### Status of a site report | `status` | Meaning | |---|---| | `ok` | Sitemaps found and read without a read failure. | | `partial` | Something could not be read (a listed sitemap answered an error, a file broke half way, was cut at the size limit), or the run stopped before the end (spending limit, timeout). The report says what. | | `no_sitemap` | The host answers but no sitemap was found; `robots.txt` / `llms.txt` are still reported. | | `blocked` | The server answered 401/403/429/503, an empty 202 (a typical WAF challenge) or a block page instead of the files, and no sitemap could be read; the record warns that a browser may see what we do not. Or `robots.txt` forbids the sitemaps to this bot or answered with a server error (RFC 9309). Free. | | `unreachable` | The host did not answer (DNS, connection, timeout, certificate). Free. | | `invalid` | The entry is not a usable web address, or points to a private or local address. Free. | ### How the score is made 100 points, seven parts, nothing hidden (`scoreBreakdown` in the record shows each one): | Part | Max | Points | |---|---|---| | `robots_txt_present_and_clean` | 20 | 20 found and without problems · 12 found with problems · 0 not found | | `robots_txt_declares_sitemaps` | 10 | 10 every declared sitemap was read · 5 some could not be read · 0 no `Sitemap:` line | | `sitemap_found_and_parsed` | 15 | 15 if at least one sitemap file was parsed | | `sitemap_files_clean` | 20 | 20 × (files without problems ÷ files known) | | `urls_without_issues` | 20 | 20 × (URLs without issues ÷ URLs found) | | `urls_with_valid_lastmod` | 5 | 5 × (URLs with a valid `lastmod` ÷ URLs found) | | `llms_txt` | 10 | 10 valid and without remarks · 7 valid with remarks · 3 found but not valid · 0 not found | `llms.txt` is not a mandatory standard, so its absence costs at most 10 points of 100. A remark of kind `cross_host_declared_in_robots` is informational and costs nothing; `missing_namespace`, `legacy_namespace` and `nonstandard_namespace` cost a few points through `sitemap_files_clean`. **Comparable scores.** The record field `scoreChecks` lists the checks the score is made of (`robots`, `sitemaps`, `llms`). If `checkRobotsTxt` or `checkLlmsTxt` is off, that part leaves the total, the rest is scaled to 100 (points earned over points possible) and a warning says so: compare scores only between runs with the same `scoreChecks`. The score is `null` (and `scoreChecks` empty) when nothing was read and when the status is `blocked` (what was not read is not a defect of the site). It describes the health of the files the site publishes for robots; it is not a measure of anything else. **When a limit stopped the reading** (`maxUrlsPerSite`, the spending limit, the run timeout, the 5,000-file cap, `maxSitemapsPerSite`, `maxSitemapMegabytes`) the record says so: `complete` is `false` and `scoreIsPartial` is `true`, and a warning calls the score provisional. **A limit never takes points away:** the parts that depend on files are computed in proportion to the files and URLs that were read, so the same clean site gets the same score read in full or in part (the notes say "3 more not opened, not judged"). What was not read is not judged; only what was read can lose points. A site that was not read at all has `score: null`: status `blocked`, `unreachable` or `invalid`, a run that stopped before the first sitemap was read, or a site that was never started. A site with a `robots.txt` and really no sitemap (`no_sitemap`) keeps its low score. `complete` is `true` only when the status is `ok` and nothing cut the reading short. ### robots.txt report Parsed under RFC 9309: groups, longest-match rule with Allow winning ties, `*` and `$` in paths. The file counts as found only with HTTP `200` and a non-empty body (an empty `200`, `202`, `204` and redirects that lead nowhere are `found: false`, with the status in `note`; `401`/`403`/`429`/`202` or a challenge page also set `botWall: true`). The answer is interpreted as the RFC says: `200` is parsed, `4xx` means no restrictions, `5xx` or no answer means everything is disallowed (and the report says so in `interpretation`). Problems reported: `syntax_error` (with line numbers, at most 20), `unknown_directive`, `too_large` (over 500 KiB), `bom`, `html_instead_of_text`, `no_user_agent_before_rule`, `sitemap_not_absolute`, `empty_file`. `aiBots` has one entry per bot: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, anthropic-ai, Google-Extended, GoogleOther, CCBot, PerplexityBot, Perplexity-User, Bytespider, Amazonbot, Applebot-Extended, meta-externalagent, FacebookBot, cohere-ai, DuckAssistBot, MistralAI-User, Diffbot, YouBot, plus Googlebot and Bingbot as a reference (list compiled 2026-10-12; your `extraUserAgents` are added). For each bot: - `rule`, one of four values: - `allowed`: no Disallow rule applies; - `blocked`: the root `/` is closed and nothing opens it again (`Disallow: /` without any Allow), whether the site says it in a group naming the bot or in the `*` group the bot follows; - `restricted`: the site wrote rules **for this bot** (a group that names it) and they do not close everything; - `default_rules`: the bot has no group of its own and follows the `*` group, which closes some paths (the ordinary `Disallow: /wp-admin/` of a WordPress site lands here, for every bot, Googlebot included); - `explicit`: `true` when the robots.txt names the bot in a group of its own; `matchedGroup`: `explicit`, `wildcard` (only the `*` group applies) or `none`; - `blockedPathCount`: number of non-empty Disallow rules that apply to the bot. In the site report the flat fields give the same answer for a spreadsheet: `aiBotsBlockedList` (rule `blocked`), `aiBotsRestrictedList` (rule `restricted`) and `aiBotsNamedList` (every bot with `explicit: true`, whatever its rule: the bots the site cares about by name). **Example: which sites let which AI bots in.** Run with `extractUrls: false` (site reports only) over a list of domains, then in Python: ```python for item in items: # items = dataset items if item["type"] == "site" and item["robotsTxtFound"]: print(item["site"], "closed to:", item["aiBotsBlockedList"], "| rules just for:", item["aiBotsRestrictedList"], "| named in robots.txt:", item["aiBotsNamedList"]) ``` `aiBotsBlockedList` is the list to act on; `default_rules` (a bot that only follows the `*` group) is not in any of the three lists on purpose: it means the site said nothing about that bot. ### llms.txt report Checked against the proposal at https://llmstxt.org/ (read on 2026-10-12): an optional BOM, one H1 (the only required part), an optional blockquote summary, optional text, then H2 sections that are lists of `- [name](url): notes`, with an `Optional` section by convention. The report gives title, summary, sections with their link counts, `hasOptionalSection`, and whether `/llms-full.txt` exists. `valid` is `true` when the file was found and its **structure** is what the proposal requires: one H1, and it comes first. It is `false` for a missing (`missing_h1`), repeated (`multiple_h1`) or late (`h1_not_first`) H1, an empty file (`empty_file`), an HTML page served for a missing file (`html_instead_of_markdown`) or a file over 1 MiB (`too_large`). Everything else is a remark that costs a few score points and does not make the file invalid: `no_summary_blockquote`, `no_sections`, `section_without_links`, `relative_link`, `wrong_content_type` and `malformed_link_line` (a list item that starts like a link but is not `[name](url)`, or is a bare URL; `[name](url): note`, `[name](url) - note` and `[name](url) note` are all well formed). The field `linksWellFormed` says whether any list item is malformed (`false` when at least one is). Only the root `/llms.txt` is checked. The check is about structure, not about whether the content is useful. ### Sitemap file problems `http_error`, `bot_wall_suspected` (HTTP 401/403/429, an empty 202 or a challenge page instead of the file: the site is answering our honest bot with a block page, so the file may be fine for a browser), `not_xml`, `invalid_line` (a plain-text sitemap with lines that are not URLs: blank lines and `#` comments are ignored, the bad lines are counted in `detail` and left out, the rest is read), `unescaped_ampersand` (a bare `&` in the file, for example in a `` with `?a=1&b=2`: not valid XML, but common; it is read as `&` in the stream so no URL after it is lost, and `detail` gives the count), `encoding_error` (bytes that are not valid in the encoding in use: the file is read all the same, with U+FFFD in the affected URLs; the encoding is taken from the byte order mark, then the `` declaration, then the `charset` of the Content-Type, else UTF-8 — UTF-16, ISO-8859-1 and Windows-1252 are read correctly), `html_instead_of_xml` (a soft 404), `wrong_root_element`, `missing_namespace`, `legacy_namespace` (the old `http://www.google.com/schemas/sitemap/0.84`), `nonstandard_namespace` (another spelling of the sitemap namespace, or of an image/video/news/xhtml extension namespace: the file is read all the same), `nonstandard_element_name` (the root is written with another case, for example ``: read all the same), `too_many_urls` (over 50,000), `too_large` (over 50 MB uncompressed), `truncated_at_limit`, `index_inside_index_too_deep`, `loop`, `empty`, `xml_error` (with the line), `gzip_error`, `listed_in_robots_but_missing` (declared in `robots.txt`, answers 404/410), `robots_disallowed`, `different_host` (a sitemap on another host that `robots.txt` does not declare), `cross_host_declared_in_robots` (informational: a sitemap on another host declared in `robots.txt`; read, not a defect). A file that breaks half way still gives you the URLs read before the break. URL problems: `different_host` (another domain than the site's), `different_subdomain` (another subdomain of the same registrable domain, for example `blog.example.com` in the sitemap of `example.com`: a light remark, kept in `issues` and in `issuesByCode`, not counted in `urlsWithIssues` or `urlIssueCount`, and weighing a quarter of a problem in the score; the domain is found with the public suffix list), `different_scheme`, `not_absolute`, `invalid_lastmod`, `future_lastmod`, `invalid_priority`, `invalid_changefreq`, `url_too_long` (over 2,048 characters), `blocked_by_robots`. `lastmod` accepts the W3C forms (year, year-month, date, date and time with or without a zone; no zone means UTC); `0000-00-00` and anything else is `invalid_lastmod` with `lastmod: null`. ### Limits, stated plainly - Sitemap indexes: 5 levels; at most 5,000 sitemap files per site (the `sitemaps` list in the record shows the first 500, `sitemapCount` is exact). - A sitemap file is read up to `maxSitemapMegabytes` uncompressed (default 50 MB); the standard allows 50,000 URLs per file and flags more. - Duplicates are detected per site with a 52-bit hash of the URL (not the full string, to keep memory flat); a URL listed twice is delivered once and counted in `totals.duplicates`. The chance of two different URLs colliding is about 1 in 4.5 quadrillion (2^52) per pair. - A sitemap hosted on another host than the site is fetched only when the `robots.txt` of the site declares it (the sitemaps protocol allows that); a sitemap of the site that redirects to another host counts as the site's own pointer (its children on that host are read too); one that is merely listed by a sitemap index, or guessed, is listed with `different_host` and not fetched. - At most 2,000,000 URLs per site are remembered by `onlyNew`. - The URLs are not requested: this Actor does not know whether they answer 200, redirect or are gone. - No JavaScript, no browser: only the files above. If a site hides its sitemap behind a bot challenge, the status is `blocked` (free; `partial` when other files could be read) and nothing more is attempted: the `User-Agent` never changes. Such a site also gets `botWall: true` on the `robots.txt` and `llms.txt` parts when they were refused, and `llmsTxtFound` is then `false`, never a file that was not served. Big sites behind an anti-bot service (online shops, film databases, news sites) are often like this. - `llms.txt` is checked at the root only; `llms-full.txt` size is counted up to 10 MiB. - Thirty seconds of silence on a download is a timeout; one retry on a dropped connection or a 5xx error (not 503). ### Run it **API (cURL)**, synchronous, returns the dataset items: ```bash curl -X POST "https://api.apify.com/v2/acts/far003~sitemap-url-extractor-validator/run-sync-get-dataset-items?token=$APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{"urls": ["example.com"], "maxUrlsPerSite": 1000, "onlyNew": true}' ``` **Python** ```python from apify_client import ApifyClient client = ApifyClient("") run = client.actor("far003/sitemap-url-extractor-validator").call(run_input={ "urls": ["example.com", "https://example.org/sitemap.xml"], "includePatterns": ["/blog/**"], "onlyNew": True, }) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): if item["type"] == "site": print(item["site"], item["status"], item["score"], item["totals"]["urlsFound"]) else: print(item["url"], item["lastmod"], item["aiBotsBlocked"]) ``` **n8n**: add an *HTTP Request* node, method `POST`, URL `https://api.apify.com/v2/acts/far003~sitemap-url-extractor-validator/run-sync-get-dataset-items`, query parameter `token`, JSON body `{"urls": ["example.com"], "onlyNew": true}`. Each URL and each site report arrives as one n8n item; filter on `{{ $json.type }}`. Schedule it with a *Schedule Trigger* and use `onlyNew` to receive only what changed. ### Good to know - The run writes a `SUMMARY` record in the key-value store (totals, statuses, sites not started) every few sites and also if it dies on an unexpected error. Entries that are the same site as an earlier one (`www.x.com` and `x.com`, a trailing slash) are read once: `totals.duplicatesIgnored` counts them and `duplicates` lists them (`input` and the earlier entry it equals, at most 100; `duplicatesListTruncated` says if there were more). - A user name and password inside an address (`https://user:pw@example.com/`) are never used: they are removed from `input`, `site`, the `url` of every record, the errors and the log, and the site report carries a warning. - A run that the platform restarts (migration) does not deliver or charge twice what is already in the dataset. - The Actor stops 60 seconds before the run timeout so the summary and the memory are written. # Actor input Schema ## `urls` (type: `array`): Domains (example.com), site URLs (https://example.com/) or the URL of a sitemap file (https://example.com/sitemap.xml). One site report per entry. Entries that are not web addresses, or point to private/local addresses, are reported as "invalid" (free), not as a run error. ## `extractUrls` (type: `boolean`): Deliver one record per URL found in the sitemaps. Turn off to get only the site reports (sitemap files, robots.txt, llms.txt, score): a sample of up to 50 sitemap files per site is read and validated (see Maximum sitemap files per site); the URLs are not delivered or charged. ## `maxUrlsPerSite` (type: `integer`): Stop reading a site after this many URL records were delivered. The site report then says that the count of URLs found is a lower bound. ## `maxSitemapsPerSite` (type: `integer`): How many sitemap files of one site are read. 0 = automatic: up to 5,000 when the URLs are extracted, a sample of 50 files when "Extract URLs" is off (the cheap site-report mode). When the limit is reached the report says "N of M sitemap files checked", is marked incomplete, and loses no score points. ## `includePatterns` (type: `array`): Only deliver URLs matching one of these patterns. `*` matches within one path segment, `**` matches anything. A pattern starting with `/` is compared with the path and query (`/blog/**`), any other with the whole URL (`https://example.com/blog/*`). Empty = all URLs. ## `excludePatterns` (type: `array`): Do not deliver URLs matching one of these patterns (same syntax as the include patterns). Exclusion wins over inclusion. Filtered URLs are still counted and checked in the site report. ## `modifiedAfter` (type: `string`): A date (2026-01-31) or a number of days ("30 days"). URLs with a lastmod before it, and URLs without a valid lastmod, are not delivered. ## `onlyNew` (type: `boolean`): Deliver only URLs that are new, or whose lastmod changed, since the previous run with this option. The first run delivers everything as new. Only URLs that were actually delivered are remembered: filters and limits never make the Actor forget or invent anything. ## `checkRobotsTxt` (type: `boolean`): Read and validate robots.txt (RFC 9309), build the table for AI bots, and mark every URL with robotsAllowed and aiBotsBlocked. ## `checkLlmsTxt` (type: `boolean`): Read /llms.txt (and look for /llms-full.txt) and validate the structure against the llmstxt.org proposal. ## `extraUserAgents` (type: `array`): Bot names (user-agent tokens, e.g. MyCompanyBot) to add to the table of AI bots, in the site report and in aiBotsBlocked. ## `respectRobotsTxt` (type: `boolean`): Do not fetch a sitemap or llms.txt that robots.txt forbids to this Actor's user-agent (SitemapRobotsLlmsBot). If robots.txt answers with a server error, RFC 9309 says to assume everything is disallowed. ## `requestsPerSecond` (type: `integer`): Between 1 and 5. A Crawl-delay in robots.txt that is slower than this wins (capped at 10 seconds). ## `maxConcurrency` (type: `integer`): How many sites are read at the same time. ## `timeoutSecs` (type: `integer`): Maximum wait for the response headers, and maximum silence while a file is downloading. ## `maxSitemapMegabytes` (type: `integer`): A sitemap larger than this is read up to the limit and reported as truncated_at_limit. The sitemap standard allows 50 MB. ## Actor input object example ```json { "urls": [ "https://www.sitemaps.org", "https://llmstxt.org" ], "extractUrls": true, "maxUrlsPerSite": 0, "maxSitemapsPerSite": 0, "includePatterns": [], "excludePatterns": [], "onlyNew": false, "checkRobotsTxt": true, "checkLlmsTxt": true, "extraUserAgents": [], "respectRobotsTxt": true, "requestsPerSecond": 2, "maxConcurrency": 5, "timeoutSecs": 30, "maxSitemapMegabytes": 50 } ``` # Actor output Schema ## `urls` (type: `string`): No description ## `sites` (type: `string`): No description ## `summary` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "urls": [ "https://www.sitemaps.org", "https://llmstxt.org" ], "includePatterns": [], "excludePatterns": [], "extraUserAgents": [] }; // Run the Actor and wait for it to finish const run = await client.actor("far003/sitemap-url-extractor-validator").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "urls": [ "https://www.sitemaps.org", "https://llmstxt.org", ], "includePatterns": [], "excludePatterns": [], "extraUserAgents": [], } # Run the Actor and wait for it to finish run = client.actor("far003/sitemap-url-extractor-validator").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "urls": [ "https://www.sitemaps.org", "https://llmstxt.org" ], "includePatterns": [], "excludePatterns": [], "extraUserAgents": [] }' | apify call far003/sitemap-url-extractor-validator --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,far003/sitemap-url-extractor-validator" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/Qvld7MdOethaSRmZe/builds/5twEO02YpX1EeZYvY/openapi.json