Sitemap URL Extractor + Robots.txt & llms.txt Validator

Reads the sitemaps of a website and hands you every URL in them, one record each, with dates, images, videos, news and language alternates. In the same pass it checks the sitemap files themselves (format, limits, errors), robots.txt (RFC 9309, with a table of which AI bots are allowed or blocked) and the structure of llms.txt. The sitemap is read as a stream, so memory stays flat whether the file has 1,000 or 1,000,000 URLs, and a site that fails never stops the others.

Data source: only files that the site itself publishes for robots (robots.txt, sitemaps, llms.txt, llms-full.txt). Nothing else on the site is fetched, no JavaScript is run, and the URLs found are not requested to see whether they answer.

What you get

Two kinds of records in the same dataset, told apart by the field type:

• url: one per URL found in a sitemap: url, normalized lastmod, changefreq, priority, the sitemap file it came from, images, videos, news, hreflang alternates, whether robots.txt allows the path (robotsAllowed), which AI bots it bars from it (aiBotsBlocked), what is wrong with it (issues) and, with Only new or changed URLs, whether it is new or updated.
• site: exactly one per entry of your input, always delivered, even when everything fails: status, the list of sitemap files read (type, HTTP status, compressed or not, size, number of URLs, problems), the robots.txt report with the AI-bot table, the llms.txt report, totals, a 0-100 score with its breakdown, warnings.

In the Output tab you find two views: URLs and Site reports. The views do not filter rows by type: the URLs view shows the site rows with empty cells and the other way round. To get only one kind, filter on type after reading (in n8n an IF/Filter node on {{ $json.type }}; in Python item["type"]), and through the API pass fields= with the columns you want (for example fields=type,site,status,score,urlsFound,aiBotsBlockedList). The site record has flat fields for a spreadsheet (urlsFound, urlsDelivered, robotsTxtFound, robotsTxtIssueCount, aiBotsBlockedCount, aiBotsBlockedList, aiBotsRestrictedList, aiBotsNamedList, llmsTxtFound, llmsTxtValid, sitemapIssueCount, urlIssueCount) next to the detailed objects, and the Site reports view shows them first.

Example url record (real run, llmstxt.org)

{
  "type": "url",
  "site": "https://llmstxt.org",
  "url": "https://llmstxt.org/core.html",
  "lastmod": "2026-09-24T18:20:59.055Z",
  "lastmodRaw": "2026-09-24T18:20:59.055Z",
  "changefreq": null,
  "priority": null,
  "sitemapUrl": "https://llmstxt.org/sitemap.xml",
  "sourceFormat": "xml",
  "images": [],
  "imageCount": 0,
  "videos": [],
  "videoCount": 0,
  "news": null,
  "alternates": [],
  "robotsAllowed": true,
  "aiBotsBlocked": [],
  "change": "new",
  "issues": [],
  "checkedAt": "2026-10-13T10:11:15.640Z"
}

Example site record (real run, llmstxt.org)

The same run; only the aiBots table is shortened here (the real record lists 24 bots).

{
  "type": "site",
  "site": "https://llmstxt.org",
  "input": "llmstxt.org",
  "status": "ok",
  "error": null,
  "robotsTxt": {
    "url": "https://llmstxt.org/robots.txt",
    "httpStatus": 200,
    "found": true,
    "botWall": false,
    "sizeBytes": 41,
    "interpretation": "parsed",
    "note": null,
    "groups": 0,
    "sitemaps": [
      "https://llmstxt.org/sitemap.xml"
    ],
    "crawlDelay": null,
    "issues": [],
    "syntaxErrorLines": [],
    "unknownDirectives": [],
    "aiBots": {
      "GPTBot": {
        "rule": "allowed",
        "matchedGroup": "none",
        "explicit": false,
        "blockedPathCount": 0
      },
      "ClaudeBot": {
        "rule": "allowed",
        "matchedGroup": "none",
        "explicit": false,
        "blockedPathCount": 0
      },
      "Googlebot": {
        "rule": "allowed",
        "matchedGroup": "none",
        "explicit": false,
        "blockedPathCount": 0
      }
    }
  },
  "llmsTxt": {
    "url": "https://llmstxt.org/llms.txt",
    "httpStatus": 200,
    "found": true,
    "botWall": false,
    "sizeBytes": 637,
    "valid": true,
    "linksWellFormed": true,
    "title": "llms.txt",
    "summary": "A proposal that those interested in providing LLM-friendly content add a /llms.txt file to their site. This is a markdown file that provides brief background information and guidance, along with links to markdown files providing more detailed information.",
    "sections": [
      {
        "title": "Docs",
        "linkCount": 3
      }
    ],
    "sectionCount": 1,
    "linkCount": 3,
    "hasOptionalSection": false,
    "llmsFullTxt": {
      "found": false,
      "sizeBytes": 0
    },
    "issues": [],
    "malformedLines": [],
    "note": null
  },
  "sitemaps": [
    {
      "url": "https://llmstxt.org/sitemap.xml",
      "type": "urlset",
      "httpStatus": 200,
      "compressed": true,
      "sizeBytes": 880,
      "urlCount": 7,
      "lastmod": null,
      "crossHost": false,
      "issues": [],
      "detail": null,
      "errorLine": null
    }
  ],
  "sitemapCount": 1,
  "totals": {
    "urlsFound": 7,
    "urlsDelivered": 7,
    "duplicates": 0,
    "filteredOut": 0,
    "heldByLimit": 0,
    "unchanged": 0,
    "resumedUrls": 0,
    "newUrls": 7,
    "updatedUrls": 0,
    "removedSinceLastRun": 0,
    "urlsWithIssues": 0,
    "urlsWithRemarks": 0,
    "issuesByCode": {},
    "sitemapIssuesByCode": {}
  },
  "complete": true,
  "score": 100,
  "scoreIsPartial": false,
  "scoreChecks": [
    "robots",
    "sitemaps",
    "llms"
  ],
  "scoreBreakdown": [
    {
      "component": "robots_txt_present_and_clean",
      "points": 20,
      "max": 20,
      "note": "robots.txt found, no problem"
    },
    {
      "component": "robots_txt_declares_sitemaps",
      "points": 10,
      "max": 10,
      "note": "every declared sitemap was read"
    },
    {
      "component": "sitemap_found_and_parsed",
      "points": 15,
      "max": 15,
      "note": "at least one sitemap file was parsed"
    },
    {
      "component": "sitemap_files_clean",
      "points": 20,
      "max": 20,
      "note": "1 of 1 sitemap files without problems"
    },
    {
      "component": "urls_without_issues",
      "points": 20,
      "max": 20,
      "note": "7 of 7 URLs without problems"
    },
    {
      "component": "urls_with_valid_lastmod",
      "points": 5,
      "max": 5,
      "note": "7 of 7 URLs have a valid lastmod"
    },
    {
      "component": "llms_txt",
      "points": 10,
      "max": 10,
      "note": "valid llms.txt, no remark"
    }
  ],
  "warnings": [],
  "charged": true,
  "checkedAt": "2026-10-13T10:11:13.629Z",
  "urlsFound": 7,
  "urlsDelivered": 7,
  "robotsTxtFound": true,
  "robotsTxtIssueCount": 0,
  "aiBotsBlockedCount": 0,
  "aiBotsBlockedList": [],
  "aiBotsRestrictedList": [],
  "aiBotsNamedList": [],
  "llmsTxtFound": true,
  "llmsTxtValid": true,
  "sitemapIssueCount": 0,
  "urlIssueCount": 0
}

Input

Field	Default	What it does
urls	required	Domains (example.com), site URLs (https://example.com/) or the URL of a sitemap file. One site report per entry. Private, local or malformed entries come back as invalid (free).
extractUrls	true	Deliver the URL records. Off = site reports only (a sample of up to 50 sitemap files per site is read and checked, see maxSitemapsPerSite; the URLs are not delivered or charged).
maxUrlsPerSite	0 (no limit)	Stop a site after this many URL records.
maxSitemapsPerSite	0 (automatic, max 5,000)	How many sitemap files of one site are read. Automatic = 5,000 when the URLs are extracted and 50 when extractUrls is off (the site-report mode reads a declared sample: the report says "N of M sitemap files checked (audit sample)", complete is false, no score points are lost). Set a number to choose your own limit.
includePatterns / excludePatterns	empty	Glob patterns on the URL: * stays inside one path segment, ** crosses segments. A pattern starting with / is matched on path and query. Exclusion wins.
modifiedAfter	none	A date (2026-01-31) or 30 days. URLs older than that, or without a valid lastmod, are not delivered.
onlyNew	false	Deliver only URLs that are new or whose lastmod changed since the previous run with this option (see below).
checkRobotsTxt	true	Read and validate robots.txt, build the bot table, fill robotsAllowed / aiBotsBlocked.
checkLlmsTxt	true	Read and validate /llms.txt, look for /llms-full.txt.
extraUserAgents	empty	Bot names to add to the table (for example your own crawler).
respectRobotsTxt	true	Do not fetch a sitemap or llms.txt that robots.txt forbids to this Actor's user-agent (SitemapRobotsLlmsBot). If robots.txt answers with a server error, RFC 9309 says to assume everything is disallowed.
requestsPerSecond	2 (1-5)	Per host. A Crawl-delay in robots.txt that is slower wins, capped at 10 seconds.
maxConcurrency	5	Sites read at the same time.
timeoutSecs	30	Maximum wait for the response headers and maximum silence while a file downloads.
maxSitemapMegabytes	50	Uncompressed size read from one sitemap; a bigger file is cut there and flagged truncated_at_limit.

A wrong option (a word where a number is expected, requestsPerSecond: 9, a bad date) stops the run before any request or charge, with the reason in the error message.

How sitemaps are found

1. Sitemap: lines of robots.txt. A line may point to another host (robots.txt of www.example.com declaring http://www.example.cn/sitemap.xml): it is read, with crossHost: true and the informational remark cross_host_declared_in_robots (not a failure, no points lost); the robots.txt of that host is read first and obeyed (when respectRobotsTxt is on), with its Crawl-delay; if that file is an index, its children are followed while they stay on that host. URLs inside it are not flagged different_host when they are on the site's host or on the host of that sitemap.
2. If there are none: /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /wp-sitemap.xml, /sitemap.xml.gz, /sitemap.txt, in this order, stopping at the first one that is a sitemap.
3. If your input is already the URL of a sitemap, the read starts there (robots.txt and llms.txt are still read from its origin).

Sitemap indexes are followed up to 5 levels deep, with loop protection. Formats: urlset, sitemapindex, RSS 2.0, Atom, plain text (one URL per line); gzip is recognised from the first bytes, not from the file name. Extensions: image:image, video:video, news:news, xhtml:link (hreflang). The sitemap elements are recognised by their name, in any namespace (or none), and the extensions with tolerance (http or https, trailing slash, any version, or the conventional prefix image: video: news: xhtml: whatever the URI is): a file whose namespace is not the standard one is still read, and says so with missing_namespace, legacy_namespace or nonstandard_namespace (a remark that costs a few score points, not a failure).

If a domain redirects to another host (for example docs.example.com to platform.example.com), the host that robots.txt finally comes from is treated as part of the site. Likewise, when a sitemap of the site redirects to another host (for example webflow.com/sitemap.xml to a CDN), the children of that index are followed on the host it was served from.

Every request obeys the robots.txt of the host it goes to (when respectRobotsTxt is on): the bare domain and www., http:// and https://, a sitemap or llms.txt on another host, the host a redirect leads to. Each of those robots.txt files is read once, before the first request to that host; if one cannot be fetched (server error, no answer), RFC 9309 says everything there is disallowed. The fields robotsAllowed and aiBotsBlocked of a URL use the robots.txt of the URL's own host when that file was loaded in this run (because something was read from that host), otherwise the robots.txt of the site.

What you pay and what you don't

Pay-per-event, one event: Result. $0.40 per 1,000 results (that is $0.0004 per result). Platform usage is included in the price.

Delivered record	Charged?
every url record	yes, 1 result
site record, status ok, partial or no_sitemap, when at least one of robots.txt, llms.txt or a sitemap was actually read	yes, 1 result
site record with status blocked (HTTP 401/403/429/503, an empty 202 or a challenge page, or robots.txt forbids it), unreachable, invalid	free
site record where nothing was read	free
site record delivered after the spending limit was reached	free, with a warning
URLs already delivered before a restart of the run (migration)	not delivered or charged again
URLs left out by filters, duplicates, unchanged URLs with onlyNew	not delivered, not charged

Example: one site with 2,000 URLs in its sitemaps = 2,000 URL results + 1 site report = 2,001 results = $0.8004. The same site with extractUrls off = 1 result = $0.0004. A domain that does not exist = $0.

The spending limit you set on the run is respected to the single result: when it is reached the Actor stops cleanly (spending limit reached), the URLs found but not delivered are counted in totals.heldByLimit, and they are not remembered by onlyNew, so the next run delivers them.

The Actor does not try to get around blocks: one honest user-agent that names this bot, no proxies, no disguise.

Only new or changed URLs

With onlyNew, the Actor keeps in a named key-value store (sitemap-robots-llms-state), for each site, a compact map from a 52-bit hash of the URL to its lastmod. The first run delivers everything as new. Later runs deliver a URL only if it is not in the map (new) or its lastmod differs (updated).

• Only what was delivered is remembered. Filters, maxUrlsPerSite, modifiedAfter and the spending limit never write into the memory, so a URL you filtered out is simply new the day you stop filtering.
• removedSinceLastRun (URLs remembered but no longer in the sitemaps) is computed only when the whole site was read, with no filter and no limit; otherwise it is null and a warning says why. A site that could not be read keeps its memory untouched.
• Limit: at most 2,000,000 URLs are remembered per site.

Status of a site report

status	Meaning
ok	Sitemaps found and read without a read failure.
partial	Something could not be read (a listed sitemap answered an error, a file broke half way, was cut at the size limit), or the run stopped before the end (spending limit, timeout). The report says what.
no_sitemap	The host answers but no sitemap was found; robots.txt / llms.txt are still reported.
blocked	The server answered 401/403/429/503, an empty 202 (a typical WAF challenge) or a block page instead of the files, and no sitemap could be read; the record warns that a browser may see what we do not. Or robots.txt forbids the sitemaps to this bot or answered with a server error (RFC 9309). Free.
unreachable	The host did not answer (DNS, connection, timeout, certificate). Free.
invalid	The entry is not a usable web address, or points to a private or local address. Free.

How the score is made

100 points, seven parts, nothing hidden (scoreBreakdown in the record shows each one):

Part	Max	Points
robots_txt_present_and_clean	20	20 found and without problems · 12 found with problems · 0 not found
robots_txt_declares_sitemaps	10	10 every declared sitemap was read · 5 some could not be read · 0 no Sitemap: line
sitemap_found_and_parsed	15	15 if at least one sitemap file was parsed
sitemap_files_clean	20	20 × (files without problems ÷ files known)
urls_without_issues	20	20 × (URLs without issues ÷ URLs found)
urls_with_valid_lastmod	5	5 × (URLs with a valid lastmod ÷ URLs found)
llms_txt	10	10 valid and without remarks · 7 valid with remarks · 3 found but not valid · 0 not found

llms.txt is not a mandatory standard, so its absence costs at most 10 points of 100. A remark of kind cross_host_declared_in_robots is informational and costs nothing; missing_namespace, legacy_namespace and nonstandard_namespace cost a few points through sitemap_files_clean.

Comparable scores. The record field scoreChecks lists the checks the score is made of (robots, sitemaps, llms). If checkRobotsTxt or checkLlmsTxt is off, that part leaves the total, the rest is scaled to 100 (points earned over points possible) and a warning says so: compare scores only between runs with the same scoreChecks. The score is null (and scoreChecks empty) when nothing was read and when the status is blocked (what was not read is not a defect of the site). It describes the health of the files the site publishes for robots; it is not a measure of anything else.

When a limit stopped the reading (maxUrlsPerSite, the spending limit, the run timeout, the 5,000-file cap, maxSitemapsPerSite, maxSitemapMegabytes) the record says so: complete is false and scoreIsPartial is true, and a warning calls the score provisional. A limit never takes points away: the parts that depend on files are computed in proportion to the files and URLs that were read, so the same clean site gets the same score read in full or in part (the notes say "3 more not opened, not judged"). What was not read is not judged; only what was read can lose points. A site that was not read at all has score: null: status blocked, unreachable or invalid, a run that stopped before the first sitemap was read, or a site that was never started. A site with a robots.txt and really no sitemap (no_sitemap) keeps its low score. complete is true only when the status is ok and nothing cut the reading short.

robots.txt report

Parsed under RFC 9309: groups, longest-match rule with Allow winning ties, * and $ in paths. The file counts as found only with HTTP 200 and a non-empty body (an empty 200, 202, 204 and redirects that lead nowhere are found: false, with the status in note; 401/403/429/202 or a challenge page also set botWall: true). The answer is interpreted as the RFC says: 200 is parsed, 4xx means no restrictions, 5xx or no answer means everything is disallowed (and the report says so in interpretation). Problems reported: syntax_error (with line numbers, at most 20), unknown_directive, too_large (over 500 KiB), bom, html_instead_of_text, no_user_agent_before_rule, sitemap_not_absolute, empty_file.

aiBots has one entry per bot: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, anthropic-ai, Google-Extended, GoogleOther, CCBot, PerplexityBot, Perplexity-User, Bytespider, Amazonbot, Applebot-Extended, meta-externalagent, FacebookBot, cohere-ai, DuckAssistBot, MistralAI-User, Diffbot, YouBot, plus Googlebot and Bingbot as a reference (list compiled 2026-10-12; your extraUserAgents are added). For each bot:

• rule, one of four values:
  • allowed: no Disallow rule applies;
  • blocked: the root / is closed and nothing opens it again (Disallow: / without any Allow), whether the site says it in a group naming the bot or in the * group the bot follows;
  • restricted: the site wrote rules for this bot (a group that names it) and they do not close everything;
  • default_rules: the bot has no group of its own and follows the * group, which closes some paths (the ordinary Disallow: /wp-admin/ of a WordPress site lands here, for every bot, Googlebot included);
• explicit: true when the robots.txt names the bot in a group of its own; matchedGroup: explicit, wildcard (only the * group applies) or none;
• blockedPathCount: number of non-empty Disallow rules that apply to the bot.

In the site report the flat fields give the same answer for a spreadsheet: aiBotsBlockedList (rule blocked), aiBotsRestrictedList (rule restricted) and aiBotsNamedList (every bot with explicit: true, whatever its rule: the bots the site cares about by name).

Example: which sites let which AI bots in. Run with extractUrls: false (site reports only) over a list of domains, then in Python:

for item in items:                      # items = dataset items
    if item["type"] == "site" and item["robotsTxtFound"]:
        print(item["site"], "closed to:", item["aiBotsBlockedList"], "| rules just for:", item["aiBotsRestrictedList"],
              "| named in robots.txt:", item["aiBotsNamedList"])

aiBotsBlockedList is the list to act on; default_rules (a bot that only follows the * group) is not in any of the three lists on purpose: it means the site said nothing about that bot.

llms.txt report

Checked against the proposal at https://llmstxt.org/ (read on 2026-10-12): an optional BOM, one H1 (the only required part), an optional blockquote summary, optional text, then H2 sections that are lists of - [name](url): notes, with an Optional section by convention. The report gives title, summary, sections with their link counts, hasOptionalSection, and whether /llms-full.txt exists.

valid is true when the file was found and its structure is what the proposal requires: one H1, and it comes first. It is false for a missing (missing_h1), repeated (multiple_h1) or late (h1_not_first) H1, an empty file (empty_file), an HTML page served for a missing file (html_instead_of_markdown) or a file over 1 MiB (too_large). Everything else is a remark that costs a few score points and does not make the file invalid: no_summary_blockquote, no_sections, section_without_links, relative_link, wrong_content_type and malformed_link_line (a list item that starts like a link but is not [name](url), or is a bare URL; [name](url): note, [name](url) - note and [name](url) note are all well formed). The field linksWellFormed says whether any list item is malformed (false when at least one is). Only the root /llms.txt is checked. The check is about structure, not about whether the content is useful.

Sitemap file problems

http_error, bot_wall_suspected (HTTP 401/403/429, an empty 202 or a challenge page instead of the file: the site is answering our honest bot with a block page, so the file may be fine for a browser), not_xml, invalid_line (a plain-text sitemap with lines that are not URLs: blank lines and # comments are ignored, the bad lines are counted in detail and left out, the rest is read), unescaped_ampersand (a bare & in the file, for example in a <loc> with ?a=1&b=2: not valid XML, but common; it is read as & in the stream so no URL after it is lost, and detail gives the count), encoding_error (bytes that are not valid in the encoding in use: the file is read all the same, with U+FFFD in the affected URLs; the encoding is taken from the byte order mark, then the <?xml encoding="..."?> declaration, then the charset of the Content-Type, else UTF-8 — UTF-16, ISO-8859-1 and Windows-1252 are read correctly), html_instead_of_xml (a soft 404), wrong_root_element, missing_namespace, legacy_namespace (the old http://www.google.com/schemas/sitemap/0.84), nonstandard_namespace (another spelling of the sitemap namespace, or of an image/video/news/xhtml extension namespace: the file is read all the same), nonstandard_element_name (the root is written with another case, for example <sitemapIndex>: read all the same), too_many_urls (over 50,000), too_large (over 50 MB uncompressed), truncated_at_limit, index_inside_index_too_deep, loop, empty, xml_error (with the line), gzip_error, listed_in_robots_but_missing (declared in robots.txt, answers 404/410), robots_disallowed, different_host (a sitemap on another host that robots.txt does not declare), cross_host_declared_in_robots (informational: a sitemap on another host declared in robots.txt; read, not a defect). A file that breaks half way still gives you the URLs read before the break.

URL problems: different_host (another domain than the site's), different_subdomain (another subdomain of the same registrable domain, for example blog.example.com in the sitemap of example.com: a light remark, kept in issues and in issuesByCode, not counted in urlsWithIssues or urlIssueCount, and weighing a quarter of a problem in the score; the domain is found with the public suffix list), different_scheme, not_absolute, invalid_lastmod, future_lastmod, invalid_priority, invalid_changefreq, url_too_long (over 2,048 characters), blocked_by_robots. lastmod accepts the W3C forms (year, year-month, date, date and time with or without a zone; no zone means UTC); 0000-00-00 and anything else is invalid_lastmod with lastmod: null.

Limits, stated plainly

• Sitemap indexes: 5 levels; at most 5,000 sitemap files per site (the sitemaps list in the record shows the first 500, sitemapCount is exact).
• A sitemap file is read up to maxSitemapMegabytes uncompressed (default 50 MB); the standard allows 50,000 URLs per file and flags more.
• Duplicates are detected per site with a 52-bit hash of the URL (not the full string, to keep memory flat); a URL listed twice is delivered once and counted in totals.duplicates. The chance of two different URLs colliding is about 1 in 4.5 quadrillion (2^52) per pair.
• A sitemap hosted on another host than the site is fetched only when the robots.txt of the site declares it (the sitemaps protocol allows that); a sitemap of the site that redirects to another host counts as the site's own pointer (its children on that host are read too); one that is merely listed by a sitemap index, or guessed, is listed with different_host and not fetched.
• At most 2,000,000 URLs per site are remembered by onlyNew.
• The URLs are not requested: this Actor does not know whether they answer 200, redirect or are gone.
• No JavaScript, no browser: only the files above. If a site hides its sitemap behind a bot challenge, the status is blocked (free; partial when other files could be read) and nothing more is attempted: the User-Agent never changes. Such a site also gets botWall: true on the robots.txt and llms.txt parts when they were refused, and llmsTxtFound is then false, never a file that was not served. Big sites behind an anti-bot service (online shops, film databases, news sites) are often like this.
• llms.txt is checked at the root only; llms-full.txt size is counted up to 10 MiB.
• Thirty seconds of silence on a download is a timeout; one retry on a dropped connection or a 5xx error (not 503).

Run it

API (cURL), synchronous, returns the dataset items:

curl -X POST "https://api.apify.com/v2/acts/far003~sitemap-url-extractor-validator/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["example.com"], "maxUrlsPerSite": 1000, "onlyNew": true}'

Python

from apify_client import ApifyClient

client = ApifyClient("<APIFY_TOKEN>")
run = client.actor("far003/sitemap-url-extractor-validator").call(run_input={
    "urls": ["example.com", "https://example.org/sitemap.xml"],
    "includePatterns": ["/blog/**"],
    "onlyNew": True,
})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    if item["type"] == "site":
        print(item["site"], item["status"], item["score"], item["totals"]["urlsFound"])
    else:
        print(item["url"], item["lastmod"], item["aiBotsBlocked"])

n8n: add an HTTP Request node, method POST, URL https://api.apify.com/v2/acts/far003~sitemap-url-extractor-validator/run-sync-get-dataset-items, query parameter token, JSON body {"urls": ["example.com"], "onlyNew": true}. Each URL and each site report arrives as one n8n item; filter on {{ $json.type }}. Schedule it with a Schedule Trigger and use onlyNew to receive only what changed.

Good to know

• The run writes a SUMMARY record in the key-value store (totals, statuses, sites not started) every few sites and also if it dies on an unexpected error. Entries that are the same site as an earlier one (www.x.com and x.com, a trailing slash) are read once: totals.duplicatesIgnored counts them and duplicates lists them (input and the earlier entry it equals, at most 100; duplicatesListTruncated says if there were more).
• A user name and password inside an address (https://user:pw@example.com/) are never used: they are removed from input, site, the url of every record, the errors and the log, and the site report carries a warning.
• A run that the platform restarts (migration) does not deliver or charge twice what is already in the dataset.
• The Actor stops 60 seconds before the run timeout so the summary and the memory are written.