On-page SEO & Schema Extractor + Sitemap Validator

Developer: Luxe_tasks.

Audit any public website (your own site, or a client's or competitor's public pages): for each page listed in its sitemaps (or in your own list) get one row with the HTTP status, redirects, title, meta description, meta robots and X-Robots-Tag, canonical, hreflang, html lang, viewport, h1/h2 counts, Open Graph and Twitter tag names, JSON-LD @types and microdata types, plus issue codes (missing title, duplicate descriptions, canonical pointing elsewhere, noindex pages listed in the sitemap, ...). The sitemaps themselves are validated against the sitemaps.org protocol: every entry gets a row with its codes (bad lastmod, URLs on another host, duplicates, fragments, ...), and every file gets its findings (size, UTF-8, namespace, nested indexes, missing from robots.txt, ...).

It reads static HTML only (no browser, no JavaScript rendering), follows no links and never leaves the one site you name.

Which tool to use: for a Lighthouse SEO score per page (rendered in a browser), use our Bulk Website Auditor; to find broken links, use our Broken Link Checker. This Actor gives the raw per-field values, structured-data types and sitemap validation for a whole site in one static pass.

How to use it

1. Site or sitemap URL: the home page (https://example.com/) or one sitemap file (https://example.com/sitemap.xml). The example input is https://crawlee.dev/ with small test limits (5 pages, 50 sitemap URLs, 2 sitemap files); change them for a real audit.
2. Optional: Sitemap URLs (otherwise the Sitemap: lines of robots.txt, else /sitemap.xml), Page URLs to audit (exactly these pages instead of the sitemap's), Max pages (default 500, at most 5,000).
3. Run it. Audited pages are in the dataset (Audited pages view); the Output tab links the sitemap rows, the sitemap file findings, the SITE_SUMMARY record and the pages that were not audited.

What it requests

1. /robots.txt of the site (once per run).
2. The sitemap files: the ones you list, else the same-site Sitemap: lines of robots.txt, else /sitemap.xml; children of a sitemap index on the same site (one level, as the protocol allows). gzip sitemaps (.xml.gz) and text sitemaps are read.
3. Each page to audit, once (GET), following at most 5 redirects on the same site. A redirect to another site is recorded and not requested.

Nothing else: no links are followed, no images, scripts or styles are loaded, no forms are submitted.

Input

Field	Default	Notes
siteUrl	(required)	One public site (protocol + host) per run. Example input: https://crawlee.dev/.
sitemapUrls	[]	Up to 20, same site.
pageUrls	[]	Up to 5,000, same site; replaces the sitemap's page list.
auditPages	true	false = validate the sitemaps only.
maxPages	500	1 to 5,000 (this Actor's limit). The example input uses 5.
validateSitemapUrls	true	false = no sitemap URL rows (the sitemaps are still read to find the pages).
maxSitemapUrls	5,000	1 to 50,000 sitemap entries read in a run (50,000 is the protocol limit per file). The example input uses 50.
maxSitemapFiles	10	1 to 50 files per run. The example input uses 2.
timeoutSecs	15	Per request, 5 to 30 seconds.

Output

• Dataset, one row per audited page: url, finalUrl, httpStatus, redirectCount, redirectChain, redirectNotFollowed, contentType, responseTimeMs, tries, responseBytes, bodyTruncated, isHtml, xRobotsTag, xRobotsNoindex, inSitemap, sitemapLastmod, title, titleLength, titleCount, metaDescription, metaDescriptionLength, metaDescriptionCount, metaRobots, metaNoindex, metaNofollow, canonicalUrl, canonicalStatus (self, other, missing, multiple, invalid), canonicalCount, htmlLang, hasViewport, metaCharset, h1Count, h2Count, hreflang (up to 30: hreflang, href, valid), hreflangCount, hreflangInvalidCount, openGraphTags, twitterTags (tag names, not values), jsonLdBlocks, jsonLdTypes, jsonLdErrors, microdataTypes, loginForm, issues, issueCount. The dataset holds audited pages, plus at most one status row for a stopped site (below).
• Status row (dataset): url, status, reason, checkedAt only. status is one of ROBOTS_DISALLOWED, ROBOTS_UNREACHABLE, ACCESS_DENIED, CHALLENGE_DETECTED, OPTED_OUT, PRIVATE_ADDRESS_REFUSED, SITE_UNREACHABLE. If a site's robots.txt disallows the audit or can't be read, the site denies access or shows a challenge, or the site's owner has opted out, the Actor stops for that site and writes one status row with the URL, a status and a reason, and no page content; status rows are not billed events, but the $0.005 per-run start fee still applies.
• SITEMAP_URLS-0001, -0002, ... (key-value store, 1,000 rows each): one row per validated sitemap entry: loc, sitemap, codes, lastmod, changefreq, priority, valid.
• SITEMAP_FILES: one entry per sitemap file: url, source (input, robots, index, default), httpStatus, kind (urlset, sitemapindex, text), entries, gzipped, bytes, codes.
• SITE_SUMMARY: totals, the robots.txt result, site findings, counts per issue code and per HTTP status, duplicate titles and meta descriptions (top 50, with 5 example pages each).
• PAGE_ISSUES-0001, ...: pages that were not audited and why (robots.txt, access denied, a challenge, an error, the time or budget limit).
• URLs are stored with their query string replaced by ?[redacted]. Page text, images and tag values other than those listed above are never stored.
• The internal DISCOVERY record (used to resume a restarted run) keeps the page list as requested, query strings included, like your input.

Issue codes

• Page: HTTP_ERROR, REDIRECTED, REDIRECT_OFF_SITE, TOO_MANY_REDIRECTS, REDIRECT_NOT_FOLLOWED, NOINDEX_HEADER, NOINDEX_META, NOT_HTML, BODY_TRUNCATED, TITLE_MISSING, TITLE_TOO_LONG (over 60 characters), TITLE_TOO_SHORT (under 10), MULTIPLE_TITLES, META_DESCRIPTION_MISSING, META_DESCRIPTION_TOO_LONG (over 160), META_DESCRIPTION_TOO_SHORT (under 50), MULTIPLE_META_DESCRIPTIONS, H1_MISSING, MULTIPLE_H1, CANONICAL_MISSING, CANONICAL_POINTS_ELSEWHERE, CANONICAL_MULTIPLE, CANONICAL_INVALID, HTML_LANG_MISSING, VIEWPORT_MISSING, HREFLANG_INVALID, JSONLD_INVALID, OPEN_GRAPH_MISSING; for pages listed in a sitemap also SITEMAP_URL_REDIRECTS, SITEMAP_URL_NOT_200, SITEMAP_URL_NOINDEX, SITEMAP_URL_CANONICALISED. Site summary adds DUPLICATE_TITLE and DUPLICATE_META_DESCRIPTION. The length thresholds are common SEO guidance, not search-engine rules.
• Sitemap entry: LOC_MISSING, LOC_NOT_ENTITY_ESCAPED, LOC_TOO_LONG (over 2,048 characters), LOC_NOT_ABSOLUTE, LOC_BAD_SCHEME, LOC_HAS_WHITESPACE, LOC_HAS_FRAGMENT, LOC_OTHER_HOST, LOC_OUTSIDE_SITEMAP_PATH, DUPLICATE_LOC, LASTMOD_INVALID (not a W3C datetime), LASTMOD_IN_FUTURE, CHANGEFREQ_INVALID, PRIORITY_INVALID.
• Sitemap file: SITEMAP_NOT_FOUND, SITEMAP_HTTP_ERROR, SITEMAP_REDIRECTED, SITEMAP_REDIRECT_OFF_SITE, SITEMAP_TOO_MANY_REDIRECTS, SITEMAP_FETCH_FAILED, SITEMAP_TOO_LARGE (over 50 MB uncompressed), SITEMAP_BAD_GZIP, SITEMAP_EMPTY, SITEMAP_NOT_UTF8, SITEMAP_NOT_SITEMAP, SITEMAP_WRONG_NAMESPACE, SITEMAP_TOO_MANY_URLS (over 50,000), SITEMAP_INDEX_TOO_MANY_SITEMAPS, SITEMAP_INDEX_HAS_URL_ENTRIES, SITEMAP_URLSET_HAS_SITEMAP_ENTRIES, SITEMAP_INDEX_NESTED, SITEMAP_INDEX_OTHER_HOST, SITEMAP_INDEX_BAD_LOC, SITEMAP_NOT_IN_ROBOTS.
• Site: ROBOTS_ACCESS_DENIED, ROBOTS_CHALLENGE_DETECTED, PRIVATE_ADDRESS_REFUSED, ROBOTS_NO_SITEMAP_LINE, ROBOTS_SITEMAP_OTHER_HOST, ROBOTS_UNREACHABLE, ROBOTS_RATE_LIMITED, SITEMAP_FILES_NOT_FETCHED, SITEMAP_URL_LIMIT_REACHED, MAX_TOTAL_CHARGE_REACHED, DISCOVERY_RESUMED.

Limits

• One site per run; at most 5,000 pages per run (default 500) and 50,000 sitemap entries read per run (this Actor's limits). One sitemap file may be at most 50 MB uncompressed and 50,000 URLs (sitemaps.org protocol); files over 50 MB are reported, not parsed; files over 50,000 URLs are flagged (SITEMAP_TOO_MANY_URLS) and read only up to this run's limit.
• Pages are read up to 5 MB (this Actor's limit); larger pages are audited from their first 5 MB (BODY_TRUNCATED).
• Static HTML only: tags added by JavaScript are not seen.
• Run timeout 6,000 seconds (this Actor's setting). At 1 request per second, 5,000 pages take about 5,000 seconds, so they fit; with a slower site, a Crawl-delay over 1 second or after a 429, fewer pages fit. No request starts in the last 120 seconds; the pages left end as time_budget_reached in PAGE_ISSUES, without a per-page charge (the $0.005 per-run start fee still applies).

Pricing

Pricing: $0.005 per page audited and $0.0005 per sitemap URL validated, plus the $0.005 start fee per run, all three as pay-per-event prices in Apify Console. Each run costs a $0.005 start fee plus $0.005 per page audited and $0.0005 per sitemap URL validated, and runs that return no results cost only the $0.005 start fee.

• Pay-per-event page-audited: $0.005 for each page row saved to the dataset, plus the $0.005 start fee per run. A page counts as audited when the site answered it (any HTTP status except access denied, challenges and the errors below, including redirects and 404s, which are SEO findings); each page is charged once per run, after the result is recorded, on top of the $0.005 per-run start fee.
• Pay-per-event sitemap-url-validated: $0.0005 for each sitemap entry row saved (SITEMAP_URLS-*), plus the $0.005 start fee per run; charged after each block of rows is saved.
• Pages that were not audited (robots.txt disallows them, the site refused or rate-limited the run, server or network errors, the time or budget limit) have no page-audited charge; the $0.005 per-run start fee still applies.
• A maximum total charge set on the run is respected: sitemap rows and pages are only saved while the budget covers them; the rest are listed as not validated or budget_reached, without a per-page or per-URL charge (the $0.005 per-run start fee still applies).
• Order within a maximum total charge: sitemap URL rows are validated and charged while the sitemaps are read, before any page is audited, so a small maximum can be used up by sitemap rows and leave few or no page rows; set validateSitemapUrls to false to spend the budget on pages only (each run still pays the $0.005 start fee).
• What you pay even when nothing is audited: the $0.005 start fee per run, charged by Apify through its synthetic apify-actor-start event (Apify docs). This Actor's memory is fixed at 1 GB (1024 MB, this Actor's setting), so the $0.005 start fee is charged once per run.
• A run that is refused before any request (an invalid input or a site on the do-not-audit list) audits no pages and costs only the $0.005 per-run start fee.
• The synthetic apify-default-dataset-item event is not used (the Actor refuses to run if it is priced).

Politeness and safety

• robots.txt is honoured for every page (RFC 9309): a disallowed page is not requested. If robots.txt cannot be read (5xx, network error), or answers 401, 403 or 430, or shows a challenge page, nothing else is requested; a 404 robots.txt means no restrictions. robots.txt and sitemap files are always read, as the protocol intends. Crawl-delay is respected.
• One request at a time to the site, at least 1 second apart (this Actor's limit).
• A 429 or 503 is retried after its Retry-After time (seconds or a date, up to 60 seconds); without one, after 10 and then 20 seconds; after 3 tries the page is given up. After any 429 the Actor waits at least 5 seconds between requests for the rest of the run (this Actor's settings). A page still answered 429 after 3 tries stops the run for the site; so do 3 pages with server errors.
• A 401, 403 or 430 answer or a challenge/CAPTCHA page (on any HTTP status, including 405 and 503; a challenge is not retried) stops the run for the site at once: a challenge is never solved, and the pages already audited are kept.
• Honest fixed User-Agent SeoSitemapValidatorActor/0.1.0 (+https://apify.com/luxe_tasks/seo-sitemap-validator; honours robots.txt); no cookies, no logins, no proxies, no custom headers. Pages with a password field are flagged (loginForm), never filled.
• Only the one site is requested (redirects to other sites are not followed); private-network addresses are refused.

Your responsibility

You choose which site to audit. You are responsible for having the right to audit it, for following that site's terms of use and the law that applies to you, and for how you use the results, as set out in section 2 of the Apify Standard Actor Contract that governs this Actor. This Actor requests only pages that are public without a login, follows each site's robots.txt and Crawl-delay, sends one request at a time at most once per second, and stops at once if the site refuses access (401, 403, 430) or shows a challenge or CAPTCHA.

Prohibited uses: running several runs against the same site at the same time; using this Actor to overload a site; using it to get around a block, rate limit, login or any other access restriction; and continuing to audit a site after its owner has asked you to stop.

Site owners: you can stop this Actor with a robots.txt group for the user agent SeoSitemapValidatorActor, or ask us through the Issues tab on the Actor page to add your site to this Actor's do-not-audit list. A run on a listed site requests nothing from the site and costs only the $0.005 per-run start fee.

The example input audits https://crawlee.dev/, the public documentation site of the open-source Crawlee library, which its maintainer Apify also uses as an example target for its own crawlers.

Privacy

The Actor stores SEO fields only (above): no page text, no images, no cookies and no form contents. Query strings in URLs are replaced by ?[redacted] in the results.

"Results are written only to your run's default Apify storage. The developer does not copy, view or retain them. If your results contain personal data, you are the controller and the developer acts only as your processor under Apify's Standard Actor Contract. Apify's retention settings apply."

Third-party software

Runtime dependencies (apify, htmlparser2, robots-parser, tldts) are listed in NOTICE with their licences; the Public Suffix List data bundled in tldts is under MPL-2.0 (licenses/publicsuffix-list-MPL-2.0, source https://publicsuffix.org/list/). Lighthouse is a trademark of Google LLC; this Actor is not affiliated with or endorsed by Google.

Runtime

Node 24 (engines, .nvmrc, Dockerfile apify/actor-node:24). Memory: fixed at 1024 MB (this Actor's setting; a 50 MB sitemap with 50,000 entries plus 5,000 pages needs more than 512 MB). Tests: npm test; end-to-end: sh scripts/e2e-netns.sh (local fixture site only).

Contact: https://apify.com/luxe_tasks · Issues tab on the actor page.