# On-page SEO & Schema Extractor + Sitemap Validator (`luxe_tasks/seo-sitemap-validator`) Actor

Audit any public website: per-page title, meta description, canonical, robots, hreflang, Open Graph and JSON-LD types, plus sitemap protocol checks. Static HTML, no browser, robots.txt honoured. Price: $0.005/page audited + $0.0005/sitemap URL validated + $0.005 start fee per run.

- **URL**: https://apify.com/luxe_tasks/seo-sitemap-validator.md
- **Developed by:** [Luxe Tasks](https://apify.com/luxe_tasks) (community)
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $5.00 / 1,000 page auditeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## On-page SEO & Schema Extractor + Sitemap Validator

Developer: Luxe_tasks.

Audit **any public website** (your own site, or a client's or competitor's public pages): for each page listed in its sitemaps (or in your own list) get one row with the
**HTTP status, redirects, title, meta description, meta robots and X-Robots-Tag, canonical, hreflang, html lang, viewport, h1/h2 counts,
Open Graph and Twitter tag names, JSON-LD `@type`s and microdata types**, plus **issue codes** (missing title, duplicate descriptions,
canonical pointing elsewhere, noindex pages listed in the sitemap, ...). The sitemaps themselves are **validated against the sitemaps.org
protocol**: every entry gets a row with its codes (bad `lastmod`, URLs on another host, duplicates, fragments, ...), and every file gets its
findings (size, UTF-8, namespace, nested indexes, missing from robots.txt, ...).

It reads static HTML only (no browser, no JavaScript rendering), follows no links and never leaves the one site you name.

**Which tool to use:** for a Lighthouse SEO score per page (rendered in a browser), use our
[Bulk Website Auditor](https://apify.com/luxe_tasks/bulk-website-auditor); to find broken links, use our
[Broken Link Checker](https://apify.com/luxe_tasks/broken-link-checker). This Actor gives the raw per-field values, structured-data
types and sitemap validation for a whole site in one static pass.

### How to use it

1. **Site or sitemap URL**: the home page (`https://example.com/`) or one sitemap file (`https://example.com/sitemap.xml`). The example input is `https://crawlee.dev/` with small test limits (5 pages, 50 sitemap URLs, 2 sitemap files); change them for a real audit.
2. Optional: **Sitemap URLs** (otherwise the `Sitemap:` lines of robots.txt, else `/sitemap.xml`), **Page URLs to audit** (exactly these pages instead of the sitemap's), **Max pages** (default 500, at most 5,000).
3. Run it. Audited pages are in the dataset (**Audited pages** view); the Output tab links the sitemap rows, the sitemap file findings, the **SITE_SUMMARY** record and the pages that were not audited.

### What it requests

1. `/robots.txt` of the site (once per run).
2. The sitemap files: the ones you list, else the same-site `Sitemap:` lines of robots.txt, else `/sitemap.xml`; children of a sitemap index on the same site (one level, as the protocol allows). gzip sitemaps (`.xml.gz`) and text sitemaps are read.
3. Each page to audit, once (`GET`), following at most 5 redirects on the same site. A redirect to another site is recorded and not requested.

Nothing else: no links are followed, no images, scripts or styles are loaded, no forms are submitted.

### Input

| Field | Default | Notes |
|---|---|---|
| `siteUrl` | (required) | One public site (protocol + host) per run. Example input: `https://crawlee.dev/`. |
| `sitemapUrls` | `[]` | Up to 20, same site. |
| `pageUrls` | `[]` | Up to 5,000, same site; replaces the sitemap's page list. |
| `auditPages` | `true` | `false` = validate the sitemaps only. |
| `maxPages` | 500 | 1 to 5,000 (this Actor's limit). The example input uses 5. |
| `validateSitemapUrls` | `true` | `false` = no sitemap URL rows (the sitemaps are still read to find the pages). |
| `maxSitemapUrls` | 5,000 | 1 to 50,000 sitemap entries read in a run (50,000 is the protocol limit per file). The example input uses 50. |
| `maxSitemapFiles` | 10 | 1 to 50 files per run. The example input uses 2. |
| `timeoutSecs` | 15 | Per request, 5 to 30 seconds. |

### Output

- **Dataset, one row per audited page**: `url`, `finalUrl`, `httpStatus`, `redirectCount`, `redirectChain`, `redirectNotFollowed`, `contentType`, `responseTimeMs`, `tries`, `responseBytes`, `bodyTruncated`, `isHtml`, `xRobotsTag`, `xRobotsNoindex`, `inSitemap`, `sitemapLastmod`, `title`, `titleLength`, `titleCount`, `metaDescription`, `metaDescriptionLength`, `metaDescriptionCount`, `metaRobots`, `metaNoindex`, `metaNofollow`, `canonicalUrl`, `canonicalStatus` (`self`, `other`, `missing`, `multiple`, `invalid`), `canonicalCount`, `htmlLang`, `hasViewport`, `metaCharset`, `h1Count`, `h2Count`, `hreflang` (up to 30: `hreflang`, `href`, `valid`), `hreflangCount`, `hreflangInvalidCount`, `openGraphTags`, `twitterTags` (tag names, not values), `jsonLdBlocks`, `jsonLdTypes`, `jsonLdErrors`, `microdataTypes`, `loginForm`, `issues`, `issueCount`. The dataset holds audited pages, plus at most one status row for a stopped site (below).
- **Status row** (dataset): `url`, `status`, `reason`, `checkedAt` only. `status` is one of `ROBOTS_DISALLOWED`, `ROBOTS_UNREACHABLE`, `ACCESS_DENIED`, `CHALLENGE_DETECTED`, `OPTED_OUT`, `PRIVATE_ADDRESS_REFUSED`, `SITE_UNREACHABLE`. If a site's robots.txt disallows the audit or can't be read, the site denies access or shows a challenge, or the site's owner has opted out, the Actor stops for that site and writes one status row with the URL, a status and a reason, and no page content; status rows are not billed events, but the $0.005 per-run start fee still applies.
- **SITEMAP_URLS-0001, -0002, ...** (key-value store, 1,000 rows each): one row per validated sitemap entry: `loc`, `sitemap`, `codes`, `lastmod`, `changefreq`, `priority`, `valid`.
- **SITEMAP_FILES**: one entry per sitemap file: `url`, `source` (`input`, `robots`, `index`, `default`), `httpStatus`, `kind` (`urlset`, `sitemapindex`, `text`), `entries`, `gzipped`, `bytes`, `codes`.
- **SITE_SUMMARY**: totals, the robots.txt result, site findings, counts per issue code and per HTTP status, duplicate titles and meta descriptions (top 50, with 5 example pages each).
- **PAGE_ISSUES-0001, ...**: pages that were not audited and why (robots.txt, access denied, a challenge, an error, the time or budget limit).
- URLs are stored with their query string replaced by `?[redacted]`. Page text, images and tag values other than those listed above are never stored.
- The internal **DISCOVERY** record (used to resume a restarted run) keeps the page list as requested, query strings included, like your input.

### Issue codes

- Page: `HTTP_ERROR`, `REDIRECTED`, `REDIRECT_OFF_SITE`, `TOO_MANY_REDIRECTS`, `REDIRECT_NOT_FOLLOWED`, `NOINDEX_HEADER`, `NOINDEX_META`, `NOT_HTML`, `BODY_TRUNCATED`, `TITLE_MISSING`, `TITLE_TOO_LONG` (over 60 characters), `TITLE_TOO_SHORT` (under 10), `MULTIPLE_TITLES`, `META_DESCRIPTION_MISSING`, `META_DESCRIPTION_TOO_LONG` (over 160), `META_DESCRIPTION_TOO_SHORT` (under 50), `MULTIPLE_META_DESCRIPTIONS`, `H1_MISSING`, `MULTIPLE_H1`, `CANONICAL_MISSING`, `CANONICAL_POINTS_ELSEWHERE`, `CANONICAL_MULTIPLE`, `CANONICAL_INVALID`, `HTML_LANG_MISSING`, `VIEWPORT_MISSING`, `HREFLANG_INVALID`, `JSONLD_INVALID`, `OPEN_GRAPH_MISSING`; for pages listed in a sitemap also `SITEMAP_URL_REDIRECTS`, `SITEMAP_URL_NOT_200`, `SITEMAP_URL_NOINDEX`, `SITEMAP_URL_CANONICALISED`. Site summary adds `DUPLICATE_TITLE` and `DUPLICATE_META_DESCRIPTION`. The length thresholds are common SEO guidance, not search-engine rules.
- Sitemap entry: `LOC_MISSING`, `LOC_NOT_ENTITY_ESCAPED`, `LOC_TOO_LONG` (over 2,048 characters), `LOC_NOT_ABSOLUTE`, `LOC_BAD_SCHEME`, `LOC_HAS_WHITESPACE`, `LOC_HAS_FRAGMENT`, `LOC_OTHER_HOST`, `LOC_OUTSIDE_SITEMAP_PATH`, `DUPLICATE_LOC`, `LASTMOD_INVALID` (not a W3C datetime), `LASTMOD_IN_FUTURE`, `CHANGEFREQ_INVALID`, `PRIORITY_INVALID`.
- Sitemap file: `SITEMAP_NOT_FOUND`, `SITEMAP_HTTP_ERROR`, `SITEMAP_REDIRECTED`, `SITEMAP_REDIRECT_OFF_SITE`, `SITEMAP_TOO_MANY_REDIRECTS`, `SITEMAP_FETCH_FAILED`, `SITEMAP_TOO_LARGE` (over 50 MB uncompressed), `SITEMAP_BAD_GZIP`, `SITEMAP_EMPTY`, `SITEMAP_NOT_UTF8`, `SITEMAP_NOT_SITEMAP`, `SITEMAP_WRONG_NAMESPACE`, `SITEMAP_TOO_MANY_URLS` (over 50,000), `SITEMAP_INDEX_TOO_MANY_SITEMAPS`, `SITEMAP_INDEX_HAS_URL_ENTRIES`, `SITEMAP_URLSET_HAS_SITEMAP_ENTRIES`, `SITEMAP_INDEX_NESTED`, `SITEMAP_INDEX_OTHER_HOST`, `SITEMAP_INDEX_BAD_LOC`, `SITEMAP_NOT_IN_ROBOTS`.
- Site: `ROBOTS_ACCESS_DENIED`, `ROBOTS_CHALLENGE_DETECTED`, `PRIVATE_ADDRESS_REFUSED`, `ROBOTS_NO_SITEMAP_LINE`, `ROBOTS_SITEMAP_OTHER_HOST`, `ROBOTS_UNREACHABLE`, `ROBOTS_RATE_LIMITED`, `SITEMAP_FILES_NOT_FETCHED`, `SITEMAP_URL_LIMIT_REACHED`, `MAX_TOTAL_CHARGE_REACHED`, `DISCOVERY_RESUMED`.

### Limits

- One site per run; at most 5,000 pages per run (default 500) and 50,000 sitemap entries read per run (this Actor's limits). One sitemap file may be at most 50 MB uncompressed and 50,000 URLs (sitemaps.org protocol); files over 50 MB are reported, not parsed; files over 50,000 URLs are flagged (`SITEMAP_TOO_MANY_URLS`) and read only up to this run's limit.
- Pages are read up to 5 MB (this Actor's limit); larger pages are audited from their first 5 MB (`BODY_TRUNCATED`).
- Static HTML only: tags added by JavaScript are not seen.
- Run timeout 6,000 seconds (this Actor's setting). At 1 request per second, 5,000 pages take about 5,000 seconds, so they fit; with a slower site, a `Crawl-delay` over 1 second or after a 429, fewer pages fit. No request starts in the last 120 seconds; the pages left end as `time_budget_reached` in PAGE_ISSUES, without a per-page charge (the $0.005 per-run start fee still applies).

### Pricing

**Pricing: $0.005 per page audited and $0.0005 per sitemap URL validated, plus the $0.005 start fee per run, all three as pay-per-event prices in Apify Console.**
Each run costs a $0.005 start fee plus $0.005 per page audited and $0.0005 per sitemap URL validated, and runs that return no results cost only the $0.005 start fee.

- Pay-per-event **`page-audited`**: $0.005 for each page row saved to the dataset, plus the $0.005 start fee per run. A page counts as audited when the site answered it (any HTTP status except access denied, challenges and the errors below, including redirects and 404s, which are SEO findings); each page is charged once per run, after the result is recorded, on top of the $0.005 per-run start fee.
- Pay-per-event **`sitemap-url-validated`**: $0.0005 for each sitemap entry row saved (SITEMAP_URLS-\*), plus the $0.005 start fee per run; charged after each block of rows is saved.
- Pages that were not audited (robots.txt disallows them, the site refused or rate-limited the run, server or network errors, the time or budget limit) have no page-audited charge; the $0.005 per-run start fee still applies.
- A maximum total charge set on the run is respected: sitemap rows and pages are only saved while the budget covers them; the rest are listed as not validated or `budget_reached`, without a per-page or per-URL charge (the $0.005 per-run start fee still applies).
- Order within a maximum total charge: sitemap URL rows are validated and charged while the sitemaps are read, before any page is audited, so a small maximum can be used up by sitemap rows and leave few or no page rows; set `validateSitemapUrls` to `false` to spend the budget on pages only (each run still pays the $0.005 start fee).
- **What you pay even when nothing is audited:** the $0.005 start fee per run, charged by Apify through its synthetic `apify-actor-start` event ([Apify docs](https://docs.apify.com/platform/actors/publishing/monetize/pay-per-event#synthetic-start-event)). This Actor's memory is fixed at 1 GB (1024 MB, this Actor's setting), so the $0.005 start fee is charged once per run.
- A run that is refused before any request (an invalid input or a site on the do-not-audit list) audits no pages and costs only the $0.005 per-run start fee.
- The synthetic `apify-default-dataset-item` event is not used (the Actor refuses to run if it is priced).

### Politeness and safety

- robots.txt is honoured for every page (RFC 9309): a disallowed page is not requested. If robots.txt cannot be read (5xx, network error), or answers 401, 403 or 430, or shows a challenge page, nothing else is requested; a 404 robots.txt means no restrictions. robots.txt and sitemap files are always read, as the protocol intends. `Crawl-delay` is respected.
- One request at a time to the site, at least 1 second apart (this Actor's limit).
- A 429 or 503 is retried after its `Retry-After` time (seconds or a date, up to 60 seconds); without one, after 10 and then 20 seconds; after 3 tries the page is given up. After any 429 the Actor waits at least 5 seconds between requests for the rest of the run (this Actor's settings). A page still answered 429 after 3 tries stops the run for the site; so do 3 pages with server errors.
- A 401, 403 or 430 answer or a challenge/CAPTCHA page (on any HTTP status, including 405 and 503; a challenge is not retried) stops the run for the site at once: a challenge is never solved, and the pages already audited are kept.
- Honest fixed User-Agent `SeoSitemapValidatorActor/0.1.0 (+https://apify.com/luxe_tasks/seo-sitemap-validator; honours robots.txt)`; no cookies, no logins, no proxies, no custom headers. Pages with a password field are flagged (`loginForm`), never filled.
- Only the one site is requested (redirects to other sites are not followed); private-network addresses are refused.

### Your responsibility

You choose which site to audit. You are responsible for having the right to audit it, for following that site's terms of use and the law that applies to you, and for how you use the results, as set out in section 2 of the Apify Standard Actor Contract that governs this Actor. This Actor requests only pages that are public without a login, follows each site's robots.txt and Crawl-delay, sends one request at a time at most once per second, and stops at once if the site refuses access (401, 403, 430) or shows a challenge or CAPTCHA.

**Prohibited uses:** running several runs against the same site at the same time; using this Actor to overload a site; using it to get around a block, rate limit, login or any other access restriction; and continuing to audit a site after its owner has asked you to stop.

**Site owners:** you can stop this Actor with a robots.txt group for the user agent `SeoSitemapValidatorActor`, or ask us through the Issues tab on the Actor page to add your site to this Actor's do-not-audit list. A run on a listed site requests nothing from the site and costs only the $0.005 per-run start fee.

The example input audits https://crawlee.dev/, the public documentation site of the open-source Crawlee library, which its maintainer Apify also uses as an example target for its own crawlers.

### Privacy

The Actor stores SEO fields only (above): no page text, no images, no cookies and no form contents. Query strings in URLs are replaced by `?[redacted]` in the results.

> "Results are written only to your run's default Apify storage. The developer does not copy, view or retain them. If your results contain
> personal data, you are the controller and the developer acts only as your processor under Apify's Standard Actor Contract. Apify's retention settings apply."

### Third-party software

Runtime dependencies (apify, htmlparser2, robots-parser, tldts) are listed in NOTICE with their licences; the Public Suffix List data bundled in tldts is under MPL-2.0 (licenses/publicsuffix-list-MPL-2.0, source https://publicsuffix.org/list/).
Lighthouse is a trademark of Google LLC; this Actor is not affiliated with or endorsed by Google.

### Runtime

Node 24 (`engines`, `.nvmrc`, Dockerfile `apify/actor-node:24`). Memory: fixed at 1024 MB (this Actor's setting; a 50 MB sitemap with 50,000 entries plus 5,000 pages needs more than 512 MB). Tests: `npm test`; end-to-end: `sh scripts/e2e-netns.sh` (local fixture site only).

Contact: https://apify.com/luxe_tasks · Issues tab on the actor page.

# Actor input Schema

## `siteUrl` (type: `string`):

The public website to audit: its home page (for example https://example.com/) or one of its sitemap files. Only this one site (protocol + host) is requested, one page at a time and at most one per second, following its robots.txt. You are responsible for having the right to audit this site and for following its terms of use.

## `sitemapUrls` (type: `array`):

Sitemap or sitemap index files to validate, on the same site. Empty = the Sitemap lines of robots.txt, else /sitemap.xml.

## `pageUrls` (type: `array`):

Audit exactly these pages (same site) instead of the pages listed in the sitemaps. No links are followed.

## `auditPages` (type: `boolean`):

Off = validate the sitemaps only (no page requests, no page-audited events).

## `maxPages` (type: `integer`):

At most this many pages per run (default 500, at most 5,000). Pages are requested one at a time, at most one per second.

## `validateSitemapUrls` (type: `boolean`):

On = every sitemap entry read (up to the limit below) gets a validation row (sitemap-url-validated). Off = sitemap entries are only used to find pages; file-level findings are still reported.

## `maxSitemapUrls` (type: `integer`):

At most this many sitemap entries are read and validated per run (the protocol allows 50,000 per file).

## `maxSitemapFiles` (type: `integer`):

At most this many sitemap files (indexes included) are requested per run.

## `timeoutSecs` (type: `integer`):

How long to wait for one answer.

## Actor input object example

```json
{
  "siteUrl": "https://crawlee.dev/",
  "sitemapUrls": [],
  "pageUrls": [],
  "auditPages": true,
  "maxPages": 5,
  "validateSitemapUrls": true,
  "maxSitemapUrls": 50,
  "maxSitemapFiles": 2,
  "timeoutSecs": 15
}
```

# Actor output Schema

## `pages` (type: `string`):

Default dataset, 'Audited pages' view: one row per audited page (one page-audited event each).

## `sitemapUrls` (type: `string`):

Key-value store records SITEMAP_URLS-0001, -0002, ...: up to 1,000 validated sitemap entries each (loc, sitemap, codes, lastmod, changefreq, priority, valid); one sitemap-url-validated event per entry.

## `sitemapFiles` (type: `string`):

SITEMAP_FILES record: one entry per sitemap file found (url, source, httpStatus, kind, entries, gzipped, bytes, codes).

## `siteSummary` (type: `string`):

SITE_SUMMARY record: totals, robots.txt result, site findings, issue counts, duplicate titles and meta descriptions.

## `pageIssues` (type: `string`):

Key-value store records PAGE_ISSUES-0001, ...: pages that were not audited (robots.txt, blocks, errors, time or budget limits), with the reason; no per-page charge for these (the $0.005 per-run start fee still applies).

## `runSummary` (type: `string`):

RUN_SUMMARY record (kit): counts by status and page-audited events since the last start or resume.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "siteUrl": "https://crawlee.dev/",
    "maxPages": 5,
    "maxSitemapUrls": 50,
    "maxSitemapFiles": 2
};

// Run the Actor and wait for it to finish
const run = await client.actor("luxe_tasks/seo-sitemap-validator").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "siteUrl": "https://crawlee.dev/",
    "maxPages": 5,
    "maxSitemapUrls": 50,
    "maxSitemapFiles": 2,
}

# Run the Actor and wait for it to finish
run = client.actor("luxe_tasks/seo-sitemap-validator").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "siteUrl": "https://crawlee.dev/",
  "maxPages": 5,
  "maxSitemapUrls": 50,
  "maxSitemapFiles": 2
}' |
apify call luxe_tasks/seo-sitemap-validator --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,luxe_tasks/seo-sitemap-validator"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/d2WC6sF5YhpuhHmjd/builds/2STOnWa8NHKcKVh63/openapi.json