Sitemap Freshness & Schema Integrity Auditor

What is Sitemap Freshness & Schema Integrity Auditor?

Sitemap Freshness & Schema Integrity Auditor reads public XML sitemaps and/or HTTP(S) HTML page URLs, then extracts page, date, and JSON-LD signals with static requests. It returns one dataset row per unique queued page plus a run summary so SEO and content teams can find stale content, conflicting dates, and structured-data gaps.

What data can it extract or produce?

• Expands XML urlset files and nested sitemapindex files, retaining sitemap lastmod, changefreq, and priority metadata.
• Combines direct page spot checks with sitemap-discovered pages under one run limit.
• Records HTTP status, final URL, redirects, content type, response size, normalized Last-Modified, and fetch errors.
• Extracts title, meta description, canonical, H1, robots meta, language, Open Graph, Twitter metadata, semantic date candidates, and a SHA-256 content hash from static HTML.
• Parses JSON-LD, selects a primary type, and checks recommended fields for common article, recipe, product, breadcrumb, and organization types.
• Detects the presence of Microdata and RDFa without claiming full graph validation.
• Compares sitemap, HTTP, JSON-LD, meta, Microdata, and semantic HTML date signals using deterministic issue codes.
• Produces a 0 to 100 freshness score and an OUTPUT summary with totals, score bands, issue counts, warnings, and critical URLs.

The score is a triage aid. Use issueCodes and the supporting issues evidence to decide what to fix.

What can you use the data for?

• Check whether sitemap lastmod values agree with schema and page-level update dates.
• Prioritize old articles, recipes, documentation, or product pages for editorial review.
• Find missing or contradictory datePublished and dateModified values in content schema.
• Review Product, ProductGroup, Recipe, Article, BlogPosting, and NewsArticle JSON-LD before releases or migrations.
• Build recurring client or internal reports from stable issue codes and freshness score bands.

How much does it cost?

This Actor uses pay-per-event pricing. At the currently published rates, an Audited URL dataset item costs $0.00022 to $0.00025 by Apify plan tier. Thus, 1,000 rows cost $0.22 to $0.25, plus Actor Start events at $0.00005 each. Start events depend on run memory at one per GB, with a minimum of one.

Estimated charge = audited URL rows x your tier's URL price + Actor Start events x $0.00005

Skipped and error rows are still dataset items. Set maxPages deliberately and confirm current rates on the Pricing tab.

How to use Sitemap Freshness & Schema Integrity Auditor

1. In Apify Console, enter Sitemap URLs, Additional URLs, or both.
2. Set maxPages to the most unique pages you intend to audit and pay for.
3. Run the Actor and review the Overview, Issues, Date signals, and OUTPUT summary views.
4. Filter or export issueCodes for remediation or reporting.

Prefer sitemap input when you need sitemapLastmod comparisons. Use direct URLs for small checks or reproducing a finding.

What input does the Actor accept?

Field	Type and limits	Behavior
sitemapUrls	URL array	Public HTTP(S) XML urlset or sitemapindex URLs.
urls	URL array	Individual HTTP(S) HTML pages, not sitemap or robots files.
maxPages	Integer, 1 to 1000; default 100	Total unique queued pages after expansion and deduplication.
maxConcurrency	Integer, 1 to 10; default 5	Maximum page workers. Higher values can increase target load and rate-limit risk.
requestDelayMs	Integer, 0 to 5000; default 250	Attempted delay between requests to the same origin.
requestTimeoutSecs	Integer, 5 to 60; default 20	Abort timeout while opening each sitemap, robots, or page request; retries can make total time longer.
respectRobotsTxt	Boolean; default true	Fetches and caches /robots.txt per origin; disallowed pages are skipped.
includeSeoBasics	Boolean; default true	Controls canonical and robots-meta issue checks; page context is still extracted.
includeSchemaValidation	Boolean; default true	Controls recommended-field issues; JSON-LD extraction and core schema/date issues still run.
includeRawDateCandidates	Boolean; default false	Adds consolidated dates.rawCandidates; useful for debugging but increases output size, not fetch count.
dateMismatchToleranceDays	Integer, 0 to 30; default 3	Flags compared date sources only when their difference exceeds this value.
userAgent	Non-empty string	HTTP User-Agent used for fetching and robots.txt evaluation.

At least one URL source is required. Sitemaps run first; direct URLs fill capacity left under maxPages. Normalization removes fragments but retains query strings, and duplicates share one row. maxPages is the billing guardrail; robots checks add a cached request per origin, while concurrency mainly changes throughput and target load.

Minimal sitemap audit:

{
  "sitemapUrls": ["https://example.com/sitemap.xml"]
}

See the Store Input tab for the current schema and defaults.

What output does the Actor produce?

The default dataset contains one row per unique queued URL. Its top-level fields are:

Field	Meaning and nullability
url, normalizedUrl, checkedAt, status	Original URL, normalized URL, check time, and ok, skipped, or error status.
finalUrl	URL after redirects, or null when no page was fetched.
source	type plus optional sitemapUrl, sitemapLastmod, sitemapChangefreq, and sitemapPriority.
request	httpStatus, contentType, responseBytes, redirectCount, lastModifiedHeader, and fetchError; unavailable values are null.
page	title, metaDescription, canonical, h1Count, h1Text, robotsMeta, language, contentHash, openGraph, twitter, and dateCandidates. Omitted from skipped/error rows; unavailable scalar values are null.
dates	published, modified, sitemapLastmod, httpLastModified, visiblePublished, visibleModified, bestKnownUpdateDate, daysSinceBestKnownUpdate, dateSourcesAvailable, and optional rawCandidates. Omitted from skipped/error rows; unavailable dates are null.
schema	found, jsonLdBlocks, jsonLdParseErrors, types, primaryType, datePublished, dateModified, hasMicrodata, hasRdfa, validationScope, warnings, and missingRecommendedFields. Omitted from skipped/error rows.
freshness	score, scoreLabel, and explanation; score is null and label is Not Scored for skipped/error rows.
issues, issueCodes	Detailed issues with optional evidence, plus unique code strings for filtering.

For the sitemap input above, selected fields from a successful row could look like this; page content and source headers determine the actual values:

{
  "url": "https://example.com/blog/post",
  "status": "ok",
  "source": {
    "type": "sitemap",
    "sitemapUrl": "https://example.com/sitemap.xml",
    "sitemapLastmod": "2026-08-20"
  },
  "request": {
    "httpStatus": 200,
    "lastModifiedHeader": null,
    "fetchError": null
  },
  "dates": {
    "bestKnownUpdateDate": "2026-08-20",
    "daysSinceBestKnownUpdate": 6,
    "dateSourcesAvailable": ["sitemap", "schema", "visible"]
  },
  "schema": {
    "found": true,
    "primaryType": "Article",
    "datePublished": "2026-08-01",
    "dateModified": "2026-08-20",
    "validationScope": "local-heuristic"
  },
  "freshness": {
    "score": 96,
    "scoreLabel": "Fresh",
    "explanation": "Best known update date is 6 days old; score reflects date completeness, consistency, schema, and SEO context signals."
  },
  "issues": [{
    "code": "HTTP_LAST_MODIFIED_MISSING",
    "severity": "info",
    "category": "date-missing",
    "message": "The page did not return an HTTP Last-Modified header."
  }],
  "issueCodes": ["HTTP_LAST_MODIFIED_MISSING"]
}

The default key-value store record OUTPUT contains the aggregate summary. Dataset views and the summary are linked from the run's Output tab. Dataset items can be exported as JSON, JSONL, CSV, Excel, XML, HTML table, or RSS. See the Store Output tab for the full schemas.

How are errors and partial results handled?

Fetch failures, non-200 responses, non-HTML content, and responses over 5 MB produce status: "error" rows with a null score. Robots disallows produce status: "skipped". Check status, request.fetchError, and issueCodes because other page rows remain available.

Sitemap fetch/parse failures, empty sitemaps, robots fetch failures, and depth limits go to OUTPUT.warnings, not page rows. Robots fetch failure allows processing to continue. Invalid input, unsupported protocols, or no URL source fails before normal output is written.

For empty results, inspect warnings and validate the sitemap XML. If direct URLs are missing, raise maxPages or narrow the sitemap. Different query strings remain distinct despite deduplication. For repeated errors, lower concurrency, increase delay or timeout, and verify the target directly.

What are the limits, performance characteristics, and best practices?

• maxPages is capped at 1,000 and applies across sitemap and direct inputs.
• Sitemap index recursion is limited to two child levels below the supplied sitemap.
• Page and sitemap fetches follow up to five redirects and retry network errors, HTTP 429, and 5xx responses twice after the first attempt; robots fetches retry once.
• Response bodies are capped at 5 MB. The Actor uses static HTTP only: no JavaScript rendering, proxy rotation, login flow, cookies, screenshots, LLM calls, or historical state.
• Runtime depends on page count, origins, response latency and size, retries, timeout, concurrency, delay, and robots checks. No fixed completion time is guaranteed.

Start with 10 to 50 pages, review source-dependent issues, then scale. Keep concurrency conservative.

How can you use the API, schedules, webhooks, and integrations?

Run sebastian-actors/sitemap-freshness-schema-integrity-auditor through the Apify API or the official Node.js and Python apify-client packages. Read page rows from the default dataset and summary JSON from key-value store key OUTPUT.

Use Apify Schedules for recurring checks. A run-succeeded webhook can trigger n8n, Make, or Zapier to filter issue codes and send rows to a spreadsheet, database, or ticket queue.

Frequently asked questions

Does it render JavaScript?

No. Client-rendered schema or dates are not visible in the static GET response.

Is this equivalent to Google's structured-data tools?

No. JSON-LD checks are deterministic local heuristics for the documented schema types. Microdata and RDFa are presence signals only, not full validation.

Can it audit private, login-gated, or proxy-only pages?

No authentication, session, or proxy input is implemented. Targets must be reachable through ordinary HTTP(S).

Why are sitemap fields null for direct URLs?

Direct input has no sitemap entry. Use sitemapUrls when you need lastmod, changefreq, priority, or sitemap-to-page comparisons.

Is it legal and responsible to use this Actor?

Use it only on sites you own, control, or are permitted to audit. Review applicable laws, contracts, website terms, and robots directives, minimize load with appropriate limits and delays, and avoid collecting unnecessary personal or sensitive data. This is general operational guidance, not legal advice.

Which related Actors may be useful?

Website Crawl Preflight Auditor is a meaningfully different companion for assessing robots, sitemap reachability, static extraction quality, JavaScript risk, crawl limits, and estimated crawl cost before a broader scraping or RAG job. Use this Actor instead when the main goal is per-page freshness and schema-date integrity.

How can you get support?

Use the Actor's Issues tab for bugs and feature requests. Include the run ID, a sanitized input, relevant issueCodes or OUTPUT.warnings, and one reproducible public URL; do not post API tokens, cookies, credentials, or private page content.