Sitemap Freshness & Schema Integrity Auditor

Find stale pages, broken schema dates, and sitemap lastmod mismatches before search engines, clients, or content teams notice.

This Actor audits sitemap-backed freshness signals with static HTTP requests only. It expands sitemap URLs, checks robots.txt, fetches each page, extracts HTML and JSON-LD context, compares date sources, and outputs deterministic issue codes for every URL.

The issue codes are the product. The freshness score is a quick triage helper.

Who it is for

• Technical SEO teams checking sitemap lastmod quality.
• Content teams auditing stale article, recipe, docs, and product pages.
• Agencies that need a repeatable client-facing freshness report.
• Site owners who want structured data/date integrity checks without running a browser crawler.

What it does

• Expands sitemap urlset and sitemapindex files, including bounded sitemap indexes.
• Checks robots.txt before page fetches when enabled.
• Fetches pages with static HTTP GET requests, redirects, retries, timeout, and body-size limits.
• Extracts title, meta description, canonical, H1, robots meta, language, Open Graph, Twitter metadata, and a content hash.
• Extracts JSON-LD entities, schema types, schema date fields, and local recommended-field signals.
• Detects Microdata and RDFa presence without claiming full graph validation.
• Extracts semantic date candidates from sitemap lastmod, HTTP Last-Modified, JSON-LD, meta tags, time[datetime], and visible semantic date elements.
• Compares cross-source date signals and emits deterministic issue codes.
• Writes exactly one dataset row per URL and writes an aggregate run summary to OUTPUT.

What it does not do

This is intentionally not a browser crawler or generic SEO suite.

• No Playwright, Puppeteer, browser rendering, screenshots, or visual diffs.
• No residential proxies or proxy rotation.
• No LLM calls, AI classification, or claim verification.
• No login-gated scraping, form submission, or paywalled content access.
• No unbounded crawling beyond provided sitemap URLs or direct URLs.
• No historical diff tracking or monitoring state.
• No full Google Rich Results Test parity.

Sitemap URLs vs direct URLs

Use sitemap input when you can. Sitemap runs are the strongest mode because the Actor can compare page signals against sitemap lastmod.

Use direct URLs when you want to inspect specific pages, reproduce an issue, or test a small page set. Direct URL runs cannot produce sitemap-specific findings unless those URLs also came from a sitemap.

Do not put robots.txt or sitemap.xml in the direct URL list. Put sitemap files in sitemapUrls; direct URLs should be HTML pages.

Example inputs

Generic sitemap audit

{
  "sitemapUrls": ["https://www.your-site.com/sitemap.xml"],
  "urls": [],
  "maxPages": 100,
  "maxConcurrency": 5,
  "requestDelayMs": 250,
  "requestTimeoutSecs": 20,
  "respectRobotsTxt": true,
  "includeSeoBasics": true,
  "includeSchemaValidation": true,
  "includeRawDateCandidates": false,
  "dateMismatchToleranceDays": 3,
  "userAgent": "Sitemap Freshness & Schema Integrity Auditor (+https://apify.com/)"
}

Direct URL spot check

{
  "sitemapUrls": [],
  "urls": [
    "https://www.your-site.com/blog/example-post",
    "https://www.your-site.com/docs/example-page"
  ],
  "maxPages": 10,
  "maxConcurrency": 2,
  "requestDelayMs": 250,
  "requestTimeoutSecs": 20,
  "respectRobotsTxt": true,
  "includeSeoBasics": true,
  "includeSchemaValidation": true,
  "includeRawDateCandidates": false,
  "dateMismatchToleranceDays": 3,
  "userAgent": "Sitemap Freshness & Schema Integrity Auditor (+https://apify.com/)"
}

Ecommerce/product sitemap audit

Use this shape for Shopify-style product sitemaps or any product sitemap with product page URLs. Replace the URL with your own product sitemap.

{
  "sitemapUrls": ["https://www.example-shop.com/sitemap_products_1.xml"],
  "urls": [],
  "maxPages": 100,
  "maxConcurrency": 2,
  "requestDelayMs": 500,
  "requestTimeoutSecs": 20,
  "respectRobotsTxt": true,
  "includeSeoBasics": true,
  "includeSchemaValidation": true,
  "includeRawDateCandidates": false,
  "dateMismatchToleranceDays": 3,
  "userAgent": "Sitemap Freshness & Schema Integrity Auditor (+https://apify.com/)"
}

Debug date extraction

Turn on raw date candidates only when you are debugging why a page received a date-related issue. It makes rows more verbose.

{
  "sitemapUrls": ["https://www.your-site.com/sitemap.xml"],
  "urls": [],
  "maxPages": 20,
  "maxConcurrency": 2,
  "requestDelayMs": 250,
  "requestTimeoutSecs": 20,
  "respectRobotsTxt": true,
  "includeSeoBasics": true,
  "includeSchemaValidation": true,
  "includeRawDateCandidates": true,
  "dateMismatchToleranceDays": 3,
  "userAgent": "Sitemap Freshness & Schema Integrity Auditor (+https://apify.com/)"
}

Output

The dataset contains one row per URL. Successful HTML rows include:

• url, normalizedUrl, finalUrl, and checkedAt
• status: ok, skipped, or error
• source: direct or sitemap metadata, including sitemap lastmod when available
• request: HTTP status, content type, response bytes, redirects, and Last-Modified
• page: static HTML context such as title, canonical, H1, robots meta, language, and content hash
• dates: selected published/modified dates, raw source availability, and best known update date
• schema: JSON-LD/schema detection and local validation context
• freshness: score, label, and short explanation
• issues: detailed deterministic issues with severity, category, message, and optional evidence
• issueCodes: unique issue-code strings for filtering and aggregation

The Actor also writes an aggregate run summary to Apify key-value store key OUTPUT. The summary includes discovered/queued/checked counts, score distribution, issue counts, top critical URLs, warnings, and cost guardrail flags.

The Actor definition includes Apify output schemas for a default overview table, an issue-focused table, a date-signal table, and the OUTPUT run summary record.

Example output row

{
  "url": "https://www.your-site.com/blog/example-post",
  "normalizedUrl": "https://www.your-site.com/blog/example-post",
  "finalUrl": "https://www.your-site.com/blog/example-post",
  "checkedAt": "2026-07-08T00:00:00.000Z",
  "status": "ok",
  "source": {
    "type": "sitemap",
    "sitemapUrl": "https://www.your-site.com/sitemap.xml",
    "sitemapLastmod": "2026-07-01"
  },
  "request": {
    "httpStatus": 200,
    "contentType": "text/html; charset=utf-8",
    "responseBytes": 123456,
    "redirectCount": 0,
    "lastModifiedHeader": null,
    "fetchError": null
  },
  "dates": {
    "sitemapLastmod": "2026-07-01",
    "httpLastModified": null,
    "bestKnownUpdateDate": "2026-07-01",
    "daysSinceBestKnownUpdate": 7,
    "dateSourcesAvailable": ["sitemap"]
  },
  "schema": {
    "found": true,
    "types": ["Article", "BreadcrumbList", "Organization"],
    "primaryType": "Article",
    "datePublished": "2026-01-10",
    "dateModified": "2026-07-01",
    "validationScope": "local-heuristic",
    "missingRecommendedFields": []
  },
  "freshness": {
    "score": 96,
    "scoreLabel": "Fresh",
    "explanation": "Best known update date is 7 days old; score reflects date completeness, consistency, schema, and SEO context signals."
  },
  "issues": [
    {
      "code": "HTTP_LAST_MODIFIED_MISSING",
      "severity": "info",
      "category": "date-missing",
      "message": "The page did not return an HTTP Last-Modified header."
    }
  ],
  "issueCodes": ["HTTP_LAST_MODIFIED_MISSING"]
}

Freshness score

Scores are clamped from 0 to 100.

• 90-100: Fresh
• 70-89: Adequate
• 50-69: Needs Attention
• 25-49: Stale
• 0-24: Critical
• null: Not Scored

Skipped rows, fetch failures, and non-HTML rows are not scored. Their freshness.score is null.

The score is explainable from issue penalties. It is useful for sorting, but the issue codes should drive action.

Issue codes

Fetch and processing

• INVALID_INPUT_URL: an input URL is invalid or unsupported.
• SITEMAP_FETCH_FAILED: a sitemap could not be fetched.
• SITEMAP_PARSE_FAILED: sitemap XML could not be parsed.
• SITEMAP_EMPTY: sitemap parsing produced no URL entries.
• SITEMAP_INDEX_DEPTH_LIMIT: sitemap index recursion reached the V1 depth limit.
• URL_FETCH_FAILED: a page URL could not be fetched.
• HTTP_STATUS_NON_200: a page returned a non-200 HTTP status.
• NON_HTML_CONTENT: the response was not HTML.
• RESPONSE_TOO_LARGE: the response exceeded the body-size limit.

Robots

• ROBOTS_DISALLOWED: robots.txt disallowed the page URL.
• ROBOTS_FETCH_FAILED: robots.txt could not be fetched; the run continues with a warning.

Schema

• JSON_LD_PARSE_ERROR: one or more JSON-LD blocks could not be parsed.
• SCHEMA_NOT_FOUND: no JSON-LD schema entities were found.
• SCHEMA_TYPE_MISSING: schema was found but no type was detected.
• SCHEMA_RECOMMENDED_FIELD_MISSING: a local recommended schema field is missing.
• SCHEMA_DATE_PUBLISHED_MISSING: content schema is missing datePublished.
• SCHEMA_DATE_MODIFIED_MISSING: content schema is missing dateModified.
• SCHEMA_MODIFIED_BEFORE_PUBLISHED: schema dateModified is earlier than datePublished.

Date source availability

• SITEMAP_LASTMOD_MISSING: sitemap input was used but the entry has no lastmod.
• HTTP_LAST_MODIFIED_MISSING: the page response did not include Last-Modified.
• VISIBLE_DATE_MISSING: no semantic visible date was found on a content page.
• DATE_PUBLISHED_MISSING: no reliable published date was found.
• DATE_MODIFIED_MISSING: no reliable modified date was found.

Date mismatches

• DATE_MISMATCH_SCHEMA_MODIFIED_VS_SITEMAP_LASTMOD
• DATE_MISMATCH_SCHEMA_MODIFIED_VS_HTTP_LAST_MODIFIED
• DATE_MISMATCH_SCHEMA_MODIFIED_VS_VISIBLE_MODIFIED
• DATE_MISMATCH_SCHEMA_PUBLISHED_VS_VISIBLE_PUBLISHED
• DATE_MISMATCH_SITEMAP_LASTMOD_VS_HTTP_LAST_MODIFIED
• SITEMAP_LASTMOD_NEWER_THAN_SCHEMA_MODIFIED
• HTTP_LAST_MODIFIED_NEWER_THAN_SCHEMA_MODIFIED
• VISIBLE_MODIFIED_NEWER_THAN_SCHEMA_MODIFIED
• FUTURE_DATE_DETECTED
• INVALID_DATE_FORMAT

Staleness

• CONTENT_OLDER_THAN_90_DAYS
• CONTENT_OLDER_THAN_6_MONTHS
• CONTENT_OLDER_THAN_12_MONTHS
• CONTENT_OLDER_THAN_24_MONTHS
• NO_RELIABLE_UPDATE_DATE

SEO context

• CANONICAL_MISSING
• CANONICAL_POINTS_ELSEWHERE
• ROBOTS_NOINDEX

SEO context is included to explain page interpretation. It is not meant to turn this Actor into a broad SEO audit tool.

Troubleshooting

I got NON_HTML_CONTENT

The URL returned XML, JSON, text, an image, or another non-HTML content type. If you were trying to audit a sitemap, move that URL into sitemapUrls instead of urls.

I got ROBOTS_DISALLOWED

The Actor respected robots.txt and skipped the URL. This is expected when respectRobotsTxt is enabled. Only disable this for sites you own, control, or have explicit permission to test.

Many rows have SITEMAP_LASTMOD_MISSING

The sitemap entries do not include lastmod. The Actor can still inspect page/schema signals, but sitemap freshness comparisons will be unavailable.

Many rows have HTTP_LAST_MODIFIED_MISSING

Many modern sites do not send Last-Modified for HTML pages. This is usually an info-level issue if sitemap or schema dates are present.

Product pages show ProductGroup

That is normal for ecommerce pages that model variants under a top-level product group. The Actor treats ProductGroup as the primary ecommerce entity when present.

Cost notes

Dogfood runs have been cheap with static HTTP:

• Apify Blog: 100 pages cost about $0.007.
• Allbirds product sitemap: 100 pages cost about $0.013.

A cautious early estimate is roughly $0.07-$0.13 per 1,000 pages, assuming similar response sizes and latency. Real costs vary by target site, response size, redirects, failures, and Apify runtime conditions. Measure your own runs before committing to a public pricing model.