On Page SEO Audit Scraper

Run a bounded on page SEO audit across one or more public websites and receive one structured dataset row per page. The Actor follows same-site links and reports HTTP status, titles, descriptions, headings, canonical URLs, robots directives, indexability, links, content signals, structured data types, image alt coverage, and deterministic issue flags.

Use the output for a one-time website review, recurring technical SEO audits, migration checks, spreadsheet exports, or your own reporting pipeline. This is an HTTP-first crawler: it audits anonymous server-rendered HTML without launching a browser or silently activating a paid proxy.

What does this on page SEO crawler do?

For every supplied start URL, the Actor:

1. validates that the URL resolves to a public network address;
2. requests the page with bounded redirects and retries;
3. extracts page-level on page SEO fields from HTML and response headers;
4. discovers normalized same-site links;
5. stops at your page, depth, link, and concurrency limits;
6. compares collected titles and descriptions for duplicates;
7. saves one typed record for every attempted page.

HTTP errors and non-HTML URLs are still represented as records. A terminal request failure gets a REQUEST_FAILED row instead of disappearing from an audit.

Who is it for?

• SEO consultants auditing small and medium public websites.
• In-house SEO teams checking releases, migrations, and recurring changes.
• Developers validating metadata and indexability in CI or scheduled jobs.
• Agencies collecting the same normalized fields across multiple client sites.
• Data teams loading page-level technical SEO records into a warehouse or dashboard.

The Actor is designed for page-source signals. Choose Lighthouse Website Audit when you need Lighthouse performance, accessibility, best-practice, and SEO scores instead.

Why use this Actor?

• Multi-page output: one reusable row per page rather than one site-level summary.
• Deterministic flags: stable issue codes are easy to filter, compare, and alert on.
• Bounded crawling: explicit page, depth, per-page link, timeout, and concurrency controls.
• Multi-site runs: start URLs may belong to unrelated public websites.
• Safe URL handling: local, private, reserved, credential-bearing, and unsafe redirect targets are rejected.
• Cost-aware HTTP route: no browser or automatic residential proxy fallback.
• Integration-ready schema: normalized URLs, nullable failure fields, arrays, counts, and timestamps.

What on page SEO data is extracted?

Field	Meaning
url, finalUrl, startUrl	Requested URL, redirect destination, and crawl seed
depth	Link distance from the supplied start URL
statusCode, contentType	Final HTTP response information
responseTimeMs	Request and redirect elapsed time
title, titleLength	Normalized HTML title and character count
metaDescription, metaDescriptionLength	Meta description and character count
h1, h2	All non-empty H1 and H2 texts
canonicalUrl	Resolved valid canonical URL, or null
robots, indexable	Robots directives and calculated indexability
language	HTML lang value
wordCount, textLength	Approximate visible-body content signals
internalLinkCount, externalLinkCount	Unique link counts
internalLinks, externalLinks	Up to 100 normalized URLs of each kind
imageCount, imagesMissingAlt	Image and missing-alt counts
structuredDataTypes	Unique JSON-LD @type values
issueFlags	Sorted deterministic page issue codes
error	Concise terminal request error, otherwise null
auditedAt	ISO 8601 audit timestamp

Which SEO issue flags are reported?

A record may contain any of these codes:

• HTTP_ERROR
• NON_HTML
• REQUEST_FAILED
• MISSING_TITLE
• TITLE_TOO_SHORT
• TITLE_TOO_LONG
• DUPLICATE_TITLE
• MISSING_META_DESCRIPTION
• META_DESCRIPTION_TOO_SHORT
• META_DESCRIPTION_TOO_LONG
• DUPLICATE_META_DESCRIPTION
• MISSING_H1
• MULTIPLE_H1
• MISSING_CANONICAL
• INVALID_CANONICAL
• NOINDEX
• MISSING_LANGUAGE
• LOW_WORD_COUNT
• IMAGES_MISSING_ALT

Threshold flags are diagnostic signals, not universal ranking rules. Review them in the context of page purpose, language, templates, and search strategy.

How to run your first website audit

1. Open the Actor in Apify Console.
2. Add one or more public pages to Start URLs.
3. Set Maximum pages to a small value such as 10 for the first run.
4. Choose a link depth; 2 is a useful bounded preview.
5. Keep concurrency low for a small or rate-limited website.
6. Click Start.
7. Open the default dataset and select the SEO audit view.
8. Export rows as JSON, CSV, Excel, XML, or another Apify-supported format.

A practical starter input is:

{
  "startUrls": [{ "url": "https://www.w3.org/" }],
  "maxPages": 10,
  "maxDepth": 2,
  "maxConcurrency": 2,
  "includeSubdomains": false,
  "followQueryParameters": false
}

Input parameters

Input	Default	Limits	Description
startUrls	required	1 or more	Public HTTP(S) crawl seeds
maxPages	10	1–10,000	Total output page records across all sites
maxDepth	3	0–10	Same-site link levels followed from each seed
maxConcurrency	3	1–20	Page requests processed at once
includeSubdomains	false	boolean	Whether discovered subdomain links are in scope
followQueryParameters	false	boolean	Whether query variants count as separate pages
maxLinksPerPage	20	1–5,000	Discovery links accepted from each page
requestTimeoutSecs	30	5–120	Per-request timeout before bounded retries

When query following is disabled, query strings are removed during URL normalization. When enabled, common tracking parameters such as utm_*, fbclid, and gclid are still removed.

Output example

This shortened record reflects real current output from a W3C page audit:

{
  "url": "https://www.w3.org/standards/",
  "finalUrl": "https://www.w3.org/standards/",
  "startUrl": "https://www.w3.org/",
  "depth": 1,
  "statusCode": 200,
  "contentType": "text/html; charset=utf-8",
  "title": "Web Standards | W3C",
  "titleLength": 19,
  "metaDescriptionLength": 61,
  "h1": ["Web standards"],
  "canonicalUrl": null,
  "indexable": true,
  "wordCount": 738,
  "internalLinkCount": 31,
  "externalLinkCount": 4,
  "issueFlags": [
    "IMAGES_MISSING_ALT",
    "META_DESCRIPTION_TOO_SHORT",
    "MISSING_CANONICAL",
    "TITLE_TOO_SHORT"
  ],
  "error": null,
  "auditedAt": "2026-08-24T00:00:00.000Z"
}

Fields that cannot be obtained are null rather than invented. Arrays are empty when no values are found.

How much does it cost to audit website pages?

Pay-per-event pricing consists of a $0.005 start fee and one page-audit event for every dataset row. The BRONZE page price is $0.004 per page audit; higher plan tiers receive lower page prices.

Examples at BRONZE pricing:

Useful page records	Estimated charge
1	$0.0090
10	$0.0450
25	$0.1050
100	$0.4050

Failed-request and HTTP-error rows are page audit records because they preserve actionable coverage evidence. No separate charge exists for links, headings, flags, or structured-data fields included in a row.

Apify compute usage is covered by pay-per-event pricing for customers. The exact active price is always displayed in Console before a run.

Recurring technical SEO audit workflow

For recurring monitoring:

1. save a Task with stable start URLs and crawl limits;
2. schedule it daily, weekly, or after deployments;
3. export each run dataset or send it through a webhook;
4. join records by url;
5. compare statusCode, indexable, metadata fields, and issueFlags;
6. alert only on changes important to your site.

The Actor emits current snapshots. It does not maintain history or send alerts by itself; Apify schedules, webhooks, datasets, and your downstream system provide those workflow layers.

Spreadsheet and data pipeline integrations

You can:

• export the dataset directly to CSV or Excel;
• connect Apify with Google Sheets through supported integrations;
• trigger a Make or Zapier workflow after a run;
• retrieve JSON through the dataset API;
• load normalized rows into BigQuery, Snowflake, PostgreSQL, or a lake;
• filter issueFlags in a dashboard;
• compare successive scheduled datasets in your own model.

Keep url as the page identity and preserve startUrl when one run contains several websites.

Run with the Apify API

Replace APIFY_TOKEN with your token.

cURL

curl -X POST \
  "https://api.apify.com/v2/acts/automation-lab~multi-page-on-page-seo-audit/runs?token=APIFY_TOKEN&waitForFinish=300" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://www.w3.org/"}],"maxPages":10,"maxDepth":2}'

JavaScript

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/multi-page-on-page-seo-audit').call({
  startUrls: [{ url: 'https://www.w3.org/' }],
  maxPages: 10,
  maxDepth: 2,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Python

from apify_client import ApifyClient

client = ApifyClient("APIFY_TOKEN")
run = client.actor("automation-lab/multi-page-on-page-seo-audit").call(run_input={
    "startUrls": [{"url": "https://www.w3.org/"}],
    "maxPages": 10,
    "maxDepth": 2,
})
items = client.dataset(run["defaultDatasetId"]).list_items().items
print(items)

Use with MCP and AI agents

Add this Actor as a tool in Claude Code:

claude mcp add --transport http apify \
  "https://mcp.apify.com?tools=automation-lab/multi-page-on-page-seo-audit"

Claude Desktop setup: add the following remote MCP server configuration.

Cursor setup: open MCP settings and add the same remote server entry.

VS Code setup: add the server entry to your MCP configuration file.

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=automation-lab/multi-page-on-page-seo-audit"
    }
  }
}

Example prompts:

• “Audit up to 20 pages from https://www.w3.org/ and summarize noindex or missing-title records.”
• “Run the multi-site technical SEO audit and return pages with HTTP errors.”
• “Export URLs whose canonical is missing and word count is below 200.”

Legality and responsible use

Only crawl public pages you are authorized to access. Follow applicable terms, policies, copyright rules, privacy law, and website operator guidance. Use conservative limits and concurrency. Do not use the Actor to probe private networks: private, loopback, link-local, reserved, credential-bearing, and unsafe redirect destinations are blocked.

This Actor does not log in, solve CAPTCHAs, bypass access controls, or automatically switch to paid residential proxies. A public website may still rate-limit or block automated requests.

Limitations

• Client-rendered metadata that is absent from server HTML is not visible.
• The crawler follows HTML anchors; it does not ingest XML sitemaps in this release.
• Query URLs are merged by default to reduce crawl traps.
• Link counts describe links on the audited page; linked destinations are not labeled broken unless their own URL is crawled and returns an error row.
• indexable reflects response status, HTML content type, and noindex directives. It is not a search-engine indexing guarantee.
• Word count is an approximate normalized visible-body count.
• Duplicate flags compare only rows in the current run.
• A page may change between recurring runs.
• Authenticated, browser-only, CAPTCHA-protected, and private pages are outside scope.

Tips for reliable audits

• Start with 10–25 pages and inspect discovered URL patterns.
• Leave query following off unless query variants are genuine pages.
• Lower concurrency when a site returns 429 or intermittent 5xx responses.
• Supply important section pages as additional start URLs when they are not linked near the root.
• Increase depth gradually; deep calendars and faceted navigation can expose many URLs.
• Keep the same inputs for comparable recurring snapshots.
• Treat content-length thresholds as review cues rather than automatic SEO verdicts.

Troubleshooting

Why did the run return fewer pages than maxPages?

The same-site crawl may have exhausted unique eligible links, reached maxDepth, skipped file assets, or merged query variants. maxPages is a ceiling, not a promised count.

Why is a page a REQUEST_FAILED row?

The page did not return a final response after bounded retries. Check error, reduce concurrency, verify the URL in a normal unauthenticated request, and try a smaller audit. The Actor retries transient network, 429, and 5xx failures, not stable parser or input errors.

Why did the Actor reject a URL?

Only anonymous public HTTP(S) destinations are supported. Local/private addresses, single-label hosts, credential-bearing URLs, and redirects to restricted networks fail closed.

Why is JavaScript-rendered content missing?

The Actor intentionally parses HTTP HTML without a browser. Use a public server-rendered or alternate URL, or choose a browser-based audit product when rendered DOM is essential.

Related Automation Lab Actors

• Lighthouse Website Audit — Lighthouse performance, accessibility, best-practice, and SEO audit scores.
• Broken Link Checker — destination-focused broken-link checks.
• Website HTML & Text Change Monitor — versioned HTML and normalized text changes.
• Multi-Site Article Content Extractor — article-focused text and metadata extraction from supplied URLs.

FAQ

Does it crawl more than one website?

Yes. Add unrelated public start URLs and set one total maxPages budget. Every row preserves its originating startUrl.

Does it execute JavaScript?

No. The HTTP-first design is faster and cheaper for server-rendered HTML but cannot see metadata created only after browser execution.

Does it use a proxy?

No automatic proxy mode is exposed. Direct anonymous HTTP is used for predictable cost and behavior.

Are errors charged?

Every saved page audit row uses the page-audit event, including HTTP-error and terminal request-failure rows because they are actionable audit results. Invalid run input fails before page processing.

Can I schedule it?

Yes. Save the input as an Apify Task and attach a schedule. Store each run dataset or forward results through an integration for historical comparison.

Can the output prove that Google indexed a page?

No. indexable is a deterministic technical signal from the fetched response and directives, not evidence from a search-engine index.

Will it publish or modify my website?

No. It performs read-only HTTP requests and writes results only to the Actor run dataset.