Website SEO Spider

Crawl a public website and export one on-page SEO row per URL. Each row includes title, meta description, H1, canonical, indexability, inlinks, outlinks, word count, schema types, and issue flags so agencies and agents can run a cloud SEO spider from Apify.

The Actor follows same-host or same-domain links up to maxPages, optionally seeds extra URLs from sitemap.xml, and writes OUTPUT plus RUN_SUMMARY for MCP readback.

Best fit for this Actor

• Export a per-URL crawl for migrations, pre-launch QA, and scheduled site audits.
• Feed titles, canonicals, and issue flags into sheets, Looker, or an agent repair queue.
• Keep the job HTTP-only when SEO tags live in the first HTML response.

For a scored multi-check site report, start with Complete SEO Audit Tool and then return here when you need one row per URL. When the next job is verifying every href status, continue with Broken Link Checker.

Practical scenario

An SEO lead pastes https://example.com, keeps crawlScope on same-hostname, and sets maxPages to 10. The run returns one dataset row per fetched URL with title length, H1, canonical, unique inlinks from the crawled set, and issueFlags such as missing_meta_description. Hitting maxPages finishes PARTIAL with the saved rows. A later scheduled run with the same input is compared on contentHash and issue counts.

Quick start input

{
  "startUrls": [{ "url": "https://example.com" }],
  "crawlScope": "same-hostname",
  "maxPages": 10,
  "maxDepth": 3
}

Input reference

Field	Type	What it controls
startUrls	array	Public http(s) URLs where the crawl begins. Example: https://example.com.
crawlScope	enum	same-hostname (default), same-domain, or page-only.
maxPages	integer	Page budget. Default 100, prefill 10, maximum 2000.
maxDepth	integer	Link hops from a start URL. 0 means unlimited within maxPages.
seedFromSitemap	boolean	Add URLs from robots.txt Sitemap entries and /sitemap.xml.
respectRobotsTxt	boolean	Honor robots.txt for the crawler user-agent. Default true.
ignoreUrlParameters	boolean	Strip query strings before uniqueness. Default false.
maxConcurrency	integer	Parallel HTTP requests. Default 10.
proxyConfiguration	object	Optional Apify proxy for rate-limited hosts.

What data you receive

One dataset item is one crawled URL.

{
  "url": "https://example.com/",
  "statusCode": 200,
  "indexability": "Indexable",
  "indexabilityStatus": "OK",
  "title": "Example Domain",
  "titleLength": 14,
  "metaDescription": "This domain is for use in illustrative examples.",
  "h1": "Example Domain",
  "canonicalUrl": "https://example.com/",
  "wordCount": 28,
  "inlinks": 0,
  "outlinks": 1,
  "issueFlags": ["title_short", "thin_content"],
  "issueCount": 2,
  "schemaTypes": [],
  "scrapedAt": "2026-08-22T00:00:00.000Z"
}

Field	Meaning
title / metaDescription / h1	On-page copy plus character lengths
canonicalUrl / indexability	Public canonical and indexability from status plus robots
inlinks / outlinks	Unique crawled inlinks and on-page href counts
issueFlags	Machine flags such as missing_title, duplicate_title, noindex
schemaTypes	JSON-LD @type values when present

OUTPUT and RUN_SUMMARY in the default key-value store hold outcome, itemsPushed, and chargedEventCounts.

Use through the API

curl -X POST "https://api.apify.com/v2/acts/khadinakbar~website-seo-spider/runs" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://example.com"}],"maxPages":10,"crawlScope":"same-hostname"}'

Download rows as JSON, CSV, Excel, or HTML from the Dataset tab.

Use with AI agents through Apify MCP

Crawl https://example.com with maxPages 10 and same-hostname scope. Return url, title, canonicalUrl, indexability, issueFlags, and inlinks. Read OUTPUT.outcome and itemsPushed.

Connect via https://mcp.apify.com. Cost signal: $0.01 per saved page plus platform usage. PARTIAL means the page budget was reached with useful rows still saved.

Connect the workflow

• For a scored technical report on the same site, start with Complete SEO Audit Tool after you have the URL list.
• When href status is the remaining question, continue with Broken Link Checker.

Pricing

This Actor uses Pay per event plus Apify platform usage. The live Pricing tab is the current source of truth for billing details.

• apify-actor-start: $0.00005 per run
• page-crawled: $0.01 per saved URL row

A 10-page prefill is about $0.10 in result events plus a short Apify platform usage window.

How it works

1. Normalize public start URLs and skip private or credentialed hosts.
2. Optionally seed extra URLs from robots.txt Sitemap entries and /sitemap.xml.
3. Crawl HTML with Cheerio, extract on-page SEO fields, then compute inlinks and duplicate titles across the crawled set.
4. Charge page-crawled for each saved row, then write the dataset item.

Best results

• Provide a public homepage or section URL you are authorized to collect.
• Start with maxPages 10–50 to confirm tag quality, then raise the budget.
• Keep page-only when you want a single landing-page check.
• Enable seedFromSitemap when the homepage link graph is thin.
• Pair a later run with the same startUrls to compare contentHash and issue counts.

Builder's note

I found that most agency "SEO spider" jobs are an Internal-tab export, not a single scored report. Shipping one row per URL with inlinks computed after the crawl keeps the dataset spreadsheet-shaped while still flagging duplicate titles that a page-at-a-time parser would miss. HTTP-only Cheerio covers the public HTML tags teams actually repair; rendered Core Web Vitals stay on specialized lab tools.

Legal and responsible use

Use this Actor on public pages you are authorized to collect, follow applicable laws and the site's terms of service, and keep the output in your own compliance workflow. This Actor is independent of Screaming Frog Ltd and is not affiliated with any desktop SEO spider vendor.

Issues and feature requests: use the Actor Issues tab on Apify.