# Website SEO Spider — On-Page Crawl Export (`khadinakbar/website-seo-spider`) Actor

Crawl a public website and export one on-page SEO row per URL (title, meta, H1, canonical, indexability, inlinks, issue flags). Use for site crawls and scheduled audits. Pair with broken-link-checker or complete-seo-audit. Charged $0.01 per page crawled.

- **URL**: https://apify.com/khadinakbar/website-seo-spider.md
- **Developed by:** [Khadin Akbar](https://apify.com/khadinakbar) (community)
- **Categories:** SEO tools, Developer tools, MCP servers
- **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $10.00 / 1,000 page crawleds

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Website SEO Spider

Crawl a public website and export **one on-page SEO row per URL**. Each row includes title, meta description, H1, canonical, indexability, inlinks, outlinks, word count, schema types, and issue flags so agencies and agents can run a cloud SEO spider from Apify.

The Actor follows same-host or same-domain links up to `maxPages`, optionally seeds extra URLs from `sitemap.xml`, and writes `OUTPUT` plus `RUN_SUMMARY` for MCP readback.

### Best fit for this Actor

- Export a per-URL crawl for migrations, pre-launch QA, and scheduled site audits.
- Feed titles, canonicals, and issue flags into sheets, Looker, or an agent repair queue.
- Keep the job HTTP-only when SEO tags live in the first HTML response.

For a scored multi-check site report, start with [Complete SEO Audit Tool](https://apify.com/khadinakbar/complete-seo-audit) and then return here when you need one row per URL. When the next job is verifying every href status, continue with [Broken Link Checker](https://apify.com/khadinakbar/broken-link-checker).

### Practical scenario

An SEO lead pastes `https://example.com`, keeps `crawlScope` on `same-hostname`, and sets `maxPages` to `10`. The run returns one dataset row per fetched URL with title length, H1, canonical, unique inlinks from the crawled set, and `issueFlags` such as `missing_meta_description`. Hitting `maxPages` finishes `PARTIAL` with the saved rows. A later scheduled run with the same input is compared on `contentHash` and issue counts.

### Quick start input

```json
{
  "startUrls": [{ "url": "https://example.com" }],
  "crawlScope": "same-hostname",
  "maxPages": 10,
  "maxDepth": 3
}
```

### Input reference

| Field | Type | What it controls |
|---|---|---|
| `startUrls` | array | Public http(s) URLs where the crawl begins. Example: `https://example.com`. |
| `crawlScope` | enum | `same-hostname` (default), `same-domain`, or `page-only`. |
| `maxPages` | integer | Page budget. Default 100, prefill 10, maximum 2000. |
| `maxDepth` | integer | Link hops from a start URL. `0` means unlimited within `maxPages`. |
| `seedFromSitemap` | boolean | Add URLs from robots.txt Sitemap entries and `/sitemap.xml`. |
| `respectRobotsTxt` | boolean | Honor robots.txt for the crawler user-agent. Default true. |
| `ignoreUrlParameters` | boolean | Strip query strings before uniqueness. Default false. |
| `maxConcurrency` | integer | Parallel HTTP requests. Default 10. |
| `proxyConfiguration` | object | Optional Apify proxy for rate-limited hosts. |

### What data you receive

One dataset item is one crawled URL.

```json
{
  "url": "https://example.com/",
  "statusCode": 200,
  "indexability": "Indexable",
  "indexabilityStatus": "OK",
  "title": "Example Domain",
  "titleLength": 14,
  "metaDescription": "This domain is for use in illustrative examples.",
  "h1": "Example Domain",
  "canonicalUrl": "https://example.com/",
  "wordCount": 28,
  "inlinks": 0,
  "outlinks": 1,
  "issueFlags": ["title_short", "thin_content"],
  "issueCount": 2,
  "schemaTypes": [],
  "scrapedAt": "2026-08-22T00:00:00.000Z"
}
```

| Field | Meaning |
|---|---|
| `title` / `metaDescription` / `h1` | On-page copy plus character lengths |
| `canonicalUrl` / `indexability` | Public canonical and indexability from status plus robots |
| `inlinks` / `outlinks` | Unique crawled inlinks and on-page href counts |
| `issueFlags` | Machine flags such as `missing_title`, `duplicate_title`, `noindex` |
| `schemaTypes` | JSON-LD `@type` values when present |

`OUTPUT` and `RUN_SUMMARY` in the default key-value store hold `outcome`, `itemsPushed`, and `chargedEventCounts`.

### Use through the API

```bash
curl -X POST "https://api.apify.com/v2/acts/khadinakbar~website-seo-spider/runs" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://example.com"}],"maxPages":10,"crawlScope":"same-hostname"}'
```

Download rows as JSON, CSV, Excel, or HTML from the Dataset tab.

### Use with AI agents through Apify MCP

> Crawl https://example.com with maxPages 10 and same-hostname scope. Return url, title, canonicalUrl, indexability, issueFlags, and inlinks. Read OUTPUT.outcome and itemsPushed.

Connect via <https://mcp.apify.com>. Cost signal: **$0.01** per saved page plus platform usage. `PARTIAL` means the page budget was reached with useful rows still saved.

### Connect the workflow

- For a scored technical report on the same site, start with [Complete SEO Audit Tool](https://apify.com/khadinakbar/complete-seo-audit) after you have the URL list.
- When href status is the remaining question, continue with [Broken Link Checker](https://apify.com/khadinakbar/broken-link-checker).

### Pricing

This Actor uses Pay per event plus Apify platform usage. The live Pricing tab is the current source of truth for billing details.

- `apify-actor-start`: $0.00005 per run
- `page-crawled`: **$0.01** per saved URL row

A 10-page prefill is about $0.10 in result events plus a short Apify platform usage window.

### How it works

1. Normalize public start URLs and skip private or credentialed hosts.
2. Optionally seed extra URLs from robots.txt Sitemap entries and `/sitemap.xml`.
3. Crawl HTML with Cheerio, extract on-page SEO fields, then compute inlinks and duplicate titles across the crawled set.
4. Charge `page-crawled` for each saved row, then write the dataset item.

### Best results

- Provide a public homepage or section URL you are authorized to collect.
- Start with `maxPages` 10–50 to confirm tag quality, then raise the budget.
- Keep `page-only` when you want a single landing-page check.
- Enable `seedFromSitemap` when the homepage link graph is thin.
- Pair a later run with the same `startUrls` to compare `contentHash` and issue counts.

### Builder's note

I found that most agency "SEO spider" jobs are an Internal-tab export, not a single scored report. Shipping one row per URL with inlinks computed after the crawl keeps the dataset spreadsheet-shaped while still flagging duplicate titles that a page-at-a-time parser would miss. HTTP-only Cheerio covers the public HTML tags teams actually repair; rendered Core Web Vitals stay on specialized lab tools.

### Legal and responsible use

Use this Actor on public pages you are authorized to collect, follow applicable laws and the site's terms of service, and keep the output in your own compliance workflow. This Actor is independent of Screaming Frog Ltd and is not affiliated with any desktop SEO spider vendor.

Issues and feature requests: use the Actor Issues tab on Apify.

# Actor input Schema

## `startUrls` (type: `array`):

Public website URLs where the crawl begins. Example: https://example.com. Same-host or same-domain links are followed up to maxPages. This is not a list of already-known broken links — for that inventory use broken-link-checker.

## `crawlScope` (type: `string`):

Which internal links to follow from each HTML page. same-hostname stays on the exact host; same-domain also follows subdomains; page-only fetches the start URLs and stops. Defaults to same-hostname. This does not control off-site outbound counts, which are always recorded.

## `maxPages` (type: `integer`):

Hard cap on HTML/HTTP pages fetched in this run. Example: 10. Default 100, maximum 2000. Prefill 10 keeps the quality sample fast. When in-scope URLs remain at this cap, the run finishes PARTIAL. This is a page budget, not a broken-link check limit.

## `maxDepth` (type: `integer`):

How many link hops from a start URL the crawler may follow. 0 means unlimited within maxPages. Default 10, prefill 3. Depth 0 is the start URL itself. This is not a sitemap-only mode — enable seedFromSitemap to add sitemap URLs as extra starts.

## `seedFromSitemap` (type: `boolean`):

When true, read robots.txt Sitemap: entries and /sitemap.xml, then add those URLs to the crawl queue up to maxPages. Defaults to false. Useful for coverage beyond the homepage link graph. This is not a sitemap validator — URLs are crawled for on-page SEO, not merely listed.

## `respectRobotsTxt` (type: `boolean`):

When true, skip paths disallowed for the crawler user-agent in robots.txt. Defaults to true. Turn off only for sites you are authorized to audit beyond the public robots policy. This setting applies to HTML crawl enqueue, not to a separate link-checker pass.

## `ignoreUrlParameters` (type: `boolean`):

When true, strip query strings before uniqueness so /page and /page?ref=nav count as one URL. Defaults to false, matching typical desktop spider defaults. Enable for tracking-parameter cleanup. This does not rewrite canonical tags on the page.

## `maxConcurrency` (type: `integer`):

Parallel HTTP requests. Default 10, prefill 5, maximum 50. Lower this when the target rate-limits. This is request parallelism, not a browser-tab count — the crawler is HTTP-only.

## `proxyConfiguration` (type: `object`):

Optional Apify proxy. Leave off for the default direct HTTP path. Enable datacenter or residential when the site rate-limits your IP. This is not a login session or cookie jar.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://example.com"
    }
  ],
  "crawlScope": "same-hostname",
  "maxPages": 10,
  "maxDepth": 3,
  "seedFromSitemap": false,
  "respectRobotsTxt": true,
  "ignoreUrlParameters": false,
  "maxConcurrency": 5
}
```

# Actor output Schema

## `results` (type: `string`):

One on-page SEO row per crawled URL.

## `output` (type: `string`):

Final outcome, itemsPushed, and chargedEventCounts.

## `runSummary` (type: `string`):

Crawl diagnostics, issue rollup, and billing counters.

## `crawlSummary` (type: `string`):

Issue-flag counts and pages crawled for this start URL.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://example.com"
        }
    ],
    "crawlScope": "same-hostname",
    "maxPages": 10,
    "maxDepth": 3,
    "seedFromSitemap": false,
    "respectRobotsTxt": true,
    "ignoreUrlParameters": false,
    "maxConcurrency": 5
};

// Run the Actor and wait for it to finish
const run = await client.actor("khadinakbar/website-seo-spider").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://example.com" }],
    "crawlScope": "same-hostname",
    "maxPages": 10,
    "maxDepth": 3,
    "seedFromSitemap": False,
    "respectRobotsTxt": True,
    "ignoreUrlParameters": False,
    "maxConcurrency": 5,
}

# Run the Actor and wait for it to finish
run = client.actor("khadinakbar/website-seo-spider").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://example.com"
    }
  ],
  "crawlScope": "same-hostname",
  "maxPages": 10,
  "maxDepth": 3,
  "seedFromSitemap": false,
  "respectRobotsTxt": true,
  "ignoreUrlParameters": false,
  "maxConcurrency": 5
}' |
apify call khadinakbar/website-seo-spider --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,khadinakbar/website-seo-spider"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/f0PPU2rWRKUBwAfeZ/builds/mMtR5WJwhZJ35dNXV/openapi.json