# Website SEO Crawler — Screaming Frog Alternative (`khadinakbar/website-seo-spider`) Actor

Crawl public website pages and export structured SEO data. Review page URLs, titles, descriptions, headings, canonicals, links, and response details for page-level audits, migration checks, or downstream reporting.

- **URL**: https://apify.com/khadinakbar/website-seo-spider.md
- **Developed by:** [Khadin Akbar](https://apify.com/khadinakbar) (community)
- **Categories:** SEO tools, Developer tools, MCP servers
- **Security:** Limited permissions
- **Stats:** 4 total users, 3 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $10.00 / 1,000 page crawleds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### What is Website SEO Crawler — Screaming Frog Alternative?

Crawl public website pages and export structured SEO data. Review page URLs, titles, descriptions, headings, canonicals, links, and response details for page-level audits, migration checks, or downstream reporting. For technical SEO analysts, each dataset row represents one crawled URL and its page-level SEO observations.

### Workflow: put the results to work

Start with the public site and a crawl cap appropriate to the review. Use page-level rows to compare titles, descriptions, links, and canonical signals across URLs. For a prioritized aggregate audit, use a report-oriented workflow after collecting the crawl evidence.

### Best fit

Choose this Actor when the job is a public, raw-HTML on-page crawl that needs a structured dataset rather than a desktop crawl project. Use it for a landing-page check, an exportable site inventory, or a scheduled data refresh.

- Start with a public homepage or section URL and a small page budget to validate the tag quality.
- Use `same-hostname` for a host-focused crawl, `same-domain` when approved subdomains matter, or `page-only` for a single URL.
- Pair the resulting URL inventory with Complete SEO Audit Tool when the next step is a scored site report.
- Continue with Broken Link Checker when the next decision is the HTTP status of every discovered href.

The collection path is direct HTTP by default. The optional residential Apify Proxy mode is for authorized public sites where direct requests are consistently rate-limited. Browser-rendered, authenticated, and local-network pages belong in a workflow designed for those inputs.

### Workflow example

An SEO lead starts with a public storefront homepage, sets `maxPages` to `25`, and saves the `overview` dataset view. The crawl follows in-scope links, records the raw HTML tags it receives, then writes one row per saved URL. The lead filters `issueFlags`, sends selected rows to an editorial queue, and schedules the same input later to compare `contentHash`, canonical, or issue changes.

### Quick-start input

```json
{
    "startUrls": [{ "url": "https://example.com" }],
    "crawlScope": "same-hostname",
    "maxPages": 10,
    "maxDepth": 3,
    "respectRobotsTxt": true,
    "fetchMode": "direct"
}
```

### Input reference

| Field                 | Type    | What it controls                                                                                                |
| --------------------- | ------- | --------------------------------------------------------------------------------------------------------------- |
| `startUrls`           | array   | One or more public http(s) seed URLs. Every saved row keeps its source `startUrl`.                              |
| `crawlScope`          | enum    | `same-hostname` (default), `same-domain`, or `page-only`.                                                       |
| `maxPages`            | integer | Page budget. Default 100, prefill 10, maximum 2,000.                                                            |
| `maxDepth`            | integer | Link hops after the seed; `0` is unlimited within `maxPages`.                                                   |
| `seedFromSitemap`     | boolean | Add in-scope URLs from robots.txt Sitemap entries and `/sitemap.xml`.                                           |
| `respectRobotsTxt`    | boolean | Honor robots.txt during crawl enqueue. Default `true`.                                                          |
| `ignoreUrlParameters` | boolean | Remove query strings before URL uniqueness.                                                                     |
| `maxConcurrency`      | integer | Parallel HTTP requests. Default 5, maximum 20, with a small same-domain delay.                                  |
| `fetchMode`           | enum    | `direct` (default) or `residential` Apify Proxy for approved public sources with repeatable direct rate limits. |
| `proxyConfiguration`  | object  | Advanced Apify Proxy configuration that overrides `fetchMode`.                                                  |

### Output contract

Each Dataset item represents one validated public URL saved by the run. The default Dataset includes an `overview` view for on-page fields and an `issues` view for prioritizing repair work.

```json
{
    "url": "https://example.com/",
    "requestedUrl": "https://example.com/",
    "statusCode": 200,
    "indexability": "Indexable",
    "indexabilityStatus": "OK",
    "title": "Example Domain",
    "titleLength": 14,
    "metaDescription": "This domain is for use in illustrative examples.",
    "canonicalUrl": "https://example.com/",
    "h1": "Example Domain",
    "inlinks": 0,
    "outlinks": 1,
    "issueFlags": ["title_short", "thin_content"],
    "issueCount": 2,
    "startUrl": "https://example.com/",
    "crawlScope": "same-hostname",
    "scrapedAt": "<ISO-8601 collection timestamp>"
}
```

| Surface            | Meaning                                                                                                                                    |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------ |
| Dataset `overview` | One saved URL row with status, titles, meta description, canonical, indexability, inlinks, outlinks, issue count, and collection time.     |
| Dataset `issues`   | A repair-oriented field set including status, indexability detail, issue flags, title/meta lengths, H1 count, and missing image alt count. |
| `OUTPUT`           | Compact terminal contract with `outcome`, `itemsPushed`, `chargedEventCounts`, estimated result-event charge, and warnings.                |
| `RUN_SUMMARY`      | Diagnostic companion with source scope, crawl counts, issue rollup, execution identifiers, and billing counters.                           |
| `CRAWL_SUMMARY`    | The source seed URLs, selected scope, issue-flag rollup, and saved page count.                                                             |

`COMPLETE` means all collected candidates were saved without a limiting condition. `PARTIAL` preserves saved rows when the page cap, charge cap, retries, timeout guard, or a persistence-readback warning limits the result. `INVALID_INPUT` reports rejected non-public or malformed seed URLs without page-row charges.

### API and AI-agent use

Run the Actor through the Apify API, then read the Dataset and `OUTPUT` record before triggering a downstream action.

```bash
curl -X POST "https://api.apify.com/v2/acts/khadinakbar~website-seo-spider/runs" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://example.com"}],"crawlScope":"same-hostname","maxPages":10,"fetchMode":"direct"}'
```

Use the same bounded input through Apify MCP when an AI agent needs an exportable crawl result.

> Crawl `https://example.com` with `maxPages` 10 and `same-hostname` scope. Return `url`, `title`, `canonicalUrl`, `indexability`, `issueFlags`, and `inlinks`; then read `OUTPUT.outcome`, `itemsPushed`, `chargedEventCounts`, `scrapedAt`, and `startUrl` before proposing a repair queue.

### How this Screaming Frog alternative compares

This is a workflow alternative for cloud-exportable public HTML SEO crawls, not a claim of full desktop-suite parity.

| Decision dimension          | Cloud SEO Crawl Export                                                                                           | Screaming Frog SEO Spider                                                                                       |
| --------------------------- | ---------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| Scope and workflow          | Runs a focused cloud crawl of public raw HTML and returns structured per-URL Dataset rows.                       | Offers a broader desktop crawler suite that includes workflows beyond this Actor's public raw-HTML export.      |
| Billing and same-job cost   | Uses PPE for saved rows with Apify platform usage included; the live Pricing tab supports a bounded, on-demand job calculation. | Uses an annual per-user desktop licensing model; the job-level comparison depends on team use and crawl volume. |
| Effective efficiency        | Returns API-readable Dataset and key-value records for a scoped automation or repair queue.                      | Provides a desktop interface and richer suite workflows for users who need them.                                |
| Input friction              | Starts from public URLs and documented crawl settings, with no source credentials in the Actor input.            | Requires the desktop product and the appropriate licensed workflow for the broader feature set.                 |
| Output contract             | Declares Dataset views plus `OUTPUT` and `RUN_SUMMARY`, including source URL and collection timestamp.           | Provides desktop crawl reports and exports suited to its broader crawler environment.                           |
| Integrations and automation | Fits an Apify API or scheduled workflow; the same input can be rerun as a bounded cloud job.                     | Documents scheduling and integrations within the desktop suite and its licensed features.                       |
| Reliability evidence        | Reviewed release evidence reconciles saved Dataset rows with page-event charges for public test cases.           | This Actor does not make a comparative reliability claim for the desktop product.                               |

For the precise alternative boundary: use this Actor for public HTML crawl exports and an Apify-native data handoff. Choose the broader suite when browser rendering, authenticated environments, local crawl history, visualizations, advanced custom extraction, or its other desktop capabilities are central to the job.

### Best results

- Use sites and paths you are authorized to collect, with a clear output purpose.
- Start at 10–50 pages, validate the `overview` view, then increase `maxPages` only when the saved rows match the audit need.
- Keep the direct mode for public sites that respond normally; choose residential mode only after observing repeatable direct rate limits and confirming the extra platform cost is appropriate.
- Enable sitemap seeding when coverage matters more than the homepage link graph.
- Compare subsequent runs using `contentHash`, canonical, title, and issue counts, while retaining each row's `scrapedAt` and `startUrl` provenance.

### Builder's note

I designed this Actor around the part of a spider crawl that is easiest to operationalize: a reliable, one-row-per-URL handoff with source provenance and explicit terminal state. I kept the first version focused on raw HTML so the output contract, billing unit, and downstream repair workflow remain clear instead of implying a browser-rendering or full-suite replacement.

### Legal and responsible use

Use this Actor only for public pages you are authorized to collect, follow applicable laws and site terms, honor robots.txt when appropriate, and keep results in your own compliance workflow.

**Screaming Frog** is a trademark of its owner. This independent Actor is not affiliated, associated, or endorsed by Screaming Frog.

Use the Actor Issues tab on Apify for feature requests or support.

### Pricing and run costs

This Actor uses **Pay per event with Apify platform usage included**. The [Pricing tab](https://apify.com/khadinakbar/website-seo-spider/pricing) lists the current event rates and billing terms.

| Event | Billing unit | When it applies |
|---|---|---|
| `apify-actor-start` | Actor Start | Charged when the Actor starts running. Number of events charged depends on Actor memory (one event per GB, minimum one event). |
| `page-crawled` | Page crawled | Charged once per crawled URL whose on-page SEO row is saved to the dataset. |

Run cost combines the charged events and Apify platform usage. Review the run charge limit and requested result count before starting.

### Connect an AI agent

Use the [Apify MCP configurator](https://mcp.apify.com) to choose an available client connection. Inspect this Actor’s current input schema and required credentials before running it.

# Actor input Schema

## `startUrls` (type: `array`):

One or more public website URLs where the crawl begins. Example: https://example.com. Each seed keeps its own startUrl provenance. Same-hostname or same-domain links are followed up to maxPages.

## `crawlScope` (type: `string`):

Which internal links to follow from each HTML page. same-hostname stays on the exact host; same-domain also follows subdomains; page-only fetches the start URLs and stops. Defaults to same-hostname.

## `maxPages` (type: `integer`):

Hard cap on public pages fetched in this run. Example: 10. Default 100, maximum 2000. Prefill 10 keeps the first sample bounded. When useful in-scope URLs remain at this cap, the run finishes PARTIAL.

## `maxDepth` (type: `integer`):

How many link hops from a start URL the crawler may follow. 0 means unlimited within maxPages. Default 10, prefill 3. Depth 0 is the start URL itself. Enable seedFromSitemap to add sitemap URLs as extra starts.

## `seedFromSitemap` (type: `boolean`):

When true, read robots.txt Sitemap entries and /sitemap.xml, then add in-scope URLs to the crawl queue up to maxPages. Defaults to false. Useful for coverage beyond the homepage link graph.

## `respectRobotsTxt` (type: `boolean`):

When true, skip paths disallowed for the crawler user-agent in robots.txt. Defaults to true. Turn off only for sites you are authorized to audit beyond the public robots policy.

## `ignoreUrlParameters` (type: `boolean`):

When true, strip query strings before uniqueness so /page and /page?ref=nav count as one URL. Defaults to false. Enable for tracking-parameter cleanup; it does not rewrite canonical tags on the page.

## `maxConcurrency` (type: `integer`):

Parallel HTTP requests. Default and prefill are 5; maximum 20. Lower this for a rate-limited target. The crawler also applies a small same-domain delay for stable collection.

## `fetchMode` (type: `string`):

Direct HTTP is the lower-cost default. Select residential Apify Proxy only for public sites you are authorized to collect when direct requests have repeatable rate limits; proxy platform costs can apply.

## `proxyConfiguration` (type: `object`):

Optional advanced Apify Proxy configuration. When supplied, it overrides Fetch mode. Use it only for public targets you are authorized to audit; it does not provide login credentials or browser rendering.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://example.com"
    }
  ],
  "crawlScope": "same-hostname",
  "maxPages": 10,
  "maxDepth": 3,
  "seedFromSitemap": false,
  "respectRobotsTxt": true,
  "ignoreUrlParameters": false,
  "maxConcurrency": 5,
  "fetchMode": "direct",
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `results` (type: `string`):

One validated on-page SEO row per saved public URL.

## `output` (type: `string`):

Final outcome, saved row count, charge acknowledgements, and warnings.

## `runSummary` (type: `string`):

Crawl diagnostics, source scope, issue rollup, and billing counters.

## `crawlSummary` (type: `string`):

Issue-flag counts and pages crawled for this start URL.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://example.com"
        }
    ],
    "crawlScope": "same-hostname",
    "maxPages": 10,
    "maxDepth": 3,
    "seedFromSitemap": false,
    "respectRobotsTxt": true,
    "ignoreUrlParameters": false,
    "maxConcurrency": 5,
    "fetchMode": "direct"
};

// Run the Actor and wait for it to finish
const run = await client.actor("khadinakbar/website-seo-spider").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://example.com" }],
    "crawlScope": "same-hostname",
    "maxPages": 10,
    "maxDepth": 3,
    "seedFromSitemap": False,
    "respectRobotsTxt": True,
    "ignoreUrlParameters": False,
    "maxConcurrency": 5,
    "fetchMode": "direct",
}

# Run the Actor and wait for it to finish
run = client.actor("khadinakbar/website-seo-spider").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://example.com"
    }
  ],
  "crawlScope": "same-hostname",
  "maxPages": 10,
  "maxDepth": 3,
  "seedFromSitemap": false,
  "respectRobotsTxt": true,
  "ignoreUrlParameters": false,
  "maxConcurrency": 5,
  "fetchMode": "direct"
}' |
apify call khadinakbar/website-seo-spider --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,khadinakbar/website-seo-spider"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/f0PPU2rWRKUBwAfeZ/builds/VgcoYYZsbn0qxADMx/openapi.json
