Website SEO Crawler — Screaming Frog Alternative

Crawl public website pages and export structured SEO data. Review page URLs, titles, descriptions, headings, canonicals, links, and response details for page-level audits, migration checks, or downstream reporting. For technical SEO analysts, each dataset row represents one crawled URL and its page-level SEO observations.

Workflow: put the results to work

Start with the public site and a crawl cap appropriate to the review. Use page-level rows to compare titles, descriptions, links, and canonical signals across URLs. For a prioritized aggregate audit, use a report-oriented workflow after collecting the crawl evidence.

Best fit

Choose this Actor when the job is a public, raw-HTML on-page crawl that needs a structured dataset rather than a desktop crawl project. Use it for a landing-page check, an exportable site inventory, or a scheduled data refresh.

• Start with a public homepage or section URL and a small page budget to validate the tag quality.
• Use same-hostname for a host-focused crawl, same-domain when approved subdomains matter, or page-only for a single URL.
• Pair the resulting URL inventory with Complete SEO Audit Tool when the next step is a scored site report.
• Continue with Broken Link Checker when the next decision is the HTTP status of every discovered href.

The collection path is direct HTTP by default. The optional residential Apify Proxy mode is for authorized public sites where direct requests are consistently rate-limited. Browser-rendered, authenticated, and local-network pages belong in a workflow designed for those inputs.

Workflow example

An SEO lead starts with a public storefront homepage, sets maxPages to 25, and saves the overview dataset view. The crawl follows in-scope links, records the raw HTML tags it receives, then writes one row per saved URL. The lead filters issueFlags, sends selected rows to an editorial queue, and schedules the same input later to compare contentHash, canonical, or issue changes.

Quick-start input

{
    "startUrls": [{ "url": "https://example.com" }],
    "crawlScope": "same-hostname",
    "maxPages": 10,
    "maxDepth": 3,
    "respectRobotsTxt": true,
    "fetchMode": "direct"
}

Input reference

Field	Type	What it controls
startUrls	array	One or more public http(s) seed URLs. Every saved row keeps its source startUrl.
crawlScope	enum	same-hostname (default), same-domain, or page-only.
maxPages	integer	Page budget. Default 100, prefill 10, maximum 2,000.
maxDepth	integer	Link hops after the seed; 0 is unlimited within maxPages.
seedFromSitemap	boolean	Add in-scope URLs from robots.txt Sitemap entries and /sitemap.xml.
respectRobotsTxt	boolean	Honor robots.txt during crawl enqueue. Default true.
ignoreUrlParameters	boolean	Remove query strings before URL uniqueness.
maxConcurrency	integer	Parallel HTTP requests. Default 5, maximum 20, with a small same-domain delay.
fetchMode	enum	direct (default) or residential Apify Proxy for approved public sources with repeatable direct rate limits.
proxyConfiguration	object	Advanced Apify Proxy configuration that overrides fetchMode.

Output contract

Each Dataset item represents one validated public URL saved by the run. The default Dataset includes an overview view for on-page fields and an issues view for prioritizing repair work.

{
    "url": "https://example.com/",
    "requestedUrl": "https://example.com/",
    "statusCode": 200,
    "indexability": "Indexable",
    "indexabilityStatus": "OK",
    "title": "Example Domain",
    "titleLength": 14,
    "metaDescription": "This domain is for use in illustrative examples.",
    "canonicalUrl": "https://example.com/",
    "h1": "Example Domain",
    "inlinks": 0,
    "outlinks": 1,
    "issueFlags": ["title_short", "thin_content"],
    "issueCount": 2,
    "startUrl": "https://example.com/",
    "crawlScope": "same-hostname",
    "scrapedAt": "<ISO-8601 collection timestamp>"
}

Surface	Meaning
Dataset overview	One saved URL row with status, titles, meta description, canonical, indexability, inlinks, outlinks, issue count, and collection time.
Dataset issues	A repair-oriented field set including status, indexability detail, issue flags, title/meta lengths, H1 count, and missing image alt count.
OUTPUT	Compact terminal contract with outcome, itemsPushed, chargedEventCounts, estimated result-event charge, and warnings.
RUN_SUMMARY	Diagnostic companion with source scope, crawl counts, issue rollup, execution identifiers, and billing counters.
CRAWL_SUMMARY	The source seed URLs, selected scope, issue-flag rollup, and saved page count.

COMPLETE means all collected candidates were saved without a limiting condition. PARTIAL preserves saved rows when the page cap, charge cap, retries, timeout guard, or a persistence-readback warning limits the result. INVALID_INPUT reports rejected non-public or malformed seed URLs without page-row charges.

API and AI-agent use

Run the Actor through the Apify API, then read the Dataset and OUTPUT record before triggering a downstream action.

curl -X POST "https://api.apify.com/v2/acts/khadinakbar~website-seo-spider/runs" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://example.com"}],"crawlScope":"same-hostname","maxPages":10,"fetchMode":"direct"}'

Use the same bounded input through Apify MCP when an AI agent needs an exportable crawl result.

Crawl https://example.com with maxPages 10 and same-hostname scope. Return url, title, canonicalUrl, indexability, issueFlags, and inlinks; then read OUTPUT.outcome, itemsPushed, chargedEventCounts, scrapedAt, and startUrl before proposing a repair queue.

How this Screaming Frog alternative compares

This is a workflow alternative for cloud-exportable public HTML SEO crawls, not a claim of full desktop-suite parity.

Decision dimension	Cloud SEO Crawl Export	Screaming Frog SEO Spider
Scope and workflow	Runs a focused cloud crawl of public raw HTML and returns structured per-URL Dataset rows.	Offers a broader desktop crawler suite that includes workflows beyond this Actor's public raw-HTML export.
Billing and same-job cost	Uses PPE for saved rows plus platform usage; the live Pricing tab supports a bounded, on-demand job calculation.	Uses an annual per-user desktop licensing model; the job-level comparison depends on team use and crawl volume.
Effective efficiency	Returns API-readable Dataset and key-value records for a scoped automation or repair queue.	Provides a desktop interface and richer suite workflows for users who need them.
Input friction	Starts from public URLs and documented crawl settings, with no source credentials in the Actor input.	Requires the desktop product and the appropriate licensed workflow for the broader feature set.
Output contract	Declares Dataset views plus OUTPUT and RUN_SUMMARY, including source URL and collection timestamp.	Provides desktop crawl reports and exports suited to its broader crawler environment.
Integrations and automation	Fits an Apify API or scheduled workflow; the same input can be rerun as a bounded cloud job.	Documents scheduling and integrations within the desktop suite and its licensed features.
Reliability evidence	Reviewed release evidence reconciles saved Dataset rows with page-event charges for public test cases.	This Actor does not make a comparative reliability claim for the desktop product.

For the precise alternative boundary: use this Actor for public HTML crawl exports and an Apify-native data handoff. Choose the broader suite when browser rendering, authenticated environments, local crawl history, visualizations, advanced custom extraction, or its other desktop capabilities are central to the job.

Best results

• Use sites and paths you are authorized to collect, with a clear output purpose.
• Start at 10–50 pages, validate the overview view, then increase maxPages only when the saved rows match the audit need.
• Keep the direct mode for public sites that respond normally; choose residential mode only after observing repeatable direct rate limits and confirming the extra platform cost is appropriate.
• Enable sitemap seeding when coverage matters more than the homepage link graph.
• Compare subsequent runs using contentHash, canonical, title, and issue counts, while retaining each row's scrapedAt and startUrl provenance.

Builder's note

I designed this Actor around the part of a spider crawl that is easiest to operationalize: a reliable, one-row-per-URL handoff with source provenance and explicit terminal state. I kept the first version focused on raw HTML so the output contract, billing unit, and downstream repair workflow remain clear instead of implying a browser-rendering or full-suite replacement.

Legal and responsible use

Use this Actor only for public pages you are authorized to collect, follow applicable laws and site terms, honor robots.txt when appropriate, and keep results in your own compliance workflow.

Screaming Frog is a trademark of its owner. This independent Actor is not affiliated, associated, or endorsed by Screaming Frog.

Use the Actor Issues tab on Apify for feature requests or support.

Pricing and run costs

This Actor uses Pay per event plus Apify platform usage. The Pricing tab lists the current event rates and billing terms.

Event	Billing unit	When it applies
apify-actor-start	Actor Start	Charged when the Actor starts running. Number of events charged depends on Actor memory (one event per GB, minimum one event).
page-crawled	Page crawled	Charged once per crawled URL whose on-page SEO row is saved to the dataset.

Run cost combines the charged events and Apify platform usage. Review the run charge limit and requested result count before starting.

Connect an AI agent

Use the Apify MCP configurator to choose an available client connection. Inspect this Actor’s current input schema and required credentials before running it.