Website Crawl Preflight Auditor

What is Website Crawl Preflight Auditor?

Website Crawl Preflight Auditor inspects public website origins through robots.txt, XML sitemaps, and a bounded sample of static HTML. Start with a site URL, sitemap URL, direct sample URLs, or a combination; it returns per-origin audits, URL-level outcomes, and a run summary so you can scope a crawler, SEO audit, or RAG ingestion job.

It is a static HTTP preflight, not a recursive crawler. It does not render JavaScript, use proxies, authenticate, take screenshots, or call an LLM.

• Evaluates robots availability, user-agent rules, crawl delay, declared sitemaps, and up to 25 disallow paths.
• Discovers explicit, robots-declared, and /sitemap.xml sources, including indexes and gzip XML.
• Counts URLs and sitemap metadata coverage.
• Samples direct URLs first, then distributes sitemap samples across path groups.
• Reports HTTP/HTML metadata, text, scripts, JSON-LD, redirects, and app-shell markers.
• Produces deterministic flags, a 0-100 triage score, and optional downstream crawler settings.
• Estimates full-site and filtered-scope static-fetch costs; estimates are not guarantees.

What data can it extract or produce?

Audit rows use itemType, status, domain, origin, checkedAt, crawlabilityScore, recommendation, robots, sitemaps, samplePages, costEstimate, recommendedSettings, flags, error, and charged. Flattened fields include robotsAccessStatus, urlsDiscovered, sampleSuccessRate, javascriptHeavyLikely, estimatedScopePages, estimatedCostHighUsd, recommendedMaxPages, useBrowserRendering, and flagCodes.

sample_page rows use itemType, status, attempted, domain, url, finalUrl, statusCode, contentType, allowedByRobots, title, textChars, scriptCount, flags, error, and checkedAt; additional fields hold redirects, HTML metadata, hashes, and diagnostics.

What can you use the data for?

• Decide whether documentation suits static RAG ingestion or needs browser rendering.
• Define page caps, URL filters, concurrency, delay, and timeout before a large crawl.
• Find robots or sitemap problems before SEO crawling or migration validation.
• Estimate the scope and static-fetch cost range of a public site or selected content section.
• Triage a portfolio of public sites and route blocked, dynamic, or low-quality origins to manual review.

How much does it cost?

The Actor uses pay per event. The live Pricing tab lists a $0.00005 run-start event plus each completed domain-audited event:

Apify plan	Price per completed origin audit
Free	$0.003
Bronze	$0.0025
Silver	$0.002
Gold, Platinum, or Diamond	$0.0015

Actor event charge = $0.00005 + (completed audits x plan rate). Three completed audits on Free produce $0.00905 in Actor events. A completed evidence report can be charged even when its evidence says crawling is blocked; invalid input, per-origin errors, and charge-limited origins are not domain-audited events.

costEstimate.fullSite uses discovered scope and recommendedScope uses filtered scope. Both predict a future static crawl, not this Actor's charge, and exclude proxies and browsers.

How to use Website Crawl Preflight Auditor

1. Choose a site URL, exact sitemap, direct samples, or a combination.
2. Keep robots and same-origin controls enabled unless required otherwise.
3. Start with small sample and sitemap limits.
4. Review flags and evidence before the heuristic score.
5. Adapt recommendedSettings to your downstream crawler.

What input does it accept?

At least one of startUrl, sitemapUrl, or urls is required. Minimal valid Input:

{
  "startUrl": "https://docs.apify.com"
}

Field	Default and behavior
startUrl	Public HTTP(S) page; enables the exact origin's /sitemap.xml fallback.
sitemapUrl	Public HTTP(S) sitemap or index. A different origin becomes a separate audit.
urls	[]; up to 500 direct URLs. Fragments and exact normalized duplicates are removed.
maxOrigins	10; range 1-10. Scheme, host, and port define an origin; origins run sequentially.
samplePages	25; range 1-250 per origin. Direct URLs come first; more pages mean more requests.
maxSitemapUrls	100000; range 1-500000 retained URLs per origin. Larger values cost time and memory.
maxSitemapDepth	2; range 0-3. More index depth can add sitemap requests.
sameDomainOnly	true; keeps sitemap records and redirects on the exact origin. Explicit origins remain separate audits.
respectRobotsTxt	true; controls samples and redirects. Robots data is always inspected.
checkSitemaps / checkSamplePages	Both true; disable an unneeded phase.
estimateApifyCosts / includeRecommendedSettings	Both true; disabling one makes its output and flattened fields null.
includeDisallowedPaths	true; includes the user-agent-specific disallow sample.
includeSampleRows	true; false suppresses the samples dataset, not audit diagnostics.
userAgent	WebsiteCrawlPreflightBot/1.0 (+https://apify.com); used for robots and HTTP.
urlAllowlistPatterns	[]; JavaScript regexes on the full normalized URL; at least one must match.
urlBlocklistPatterns	[]; matches are excluded after allowlisting. Invalid regexes fail validation.
requestTimeoutSecs	20; range 5-60 per request attempt, including body consumption.
maxConcurrency	5; range 1-10 for samples only. Sitemaps and origins are sequential.
costModel	Optional: computeCostPerCuUsd 0.2, staticFetchCuPerPageLow 0.001, staticFetchCuPerPageHigh 0.003, highCostThresholdUsd 25. Values are positive; high must be at least low.

What output does it produce?

Output link audits opens the default per-origin dataset; samples opens the URL-level dataset. Key-value link runSummary opens OUTPUT, whose fields are actorName, runStartedAt, runFinishedAt, durationSeconds, originsTotal, originsCompleted, originsFailed, chargedDomains, billingEnabled, totals, audits, warnings, and costGuardrails.

Compact ragReadiness projection for the same scenario; values remain source-dependent:

{
  "domain": "docs.apify.com",
  "crawlabilityScore": 96,
  "recommendation": "Excellent static crawl candidate",
  "robotsAccessStatus": "found",
  "urlsDiscovered": 1718,
  "sampleSuccessRate": 1,
  "estimatedCostHighUsd": 0.3,
  "useBrowserRendering": false,
  "flagCodes": ["LOW_LASTMOD_COVERAGE"]
}

In audit error rows, crawlabilityScore, recommendation, costEstimate, recommendedSettings, estimatedScopePages, estimatedCostHighUsd, recommendedMaxPages, and useBrowserRendering are null; error is null on completed rows. In samples, finalUrl, statusCode, contentType, title, and error can be null.

See the run's Output tab for full records and schemas. Export either dataset as JSON, JSONL, CSV, Excel, XML, RSS, or HTML table; JSON/JSONL best preserves nesting. OUTPUT is JSON in the default key-value store.

How are errors and partial results handled?

One origin can fail without discarding others. Origin failures use status: "error"; samples use ok, skipped, or error; sitemap problems remain in sitemaps.errors and run warnings. A completed audit can contain critical flags: completion means the report was assembled, not that the site is crawlable.

robotsAccessStatus is found, unavailable (4xx, allowed), unreachable (network/5xx, disallowed), or invalid (unusable, disallowed).

• Empty samples: check sample toggles, filters, origin scope, sitemap candidates, and sitemaps.errors; add direct urls if needed.
• Partial results: inspect robots status, flags, sample error.code, HTTP/size errors, and sitemap limits.
• Duplicates: fragments and exact normalized duplicates are removed, but query strings, trailing-slash variants, schemes, and ports can remain distinct.
• Failed runs: check required input, regexes, origin count, public HTTP(S), credentials, and private or mixed DNS results.

What are the limits, performance, and best practices?

HTML is capped at 5 MiB, robots.txt at 512 KiB, and sitemap XML, including decompressed gzip, at 50 MiB. Redirects stop after five hops. Retryable sample and sitemap requests can run up to three attempts. Source latency, rate limits, blocking, and input size determine runtime; throughput is not guaranteed.

Keep origin and robots controls enabled, begin with 10-25 samples, and allowlist large sitemaps. Raising sample, URL, or depth limits increases work. A robots crawl delay makes recommended concurrency 1. Test a browser crawler downstream when JavaScript-heavy flags appear.

How can you use the API, scheduling, webhooks, and integrations?

The API tab generates REST, Node.js, and Python examples. Both clients can run sebastian-actors/website-crawl-preflight-auditor, read its datasets, and fetch key OUTPUT.

Use Schedules for recurring checks and webhooks for notifications or downstream crawls. n8n, Make, and Zapier can route the JSON into SEO or RAG workflows.

Frequently asked questions

Does it crawl the whole website?

No. It reads bounded sitemaps and selected samples, without recursively following page links.

Does it render JavaScript or use a proxy?

No. It only detects signals that browser rendering may be needed.

Is the crawlability score a guarantee?

No. It reflects observed responses and settings; sites, DNS, robots rules, and blocking can change. Review the evidence.

Is costEstimate my charge for this run?

No. It models a downstream static crawl; this Actor is billed by the events above.

Is it legal and responsible to use this Actor?

Use it only for public websites you are authorized to assess. Robots compliance does not determine legality; consider law, copyright, privacy, database rights, site terms, and request rates. Do not target private networks, credentials, login-gated material, or impermissible data. This is not legal advice.

Which related Actors should you use?

• Website Content Crawler recursively extracts RAG content with HTTP or browser modes.
• Web Scraper provides programmable browser extraction for JavaScript pages.

These Actors collect page content; Website Crawl Preflight Auditor is for deciding how, whether, and at what scope to run them.

How do you get support?

Use the Actor's Issues tab. Include the run ID, origin, codes, and sanitized input; never post credentials, tokens, private URLs, or sensitive output.