Website Crawl Preflight Auditor

Audit robots.txt, sitemaps, static HTML, JavaScript risk, crawl limits, and estimated Apify costs before starting a scraper, SEO audit, or RAG job.

Website Crawl Preflight Auditor performs a bounded, static HTTP-only inspection of up to 10 public website origins. It produces one crawl-readiness audit per origin, explicit outcomes for direct sample URLs, scoped cost scenarios, and practical settings for a downstream crawler.

What It Checks

• robots.txt access state, user-agent rules, crawl delay, sitemap declarations, and representative disallow paths
• sitemap discovery, index depth, URL counts, metadata coverage, malformed entries, gzip bounds, and parse or fetch errors
• HTTP status, redirects, content type, decoded response size, title, description, canonical, H1s, and robots meta
• static text volume, scripts, JSON-LD, and app-shell markers such as __NEXT_DATA__, __NUXT__, ng-version, id="root", and id="app"
• likely JavaScript dependence, noindex pages, canonical mismatches, non-HTML URLs, redirects, and low-content pages
• full-site and recommended-scope static-fetch cost ranges
• recommended page limits, concurrency, request delay, timeouts, allowlists, blocklists, and rendering mode

Input

Provide at least one of startUrl, sitemapUrl, or urls. Inputs may span multiple origins; each origin is audited independently and processed sequentially.

{
  "startUrl": "https://docs.apify.com",
  "urls": [
    "https://blog.apify.com/",
    "https://www.example.com/about"
  ],
  "maxOrigins": 10,
  "samplePages": 25,
  "maxSitemapUrls": 100000,
  "maxSitemapDepth": 2,
  "sameDomainOnly": true,
  "respectRobotsTxt": true,
  "checkSitemaps": true,
  "checkSamplePages": true,
  "estimateApifyCosts": true,
  "includeRecommendedSettings": true,
  "includeSampleRows": true
}

Limits apply independently per origin:

• up to 10 explicit origins per run
• up to 500 direct URLs per input
• up to 250 sampled pages per origin
• up to 500,000 retained sitemap URLs per origin
• sitemap recursion depth up to 3
• sample concurrency up to 10 within one origin

sameDomainOnly keeps sitemap records and redirects on the exact audit origin, including scheme and port. It does not prevent you from supplying several explicit origins.

Use regex filters to narrow the recommended crawl scope:

{
  "sitemapUrl": "https://example.com/sitemap.xml",
  "urlAllowlistPatterns": ["/docs/", "/products/"],
  "urlBlocklistPatterns": ["/search", "/tag/", "\\?sort="],
  "samplePages": 50
}

Outputs

The default dataset contains one crawl_preflight_audit row per origin. A fully assembled robots-blocked or network-failed report is a completed audit because it still contains the promised evidence. Unsafe input, an internal processing failure, or a charge-limited origin produces an uncharged error row.

The separate samples dataset contains URL-level rows with status: ok | skipped | error. Every explicit URL receives an outcome, including:

• SAMPLE_LIMIT_REACHED
• SAMPLE_CHECK_DISABLED
• URL_NOT_ALLOWLISTED
• URL_BLOCKLISTED
• OUTSIDE_AUDIT_ORIGIN
• robots, safety, timeout, response-size, redirect, HTTP, and network failures

Only sitemap URLs selected for sampling receive URL-level rows. Other sitemap URLs remain aggregate counts so output stays bounded.

Output shortcuts:

• audits: the default audit dataset view
• samples: the separate URL-outcomes dataset view
• runSummary: the aggregate OUTPUT record

OUTPUT reports origin totals, completed and error counts, sample totals, billing state, charged-domain count, warnings, cost guardrails, and the complete audit summaries.

Robots Behavior

robots.txt is reported as one of:

• found: usable rules were fetched and evaluated for the configured user agent
• unavailable: a 4xx response; crawling is allowed
• unreachable: a network failure or 5xx; crawling is disallowed
• invalid: unusable or oversized content; crawling is disallowed

This follows RFC 9309 availability behavior. Rules, crawl delay, and disallow-path samples are scoped to the configured user agent. Robots decisions are cached per origin, and redirects are checked against the destination origin's rules before being followed.

Network Safety

The Actor only accesses public http and https targets. It rejects URL credentials and loopback, private, link-local, reserved, IPv4-mapped IPv6, or mixed public/private DNS results. URL and DNS safety are enforced on connections and redirects.

Decoded responses are bounded:

• robots.txt: 512 KiB
• HTML samples: 5 MiB
• sitemap XML, including decompressed .xml.gz: 50 MiB

Declared oversized responses are rejected before reading the body, streamed responses are aborted when they cross the limit, and the timeout remains active through body consumption. DOCTYPE and entity declarations are rejected in sitemap XML.

Cost Estimates

Cost output separates:

• fullSite: estimated static-fetch cost using total sitemap discovery
• recommendedScope: estimated cost after retained URL limits, filtering, and the recommended maxPages

The default model assumes $0.20 per compute unit as a Free/Starter baseline and 0.001 to 0.003 compute units per static-fetched page. You can override the model in costModel. Estimates exclude browser rendering and proxies, and actual prices vary by plan tier, retries, response sizes, storage, and platform pricing.

When robots.txt declares a crawl delay, the recommendation sets maxConcurrency to 1 and exposes the delay as requestDelaySecs.

Crawlability Score

The score is a triage signal, not a guarantee:

• 90-100: excellent static crawl candidate
• 75-89: good candidate with minor warnings
• 55-74: crawlable with constraints
• 30-54: risky or needs manual review/browser rendering
• 0-29: blocked or not recommended

Flags and their evidence are more important than the score. Stable flags include ROBOTS_TXT_UNREACHABLE, ROBOTS_TXT_INVALID, ROBOTS_DISALLOWS_TARGET, SITEMAP_FETCH_FAILED, SITEMAP_INDEX_TOO_DEEP, LARGE_SITEMAP, SAMPLE_LOW_SUCCESS_RATE, JAVASCRIPT_HEAVY_LIKELY, LOW_STATIC_TEXT_CONTENT, MANY_NOINDEX_PAGES, CANONICAL_MISMATCHES, REDIRECT_HEAVY, and COST_ESTIMATE_HIGH.

Billing

Pay-per-event runs charge once for each completed origin audit using the domain-audited event. The Actor checks remaining event capacity before starting an origin. When capacity is exhausted, remaining origins receive minimal CHARGE_LIMIT_REACHED rows and no further requests are made.

Invalid or private inputs, internal failures, and budget-limited origins are not charged. An origin blocked by robots or affected by fetch failures is charged only when the Actor has completed and assembled its evidence report.

Recommended Workflow

1. Run Website Crawl Preflight Auditor with representative input.
2. Review robots access, discovered versus retained sitemap counts, sample outcomes, flags, and both cost scenarios.
3. Apply the recommended scope, delay, concurrency, timeout, allowlist, and blocklist settings to the downstream Actor.
4. Use browser rendering only when static HTML evidence shows it is needed.

V1 Limitations

V1 intentionally excludes browser rendering, proxies, recursive link crawling, authentication, screenshots, and LLM calls. It does not bypass anti-bot systems, crawl login-gated content, perform a full SEO audit, or guarantee exact costs.

Only use this Actor on public websites you are allowed to analyze. You remain responsible for applicable laws, website terms, and crawl policies.