Website Crawl Test — Block & Cost Risk

Test a website before you spend time and compute on a full crawl.

This Actor performs a lightweight preflight against a website homepage, robots.txt and a bounded number of sitemap files. It tells you whether raw HTTP looks safe, a browser/proxy is likely needed, how large the sitemap appears, and a conservative starting maxPages value.

What it answers

• Is the homepage reachable from the crawler environment?
• Is there an obvious CAPTCHA, access-denied page or rate-limit block?
• Does the site look static enough for HTTP or likely need a browser?
• Does robots.txt advertise a sitemap?
• Roughly how many URLs are exposed by the sitemap sample?
• What is a safe first maxPages value?
• Is the crawl risk low, medium or high?

Why this exists

Full website crawlers are already a heavily validated category on Apify. The official Website Content Crawler has about 11K monthly active users and public issues reporting slow runs, integrations timing out, crawls stuck for hours, and unexpectedly high compute consumption. A cheap preflight can catch obvious block/size/mode problems before a user launches a large crawl.

Important: no fake cost precision

This Actor does not claim an exact dollar cost. Actual crawl cost depends on page behavior, rendering mode, proxy use, retries and downstream processing. The output is deliberately low / medium / high risk plus recommended starting settings.

Output

• blockRisk: low | medium | high
• crawlerMode: HTTP | Browser | Browser/Proxy likely required
• costRisk: low | medium | high
• recommendedMaxPages
• homepage status and response time
• robots.txt status and sitemap directives
• bounded sitemap file count and URL count
• clear next action

Validation

• Pure logic suite: 10 tests pass against the exact current implementation.
• A remote Runner queue issue prevented a clean final on-device Node test/smoke in the night session; the next Codex review must rerun node --test and node scripts/smoke.js before Store Preview.

Suggested short description

Check blocks, crawler mode and sitemap size before a full website crawl.

Suggested pricing experiment

PPE event crawl-test, initially around $0.01 per website preflight. Treat this as a pricing experiment, not a proven optimum.