Audit documentation crawl readiness for RAG
Created by
Sebastián S
Assess crawl readiness for documentation before RAG ingestion. Get insights on robots.txt, sitemaps, static HTML, and estimated crawl costs.
Website Crawl Preflight Auditorsebastian-actors/website-crawl-preflight-auditor
Domain
Score
Recommendation
Robots status
+6 fieldsTextNumberBooleanListObject
Input
Start URL:https://docs.apify.com
Sitemap URL:https://docs.apify.com/sitemap.xml
Sample pages:10
Max sitemap URLs:500
Respect robots.txt:true
Check sitemaps:true
Check sample pages:true
Estimate Apify costs:true
Include recommended settings:true
Output fields
Domain
Score
Recommendation
Robots status
URLs discovered
Sample success rate
Estimated high cost (USD)
Use browser rendering
Flag codes
Checked at
Sign up on Apify01
Create your Apify account to access the Website Crawl Preflight Auditor.
Start the run02
The Actor will start running based on the input automatically.
Receive the output03
Monitor the progress in real-time. You will be notified as soon as your dataset is complete and ready for review.
Integrate into your workflow04
The final output is delivered in JSON, CSV, or Excel format, ready to be plugged into your workflow.
