# Audit documentation crawl readiness for RAG

**Use case:** 

Assess crawl readiness for documentation before RAG ingestion. Get insights on robots.txt, sitemaps, static HTML, and estimated crawl costs.

## Input

```json
{
  "startUrl": "https://docs.apify.com",
  "sitemapUrl": "https://docs.apify.com/sitemap.xml",
  "urls": [],
  "maxOrigins": 1,
  "samplePages": 10,
  "maxSitemapUrls": 500,
  "maxSitemapDepth": 2,
  "sameDomainOnly": true,
  "respectRobotsTxt": true,
  "checkSitemaps": true,
  "checkSamplePages": true,
  "estimateApifyCosts": true,
  "includeRecommendedSettings": true,
  "includeDisallowedPaths": true,
  "includeSampleRows": true,
  "userAgent": "WebsiteCrawlPreflightBot/1.0 (+https://apify.com)",
  "urlAllowlistPatterns": [],
  "urlBlocklistPatterns": [],
  "requestTimeoutSecs": 20,
  "maxConcurrency": 5,
  "costModel": {
    "computeCostPerCuUsd": 0.2,
    "staticFetchCuPerPageLow": 0.001,
    "staticFetchCuPerPageHigh": 0.003,
    "highCostThresholdUsd": 25
  }
}
```

## Output

```json
{
  "domain": {
    "label": "Domain",
    "format": "string"
  },
  "crawlabilityScore": {
    "label": "Score",
    "format": "integer"
  },
  "recommendation": {
    "label": "Recommendation",
    "format": "string"
  },
  "robotsAccessStatus": {
    "label": "Robots status",
    "format": "string"
  },
  "urlsDiscovered": {
    "label": "URLs discovered",
    "format": "integer"
  },
  "sampleSuccessRate": {
    "label": "Sample success rate",
    "format": "number"
  },
  "estimatedCostHighUsd": {
    "label": "Estimated high cost (USD)",
    "format": "number"
  },
  "useBrowserRendering": {
    "label": "Use browser rendering",
    "format": "boolean"
  },
  "flagCodes": {
    "label": "Flag codes",
    "format": "array"
  },
  "checkedAt": {
    "label": "Checked at",
    "format": "string"
  }
}
```

## About this Actor

This example demonstrates how to use [Website Crawl Preflight Auditor](https://apify.com/sebastian-actors/website-crawl-preflight-auditor) with a specific input configuration. Visit the [Actor detail page](https://apify.com/sebastian-actors/website-crawl-preflight-auditor) to learn more, explore other use cases, and run it yourself.