AI-Ready Documentation & RAG Ingest

This Actor crawls public documentation, blog, or Notion HTML pages and returns clean Markdown without navigation, scripts, styles, forms, or other boilerplate. Each page includes an exact cl100k_base token count plus stable, overlapping chunks designed for vector ingestion.

Input

{
  "startUrls": ["https://docs.python.org/3/library/asyncio.html"],
  "maxDepth": 3,
  "chunkSize": 1000,
  "chunkOverlapTokens": 100,
  "outputFormat": "MARKDOWN_CHUNKS",
  "maxRetries": 2
}

The crawler stays on each start URL origin, follows same-origin links only, and emits pages as soon as they are processed. It does not execute JavaScript or bypass authentication; JS-only pages may therefore produce an empty-page or fetch diagnostic in RUN_SUMMARY. Public-site markup and anti-bot behavior can change; direct retries are bounded, challenge pages are reported, and CAPTCHAs are never solved or bypassed. Unexpected empty Markdown pages become bounded, query-stripped source warnings instead of silent healthy results.

Output

{
  "url": "https://docs.example/guide/create",
  "title": "Create a charge",
  "markdown": "# Create a charge\nTo charge a credit card...",
  "tokenCount": 842,
  "tokenizer": "cl100k_base",
  "chunks": ["Chunk 1 content...", "Chunk 2 content..."],
  "chunkMetadata": [{
    "chunkId": "9f75ef57c3d8948e6b77e59e6c2f764fce6b5804f63f28e84762eb86d9d96a95",
    "chunkIndex": 0,
    "headingPath": ["Create a charge"],
    "tokenCount": 842,
    "overlapTokenCount": 0
  }],
  "ingestStatus": "ready",
  "isChargeable": true
}

Resource controls

• BFS uses an index cursor instead of repeated shift() and has a hard page/queue cap.
• HTML responses and Markdown per page have byte/character limits before expensive parsing; truncation is explicit in truncated.
• The tokenizer is initialized once per run, and chunking uses bounded text segments instead of retaining every page.
• Pages are not retained after Actor.pushData; only bounded URL sets and the current DOM are live.
• No concurrent request fan-out is used, which limits peak parser memory and source load.

Monetization

The manifest declares documentation-page at $0.0008 per substantive page ($0.80 per 1,000). Empty or low-content pages are returned with isChargeable: false and are not charged.

The published schema includes one official Python documentation prefill with maxDepth: 0 and maxPages: 1, so the Store health test receives one useful page without an uncontrolled crawl. Replace it with your own documentation URLs for production use.

Local development

npm install
npm test
npm run check
apify validate-schema .actor/input_schema.json
apify run --purge