API Documentation Crawler for RAG & AI Agents

Crawl API documentation sites and extract clean, structured content for AI models, RAG pipelines, and vector databases. Use it as a normal Apify Actor or connect it to Claude, Cursor, and other MCP-compatible clients.

What it does

This Actor crawls API documentation websites and returns structured records for each page:

• Title & description
• Heading hierarchy (with anchor IDs)
• Markdown-formatted content
• Plain text (for simple embedding)
• Detected API endpoints (method + path, when visible on the page)
• Code examples
• Outbound links
• Metadata (canonical URL, OpenGraph tags)

You can also switch to chunking mode to get pre-split text snippets ready for vector DB ingestion.

Why use this?

• Feed RAG pipelines with clean, structured API docs.
• Power AI agents that need to answer questions about an API.
• Avoid brittle copy-paste from docs into LLM context windows.
• MCP-native: call it as a tool from Claude, Cursor, ChatGPT, and other MCP clients.

How to use

Apify Console / API

Provide one or more start URLs and optional limits:

{
  "startUrls": ["https://docs.apify.com/api/v2"],
  "maxPages": 20,
  "maxDepth": 2,
  "outputFormat": "full"
}

MCP mode

Connect the Actor as an MCP server in Cursor/Claude Desktop:

{
  "mcpServers": {
    "api-docs-crawler": {
      "url": "https://AHampton83--api-docs-rag-crawler.apify.actor/mcp"
    }
  }
}

Available tools:

• crawl_api_docs(startUrl, maxPages?, maxDepth?) — crawl docs and return structured page summaries.
• get_page_chunks(startUrl, maxPages?, chunkSize?, chunkOverlap?) — return pre-chunked text for vector DBs.

Output schema

Full format (outputFormat: "full")

{
  "url": "https://docs.example.com/api/v1/users",
  "title": "Users API",
  "description": "Manage users via the REST API",
  "headings": [{ "level": 1, "text": "Users API", "id": "users-api" }],
  "markdown": "# Users API\n\n## Get all users\n\n`GET /api/v1/users`",
  "text": "Users API Get all users GET /api/v1/users ...",
  "endpoints": [{ "method": "GET", "path": "/api/v1/users", "confidence": "high" }],
  "codeBlocks": [{ "language": "bash", "content": "curl /api/v1/users" }],
  "links": ["https://docs.example.com/api/v1/orders"],
  "metadata": { "canonicalUrl": "...", "ogTitle": "...", "ogDescription": "..." },
  "crawledAt": "2026-07-04T00:00:00.000Z"
}

Chunks format (outputFormat: "chunks")

{
  "chunkIndex": 0,
  "url": "https://docs.example.com/api/v1/users",
  "title": "Users API",
  "text": "...",
  "headings": [{ "level": 2, "text": "Get all users" }],
  "tokenEstimate": 250
}

Pricing

This Actor uses Pay Per Event (PPE) billing:

Event	Price	Notes
apify-actor-start	$0.00005	Platform-provided synthetic start event
page-crawled	$0.005	First 10 pages per run/tool-call are free
chunk-generated	$0.001	First 10 chunks per run/tool-call are free
tool-call	$0.01	Per MCP tool call

Volume discounts apply based on the user's Apify plan tier.

Publish & monetize

To activate monetization and publish this Actor on Apify Store:

1. Open the Actor in Apify Console: https://console.apify.com/actors/Dtlhc1snKkwxoikZh
2. Go to the Publication tab and click Set up monetization.
3. In the pricing step, keep apify-actor-start and add these custom events with the exact slugs used in the code:
   • page-crawled — $0.005
   • chunk-generated — $0.001
   • tool-call — $0.01
4. Do not enable apify-default-dataset-item: the Actor already charges manually per page/chunk, so the synthetic dataset-item event would double-charge users.
5. Complete the Primary event and Review steps, then submit for Store approval.

Payouts require a verified developer account with tax and payment details in Apify Console.

Limits

• maxPages: up to 500 per normal run, 100 per MCP tool call.
• Crawls only same-origin links by default (configurable via includeUrlGlobs).
• Static HTML only — Cheerio crawler does not execute JavaScript.

Use cases

• Build a custom API assistant for your product.
• Keep vector store embeddings in sync with live docs.
• Generate SDK examples from existing documentation.
• Competitive analysis of public APIs.

Support

Open an issue or reach out via the Apify Store page.