GitHub Repo Docs Scraper for RAG & AI Agents

Fetch documentation from GitHub repositories — READMEs, docs folders, and custom paths — and convert to clean, chunked markdown optimized for RAG pipelines and AI agents.

Perfect for building RAG systems over your own repos, analyzing open-source documentation, and feeding code docs to AI agents.

Features

• README extraction — fetch and chunk any repo's README
• Docs folder traversal — recursively scan docs/, documentation/, doc/, and custom paths
• Markdown chunking — header-aware splitting with configurable chunk size and overlap
• Rich metadata — every chunk includes file path, line numbers, heading, and token count
• Optional GitHub token — supports authenticated requests (5000 req/hr vs 60 unauthenticated)
• Dual-mode — run as a normal Apify Actor OR connect as an MCP server for Claude, Cursor, and other AI agents
• No browser required — uses the GitHub REST API directly

Use Cases

• RAG pipelines — chunk repo docs and ingest into vector databases for Q&A over code documentation
• AI agents — let Claude/Cursor read repo documentation via MCP tools
• Documentation analysis — compare docs across repos, find gaps, track changes
• Onboarding — generate summaries of a repo's documentation for new contributors
• Search — keyword search across a repo's entire documentation

Input (Normal Actor Mode)

Field	Type	Description
repoUrl	string	GitHub repo URL or owner/repo shorthand
githubToken	string	Optional PAT for higher rate limits
includeReadme	boolean	Fetch README (default true)
includeDocsFolder	boolean	Scan docs/ directory (default true)
docsPaths	array	Additional paths to scan
maxFiles	integer	Max files to fetch (default 50, max 500)
chunkSize	integer	Words per chunk (default 1000)
chunkOverlap	integer	Overlap between chunks (default 200)

Output

{
  "repo": "facebook/react",
  "filePath": "README.md",
  "fileName": "README.md",
  "fileType": "md",
  "size": 5317,
  "url": "https://github.com/facebook/react/blob/main/README.md",
  "contentMarkdown": "# React · ...",
  "chunks": [
    {
      "id": "facebook_react_README_md_0",
      "text": "# React ...",
      "tokens": 465,
      "filePath": "README.md",
      "startLine": 1,
      "endLine": 10,
      "heading": "React"
    }
  ]
}

MCP Tools

When connected as an MCP server, the following tools are available:

Tool	Description
get_readme	Fetch and chunk a repo's README.
get_docs	Fetch and chunk files from a docs folder.
search_repo_docs	Fetch docs and filter by keyword.

Pricing

Pay-per-event. First results are free; subsequent events billed at:

Event	Price
File fetched	$0.003
Chunk generated	$0.001
MCP tool call	$0.01
Actor start	$0.00005

Volume discounts available at higher tiers.