GitHub Repo to RAG

Turn any public GitHub repository into a clean, structured dataset of its code and documentation. Built for AI coding assistants, codebase Q&A bots, RAG pipelines and LLM fine-tuning.

Why this is different

No API token. No rate limits. Most GitHub scrapers walk the REST API, which allows 60 requests an hour without a token, so they either crawl painfully slowly or demand you create a personal access token. This Actor downloads the repository archive in a single request, then extracts locally. A 50 MB repo with thousands of files costs one HTTP call.

The noise is already removed. Feeding a raw repo to a model wastes most of your context on things that carry no meaning: node_modules, dist, build, vendor, __pycache__, .next, coverage output, and lockfiles like package-lock.json, yarn.lock, Cargo.lock and go.sum. Minified bundles and source maps go too. All of it is excluded by default.

Binary files can't slip through. Extension checks alone miss binaries with text-like names, so every file is checked for null bytes before it's kept.

Language is tagged for you, so you can filter or chunk by language without inferring it from the extension later.

What you get

Field	Description
path	Path within the repository
name	File name
extension	File extension
language	Detected language, e.g. typescript, python, markdown
isDocumentation	True for .md, .mdx, .rst, .txt, .adoc
content	Full file contents
lines	Line count
sizeBytes	File size
url	Direct link to the file on GitHub
repository / repositoryUrl	Source repo
scrapedAt	ISO timestamp

Example input

Documentation only, which is usually what you want for a support or docs bot:

{
  "repoUrl": "apify/crawlee",
  "includeDocs": true,
  "includeCode": false
}

Just TypeScript source, for a code assistant:

{
  "repoUrl": "https://github.com/apify/crawlee",
  "extensions": ["ts", "tsx"],
  "excludePattern": "\\.spec\\.|\\.test\\.",
  "maxFiles": 2000
}

Options

• Repository — full URL or owner/repo
• Branch — leave blank to try main, then master
• Include documentation / Include source code — the two common modes
• Only these extensions — overrides both toggles when you want something specific
• Skip paths matching — a regular expression against the file path, e.g. ^test/
• Max files and Max file size — cap the run so cost is predictable, and keep generated bundles out of your corpus

Notes

• Public repositories only. No token is used, so nothing private is accessible.
• Supported out of the box: JavaScript, TypeScript, Python, Ruby, Go, Rust, Java, Kotlin, Swift, C, C++, C#, PHP, Scala, shell, SQL, Vue, Svelte, plus YAML, TOML and JSON config.