# GitHub Repo to RAG Chunks & LLM Context (`mmdincer/github-repo-rag-extractor`) Actor

Turn any public or private GitHub repo into token-counted, line-cited RAG chunks and one LLM-ready context file. One archive download, no API quota, secrets masked, license and commit included. Not affiliated with GitHub.

- **URL**: https://apify.com/mmdincer/github-repo-rag-extractor.md
- **Developed by:** [Muhammet Mustafa Dincer](https://apify.com/mmdincer) (community)
- **Categories:** AI, Developer tools, Automation
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.10 / 1,000 processed repositories

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

![GitHub Repo to RAG Chunks and LLM Context: one repository becomes a repository row and token-counted chunks with file and line citations](https://api.apify.com/v2/key-value-stores/vbQuJ5uhWVgUngr57/records/github-rag-banner.png?signature=1mHrG8dtWk5GlR7VuHtTR)

### What is GitHub Repo to RAG Chunks & LLM Context?

**GitHub Repo to RAG Chunks & LLM Context** turns a GitHub repository into data an LLM pipeline can use. Give it one repository, a hundred, or the ID of another Actor's dataset, and you get back:

- **one row per repository** (`recordType: "repository"`) with the **exact commit**, detected **license**, language breakdown, file and token counts, what was skipped and why, a folder tree and, when GitHub answers, stars, topics and description;
- **one row per chunk** (`recordType: "chunk"`) with the chunk text, its **token count**, the **file path and line range** it came from (`lib/router/index.js:120-188`) and a GitHub link **pinned to the commit**, so a generated answer can cite the exact lines instead of "somewhere in the repo";
- optionally **one row per file** with the whole text, and **one Markdown context file per repository** (folder tree plus files in reading order, README first, cut at a token budget you choose) that you can paste straight into an LLM chat.

It needs no browser, no GitHub account for public repositories and no cloning. It downloads one archive per repository over plain HTTP and reads it in JavaScript.

### Quick answer: when should you (or an AI agent) pick this Actor?

**Pick this Actor when** you are building a retrieval pipeline over code and docs and you need chunks, not a text dump: a "chat with this library" assistant, a code-search index, a docs Q\&A bot, an onboarding agent for a codebase, or a nightly job that re-indexes a list of repositories. It is also the right tool when your repository links already sit in an Apify dataset (from a GitHub search or trending Actor) and you want them chunked without copying links by hand.

**Pick something else when** you want GitHub *data* rather than file *content*: issues, pull requests, contributors, star history, release notes or user profiles. This Actor reads the files of a repository at one commit and does not return the issue tracker, the git history or any person's data. For that, use a GitHub data scraper.

**Defaults that work:** send `{"repositories": ["https://github.com/expressjs/express"]}` and you get one repository row plus 595 chunks at 512 tokens each (213 files, 191,908 tokens) in about four seconds. For a vector store send `{"repositories": [...], "mode": "chunks", "chunkSize": 512, "chunkOverlap": 64}` and the dataset is exactly what you feed to embeddings. To read only the docs, add `"paths": ["docs", "README.md"]`.

### At a glance

| | |
|---|---|
| **Does** | Downloads a repository archive and returns token-counted chunks with file path, line range and a commit-pinned link, a repository row (commit, license, counts, skipped files) and a Markdown context file |
| **Input** | `repositories` (or `datasetId` to read links from another dataset). Optional: `mode`, `ref`, `paths`, `includePatterns`, `excludePatterns`, `chunkSize`, `chunkOverlap`, `redactSecrets`, `includeMetadata`, `buildContextFile`, `contextTokenBudget`, `maxRepositories`, `maxFilesPerRepository`, `maxFileKb`, `maxArchiveMb`, `maxChunksPerRepository`, `githubToken` |
| **Output** | `recordType: "repository"`, `"chunk"` and (optional) `"file"` rows in one dataset, plus `CONTEXT-owner-repo.md` and `RUN_SUMMARY` in the key-value store |
| **Needs** | A public `github.com` repository. A GitHub token only for private repositories and for reliable metadata |
| **Speed** | Measured on the platform: 7 files in 3 s, 213 files in 4 s, 500 files and 3,373 chunks of a very large repository in 10 s |
| **Pricing** | Pay per event: $0.003 per repository with readable files, $0.0001 per file read, $0.0005 per run on the Free plan. Chunks are free. Failed, missing, oversized and empty repositories are free |
| **Does not** | Return issues, pull requests, history or people; read submodules or Git LFS files; run code; clone over SSH; access anything but `github.com` |

### Key terms used in this README

- **Repository row:** a dataset row with `recordType: "repository"`. One per repository reference, always written, even when the repository failed. The failure reason is in the row.
- **Chunk row:** a dataset row with `recordType: "chunk"`. A run of whole lines from one file, up to `chunkSize` tokens, with `path`, `lineStart`, `lineEnd`, a ready-made `citation` and a `sourceUrl`.
- **Commit SHA:** the exact commit the archive was built from, read from the archive itself. Chunk links are pinned to it, so they keep pointing at the same text after the branch moves.
- **Token count:** the number of tokens the text becomes in **cl100k_base**, the tokenizer used by OpenAI's embedding and chat models. It is counted, not estimated from characters.
- **Overlap:** how many tokens (in whole lines) from the end of one chunk are repeated at the start of the next, so a function that straddles a boundary is still retrievable.
- **Context file:** one Markdown file per repository in the key-value store: directory tree, then each file under its path in reading order (README, license, docs, source, tests), cut when the token budget is reached.
- **Secret masking:** replacing private keys and well-known token formats with `[REDACTED:type]` before anything is written.

### What can you use it for?

#### RAG and vector stores over code and docs

`mode: "chunks"` is a ready-made embedding input: one row per chunk, inside your token budget, with `repository`, `commitSha`, `path` and `language` as metadata. Store `lineStart` and `lineEnd` next to the vector and an answer can cite `src/server.ts:44-97`.

#### "Chat with this library" agents

Point an agent at this Actor through MCP, give it a repository link and it receives citable chunks and a context file instead of an HTML page. The rows are flat and identical in shape, so the agent can filter and quote without knowing anything about Git.

#### Dropping a whole codebase into a long-context model

`buildContextFile` writes one Markdown file with the directory tree and the files in priority order. `contextTokenBudget` (default 100,000) stops it at the size your model accepts, and `contextFileTokens` on the repository row tells you the exact count.

#### Re-indexing on a schedule

Pass `ref` as a tag or commit for reproducible results, or leave it empty to follow the default branch. `commitSha` on every row tells your pipeline whether anything changed since the last run.

#### Chaining after a GitHub search

Feed the dataset ID of a GitHub search, topic or trending Actor into `datasetId`. The Actor finds the repository link in each row (`repositoryUrl`, `url`, `html_url`, `fullName` and similar fields) and returns chunks for every repository.

#### Licence and size triage

The repository row carries the detected `license` (and whether it came from the LICENSE text or the GitHub API), `tokenCount`, `languages` and `skippedFiles`, which is enough to decide whether a repository is worth indexing before you embed it.

### How does it work?

1. The repository list is built from `repositories` and, if you set one, from the links found in `datasetId`. Each entry is parsed (`https://github.com/owner/repo`, `owner/repo`, `owner/repo@v1.2.0`, `/tree/<branch>/<folder>` links) and duplicates are removed. Entries that are not a `github.com` repository get a row with `failureReason: "invalid_reference"` and cost nothing.
2. For each repository one archive is downloaded from `codeload.github.com`, the host Git tooling and package managers use for snapshots. This needs **no GitHub API call**, so the 60 requests an hour that GitHub gives anonymous API users do not apply. With a token, the archive comes from the official REST API instead.
3. The archive is streamed to disk with a size cap, and the commit SHA is read from it. Files are decompressed one at a time.
4. Files are chosen: lockfiles, build output, vendored and cache folders, binaries, minified or generated files, empty files and anything over `maxFileKb` are skipped, and every skip is counted by reason on the repository row. Your `paths`, `includePatterns` and `excludePatterns` narrow it further.
5. Files are read in priority order (README, license, docs, source, tests), so a file limit cuts tests before it cuts the README. Secrets are masked, text is normalised, and each file is split into chunks of whole lines. Chunks break before a heading or a top-level declaration when they can, then at a blank line, and mid-block only when a block is larger than the budget.
6. Rows are written. The repository row is charged once and each file read is charged once; chunks are free. A missing repository, a refusal from GitHub, an oversized archive and a repository with nothing readable write a row and cost nothing.

If GitHub answers 403 or 429 the Actor **stops** and says so. It does not retry around a refusal.

### How to turn a GitHub repository into RAG chunks (step by step)

1. Open the Actor and paste one or more **GitHub repositories**, one per line, for example `https://github.com/expressjs/express`.
2. Leave **What to return** on *Chunks (RAG)*, or choose *Whole files* or *Chunks and whole files*.
3. Set **Chunk size** to 512 for general retrieval. Use 256 for narrow lookups and 1024 or more to keep whole functions together.
4. Optionally narrow the run with **Folders or files** (for example `docs`) or include and exclude patterns (for example `*.py`, `**/*.test.js`).
5. Click **Start**. In the **Output** tab open **Repositories** to see the commit, license and counts, or **Chunks** for the embedding input. The Markdown context file is in the key-value store under `CONTEXT-owner-repo.md`.
6. Download the dataset as JSON, JSONL, CSV or Excel, or read it from the API. `recordType` tells the row kinds apart.

### Input

| Field | Type | Default | What it does |
|---|---|---|---|
| `repositories` | list of text | `expressjs/express` | `https://github.com/owner/repo`, `owner/repo`, `owner/repo@ref`, or a `/tree/<ref>/<folder>` link. Up to 100 per run |
| `datasetId` | text | empty | Read repository links from an existing Apify dataset. Looks for `repositoryUrl`, `url`, `html_url`, `fullName` and similar fields |
| `mode` | choice | `chunks` | `chunks` writes the repository row and chunk rows; `files` writes whole files; `both` writes everything |
| `ref` | text | default branch | Branch, tag or commit. A ref inside the repository link wins |
| `paths` | list of text | empty | Only these folders or files, from the repository root |
| `includePatterns` | list of text | empty | Glob patterns a file must match, for example `*.py` or `src/**/*.ts` |
| `excludePatterns` | list of text | empty | Glob patterns to leave out. Exclude always wins |
| `chunkSize` | number | 512 | Token budget per chunk (64 to 4096), counted in cl100k_base |
| `chunkOverlap` | number | 64 | Tokens repeated from the previous chunk, at most half of `chunkSize` |
| `redactSecrets` | yes/no | yes | Replace private keys and well-known token formats with `[REDACTED:type]` |
| `includeMetadata` | yes/no | yes | Add description, stars, forks, topics and last push from the GitHub API. May be skipped when GitHub's anonymous limit is reached; the files are still complete |
| `buildContextFile` | yes/no | yes | Save `CONTEXT-owner-repo.md` per repository to the key-value store |
| `contextTokenBudget` | number | 100000 | Token budget of the context file |
| `maxRepositories` | number | 100 | Upper bound on repositories processed in one run. Extra repositories are left out and counted in `RUN_SUMMARY` |
| `maxFilesPerRepository` | number | 500 | Stop after this many files, in priority order |
| `maxFileKb` | number | 256 | Skip files larger than this |
| `maxArchiveMb` | number | 100 | Refuse repositories whose archive is larger than this |
| `maxChunksPerRepository` | number | 5000 | Safety cap on chunk rows for one repository |
| `githubToken` | secret | empty | Needed only for private repositories and reliable metadata. Stored encrypted, never written to the dataset or the log |

### Output

Every run writes one dataset. Filter on `recordType`.

![Output dataset: one repository row and its chunks, with citations and commit links](https://api.apify.com/v2/key-value-stores/vbQuJ5uhWVgUngr57/records/github-rag-sample-output.png?signature=rSp7LtmEk1e0Ki8omI7r)

**Repository row.** Real output for `expressjs/express` limited to the `lib` folder and `Readme.md` (folder tree shortened):

```json
{
  "recordType": "repository",
  "input": "https://github.com/expressjs/express",
  "repository": "expressjs/express",
  "url": "https://github.com/expressjs/express",
  "status": "processed",
  "failureReason": null,
  "message": null,
  "owner": "expressjs",
  "name": "express",
  "ref": null,
  "commitSha": "43fca52d28a422c8be48fd489cbd4f2d6152bab6",
  "subPath": "lib, Readme.md",
  "description": "Fast, unopinionated, minimalist web framework for node.",
  "stars": 69582,
  "forks": 25365,
  "topics": ["express", "javascript", "nodejs", "server"],
  "primaryLanguage": "JavaScript",
  "defaultBranch": "master",
  "license": "MIT",
  "licenseSource": "license_file",
  "metadataStatus": "ok",
  "fileCount": 7,
  "archiveFileCount": 216,
  "chunkCount": 52,
  "tokenCount": 19100,
  "languages": { "JavaScript": { "files": 6, "tokens": 16034 }, "Markdown": { "files": 1, "tokens": 3066 } },
  "skippedFiles": { "not_selected": 209 },
  "truncated": false,
  "secretsRedacted": 0,
  "directoryTree": "lib/\n  application.js\n  express.js\n  …\nReadme.md",
  "contextFileKey": "CONTEXT-expressjs-express.md",
  "contextFileTokens": 19272
}
```

**Chunk row.** From the same run (text shortened):

```json
{
  "recordType": "chunk",
  "repository": "expressjs/express",
  "ref": null,
  "commitSha": "43fca52d28a422c8be48fd489cbd4f2d6152bab6",
  "path": "lib/express.js",
  "language": "JavaScript",
  "chunkIndex": 0,
  "fileChunkCount": 1,
  "text": "/*!\n * express\n * Copyright(c) 2009-2013 TJ Holowaychuk\n …",
  "tokenCount": 376,
  "charCount": 1604,
  "lineStart": 1,
  "lineEnd": 81,
  "citation": "lib/express.js:1-81",
  "sourceUrl": "https://github.com/expressjs/express/blob/43fca52d28a422c8be48fd489cbd4f2d6152bab6/lib/express.js#L1-L81",
  "headings": null,
  "isPartial": false
}
```

**Field reference**

| Field | Row | Meaning |
|---|---|---|
| `status` | repository | `processed` (charged), `empty` (downloaded, nothing readable, free) or `failed` (free) |
| `failureReason` | repository | `invalid_reference`, `not_found`, `token_rejected`, `blocked`, `too_large`, `network_error`, `http_error`, `corrupt_archive`, `unavailable_for_legal_reasons`, `no_readable_files` |
| `commitSha` | all | Commit the archive was built from; links are pinned to it |
| `license`, `licenseSource` | repository | SPDX id from the LICENSE text (`license_file`) or the GitHub API (`github_api`). A keyword heuristic, not legal advice |
| `fileCount`, `archiveFileCount` | repository | Files read and charged, and all files in the archive |
| `skippedFiles` | repository | Counts by reason: `excluded_directory`, `lockfile`, `minified_or_generated`, `binary`, `too_large`, `empty`, `not_utf8_text`, `not_selected`, `file_limit` |
| `truncated`, `truncatedReason` | repository | A limit stopped the repository early: `file_limit`, `chunk_limit`, `text_size_limit` or `budget` |
| `secretsRedacted`, `secretTypes` | repository, file | How many secret-looking values were masked, and which kinds |
| `metadataStatus` | repository | `ok`, `unavailable` (GitHub API refused; files are still complete) or `disabled` |
| `path`, `language` | chunk, file | File path and language from the file name |
| `lineStart`, `lineEnd`, `citation` | chunk | 1-based line range and a `path:start-end` string to show next to an answer |
| `sourceUrl` | chunk, file | GitHub link to the exact lines at the exact commit |
| `tokenCount` | all | cl100k_base tokens in the text of that row |
| `headings` | chunk | For Markdown-like files, the headings the chunk sits under or contains |
| `isPartial` | chunk | True when a single line was longer than the chunk size and had to be cut mid-line |

### How much does it cost?

The Actor uses pay-per-event pricing. On the Free plan:

| Event | Price | When it is charged |
|---|---|---|
| Actor Start | $0.0005 | Once per run (per GB of memory) |
| Processed repository | $0.003 | Per repository that had at least one readable file |
| Processed file | $0.0001 | Per file read, masked and chunked |

Chunks, whole-file rows, the context file and metadata are free, and so is a repository that fails, is missing, is too large or has nothing readable. Paid plans get lower prices per event.

Measured examples, charged at Free-plan prices (platform runs on 2026-11-04):

| Run | Files | Chunks | Tokens | Run time | You pay |
|---|---|---|---|---|---|
| `expressjs/express`, folders `lib` and `Readme.md` (the prefill) | 7 | 52 | 19,100 | 3 s | $0.0042 |
| `expressjs/express`, whole repository | 213 | 595 | 191,908 | 4 s | $0.0248 |
| `pallets/click`, whole repository | 169 | 964 | 323,134 | 6 s | $0.0204 |
| `facebook/react`, first 500 files by priority, chunks and whole files | 500 | 3,373 | 1,175,868 | 10 s | $0.0535 |

To cap spending, lower `maxFilesPerRepository` or `maxRepositories`, narrow the run with `paths`, or set a maximum cost per run in the Apify run options: the Actor stops cleanly when the limit is reached and says so in the repository row (`truncatedReason: "budget"`).

### How does it compare with other GitHub-to-LLM Actors?

Checked on the Apify Store on 2026-11-04. These Actors solve a related problem and some of them are the better choice for a simpler job.

| | This Actor | Gitingest-style converters (for example `automation-lab/github-repository-to-llm-context-gitingest`) | `get_anything/github-repo-scraper` | README-only extractors |
|---|---|---|---|---|
| One context file for an LLM | Yes, with a token budget | Yes, this is their main output | File rows | README only |
| Token count | Exact cl100k_base, per row | Not provided by some; estimates by others | Approximate | No |
| Embedding-ready chunks with line ranges | Yes | No | No | No |
| Commit pinned in every row and link | Yes | No | No | No |
| Needs GitHub API calls | No (one archive download). A token is optional | Varies | Yes, one request per file, 60 requests an hour without a token | Varies |
| Secret masking | Yes, on by default | Not documented | Not documented | No |
| Issues, pull requests, stars history | No | No | Some metadata | No |
| Price (Free plan) | $0.003 per repository + $0.0001 per file | Starts at about $0.0001 per run | About $0.0001 per file | Varies |

In short: if you need one text file to paste into a chat and nothing else, a Gitingest-style converter is simpler and cheaper. If you need **chunks you can embed, cite and re-index**, with exact token counts and a commit you can trust, this Actor is built for that.

### Is it legal to extract a GitHub repository?

**This Actor reads public repository files that you name. It is not a crawler.**

- It downloads the archive from `codeload.github.com` (or the official REST API when you give a token), not from `github.com` web paths that GitHub's `robots.txt` disallows for archives. It does not log in, does not collect user profiles, commit authors or e-mail addresses, and does not follow links.
- A repository's **content belongs to its authors under the license in the repository**. The `license` field tells you what the maintainers declared; it does not grant you any right. Check the license before you redistribute code, train a model on it or ship it in a product. Many licenses require attribution; some forbid commercial use.
- Respect GitHub's [Terms of Service](https://docs.github.com/en/site-policy/github-terms/github-terms-of-service) and [Acceptable Use Policies](https://docs.github.com/en/site-policy/acceptable-use-policies/github-acceptable-use-policies). You are responsible for what you do with the data.
- If GitHub refuses a download (HTTP 403 or 429) the Actor stops and reports `blocked`. It does not rotate IPs or try to get around the limit.

Not affiliated with, endorsed by or sponsored by GitHub, Inc. GitHub is a trademark of GitHub, Inc.

### Can it mask secrets?

By default the Actor replaces private keys and well-known token formats (AWS access keys, GitHub tokens, Slack tokens, Google API keys, Stripe live keys, OpenAI keys, npm tokens, Twilio keys) with `[REDACTED:type]` before any text is written. The repository row says how many values were masked and of which kind. This is a **pattern check, not a scanner**: a secret in a format that is not listed is left alone, so do not rely on it to publish a private repository. Turn it off with `redactSecrets: false` if you need the raw text.

### Using the API

#### MCP (AI agents)

**Apify MCP server (ChatGPT, Claude, Cursor and other MCP clients).** Connect `https://mcp.apify.com`, then:

1. `search-actors` with a query such as `GitHub repository to RAG chunks`.
2. `fetch-actor-details` for `mmdincer/github-repo-rag-extractor` to read the input schema.
3. `call-actor` with the input below.
4. `get-dataset-items` with the returned dataset ID and `recordType: "chunk"` as your filter.

```json
{
  "actor": "mmdincer/github-repo-rag-extractor",
  "input": {
    "repositories": ["https://github.com/expressjs/express"],
    "mode": "chunks",
    "chunkSize": 512,
    "chunkOverlap": 64,
    "paths": ["lib", "Readme.md"]
  },
  "callOptions": { "maxTotalChargeUsd": 0.5 }
}
```

To give an MCP client this Actor as its own tool (no search step), connect `https://mcp.apify.com?tools=mmdincer/github-repo-rag-extractor`.

#### JavaScript

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('mmdincer/github-repo-rag-extractor').call({
    repositories: ['https://github.com/expressjs/express'],
    mode: 'chunks',
    chunkSize: 512,
    chunkOverlap: 64,
    paths: ['lib', 'Readme.md'],
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
const chunks = items.filter((item) => item.recordType === 'chunk');
console.log(chunks[0].citation, chunks[0].tokenCount);
```

#### Python

```python
from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("mmdincer/github-repo-rag-extractor").call(run_input={
    "repositories": ["https://github.com/expressjs/express"],
    "mode": "chunks",
    "chunkSize": 512,
    "chunkOverlap": 64,
    "paths": ["lib", "Readme.md"],
})
items = client.dataset(run["defaultDatasetId"]).list_items().items
chunks = [item for item in items if item["recordType"] == "chunk"]
print(chunks[0]["citation"], chunks[0]["tokenCount"])
```

#### curl

```bash
curl -s -X POST "https://api.apify.com/v2/acts/mmdincer~github-repo-rag-extractor/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"repositories": ["https://github.com/expressjs/express"], "paths": ["lib", "Readme.md"]}'
```

The context file of a run is a key-value store record: `https://api.apify.com/v2/key-value-stores/<defaultKeyValueStoreId>/records/CONTEXT-expressjs-express.md`.

### FAQ

#### Do I need a GitHub account or token?

Not for public repositories. A token (a personal access token with read access to contents) is needed for **private repositories** and makes the optional metadata reliable. Without a token, GitHub limits anonymous API use to 60 requests an hour per IP, which a shared platform can exhaust; the Actor still downloads and chunks every file, and the repository row says `metadataStatus: "unavailable"`.

#### Why not just `git clone` and run a tokenizer myself?

You can, and for one repository on your laptop that is the cheapest route. This Actor is for the cases where you cannot or do not want to: n8n, Make or agent workflows, many repositories on a schedule, a consistent output shape across repositories, PDFs and other sources, and no Git or tokenizer installation.

#### How accurate is the token count?

It is counted with the cl100k_base tokenizer, which OpenAI's embedding and chat models use, not estimated from characters. Other model families tokenize differently, so use the count as a budget guide for them.

#### Which files are skipped?

Lockfiles, build and dependency folders (`node_modules`, `dist`, `build`, `vendor`, `target` and similar), binaries and images, minified and generated files (`*.min.js`, `*.map`, protobuf output), empty files, files that are not UTF-8 text and files over `maxFileKb`. The repository row counts every skip by reason, so nothing disappears silently.

#### Does it include Git history, submodules or LFS files?

No. The archive holds the files of one commit. Submodule content is not included and Git LFS files appear as their small pointer text.

#### How do I read a single branch, tag or commit?

Use `owner/repo@v1.2.0`, a `/tree/<branch>/<folder>` link or the `ref` field. Branch names that contain slashes work. A commit SHA gives reproducible output.

#### Why did a repository come back `not_found`?

GitHub answers 404 for repositories that do not exist and for private ones you cannot see. Check the spelling, or add a `githubToken` that can read it. A wrong tag name also gives 404.

#### What if the repository is huge?

The archive is refused before anything is read when it exceeds `maxArchiveMb` (default 100 MB), and the row says `too_large`. Narrowing with `paths` does not shrink the download, so raise the limit deliberately. Within the archive, `maxFilesPerRepository`, `maxFileKb` and `maxChunksPerRepository` keep a run predictable, and `truncated` tells you when one of them applied.

#### Is the output deterministic?

For the same commit, `ref` and settings, yes: files are read in a fixed order and chunk boundaries do not depend on timing.

#### Where can I report a problem or request a feature?

Use the **Issues** tab of the Actor in Apify Console. Include the input, the run ID and the repository row.

# Changelog

This Actor's version history is a separate document: https://apify.com/mmdincer/github-repo-rag-extractor/changelog.md

# Actor input Schema

## `repositories` (type: `array`):

One per line. Accepts <code>https://github.com/owner/repo</code>, a link into a branch or folder (<code>…/tree/main/docs</code>), <code>owner/repo</code> and <code>owner/repo@v1.2.0</code> (branch, tag or commit). Up to 100 per run. Forks, private repositories (with a token) and archived repositories all work.

## `datasetId` (type: `string`):

Optional. Pick an Apify dataset whose items contain GitHub repository links, for example the output of a GitHub search or trending Actor. The Actor looks for <code>repositoryUrl</code>, <code>url</code>, <code>html_url</code>, <code>fullName</code> and similar fields, so you can chain it without copying links by hand. The dataset is opened read-only.

## `mode` (type: `string`):

<b>Chunks</b> writes one repository row plus one row per token-counted chunk, which is what an embedding job wants. <b>Whole files</b> writes one row per file with its full text. <b>Both</b> writes everything. The Markdown context file is controlled separately below.

## `chunkSize` (type: `integer`):

Token budget per chunk, counted with the cl100k_base tokenizer that OpenAI embedding and chat models use. 512 is a good default for code and docs; use 256 for narrow lookups and 1024 or more to keep whole functions together.

## `chunkOverlap` (type: `integer`):

Text from the end of one chunk repeated at the start of the next, in whole lines, so a function that straddles a boundary is still findable. 64 by default. At most half the chunk size.

## `ref` (type: `string`):

Read this ref instead of the default branch. A <code>/tree/…</code> link or <code>owner/repo@ref</code> in the repository list wins over this field. Pin a tag or a commit SHA for reproducible results.

## `paths` (type: `array`):

Only read these folders or files (paths from the repository root, one per line, case does not matter). Empty means the whole repository. Reading <code>docs</code> alone costs only the files in <code>docs</code>.

## `includePatterns` (type: `array`):

Glob patterns a file must match to be read, for example <code>*.py</code> or <code>src/\*\*/*.ts</code>. A pattern without a slash matches the file name anywhere; case does not matter. Empty means everything the default filters allow.

## `excludePatterns` (type: `array`):

Glob patterns to leave out, for example <code>\*\*/\*.test.ts</code> or <code>docs/</code> (a trailing slash means the folder). Exclude always wins over include.

## `redactSecrets` (type: `boolean`):

Replace private keys and well-known token formats (AWS, GitHub, Slack, Stripe, OpenAI, Google, npm, Twilio) with <code>\[REDACTED:type]</code> before the text is written. This is a pattern check, not a guarantee; do not rely on it to publish a private repository.

## `includeMetadata` (type: `boolean`):

Add description, stars, forks, topics, default branch and last push from the GitHub API with one request per repository. Anonymous API access is limited to 60 requests an hour per IP, so on a busy platform this can be skipped: the row then says <code>metadataStatus: unavailable</code> and the files are still complete. A GitHub token makes it reliable.

## `buildContextFile` (type: `boolean`):

Save one <code>CONTEXT-owner-repo.md</code> per repository to the key-value store: directory tree plus the files in reading order (README first), cut at the token budget below. Paste it straight into an LLM chat.

## `contextTokenBudget` (type: `integer`):

The context file stops adding files when this many tokens are reached, so it always fits the model you target. 100000 fits most long-context models.

## `maxRepositories` (type: `integer`):

Upper bound on how many repositories are processed in one run, after the list and the dataset are merged. Extra repositories are left out and counted in the run summary. A low number is a simple way to cap the cost of a first run.

## `maxFilesPerRepository` (type: `integer`):

Stop after this many files. Files are read in priority order (README, license, docs, source, tests), so a limit cuts tests and examples before it cuts the README.

## `maxFileKb` (type: `integer`):

Skip files larger than this. Large files are usually data dumps, generated code or logs.

## `maxArchiveMb` (type: `integer`):

Refuse repositories whose compressed archive is larger than this, before reading anything. For a huge repository, a folder from the Folders field does not shrink the download, so raise this limit deliberately.

## `maxChunksPerRepository` (type: `integer`):

Safety cap on chunk rows for one repository. A very small chunk size on a very large repository would otherwise write tens of thousands of rows. The repository row says <code>truncated</code> when the cap is hit.

## `githubToken` (type: `string`):

A personal access token with read access to contents. Needed only for private repositories and for reliable metadata. With a token the archive comes from the official GitHub API and counts against your own rate limit. The token is stored encrypted, never written to the dataset or the log.

## Actor input object example

```json
{
  "repositories": [
    "https://github.com/expressjs/express"
  ],
  "datasetId": "wL9ZqWz8Kk3vJm2Rt",
  "mode": "chunks",
  "chunkSize": 512,
  "chunkOverlap": 64,
  "ref": "v5.1.0",
  "paths": [
    "lib",
    "Readme.md"
  ],
  "includePatterns": [
    "*.md",
    "*.ts"
  ],
  "excludePatterns": [
    "**/*.test.js",
    "fixtures/"
  ],
  "redactSecrets": true,
  "includeMetadata": true,
  "buildContextFile": true,
  "contextTokenBudget": 100000,
  "maxRepositories": 100,
  "maxFilesPerRepository": 500,
  "maxFileKb": 256,
  "maxArchiveMb": 100,
  "maxChunksPerRepository": 5000
}
```

# Actor output Schema

## `repositories` (type: `string`):

One row per repository with the commit, license, metadata, file and token counts, what was skipped and why, and the reason for any failure. Filter on recordType repository.

## `chunks` (type: `string`):

One row per token-counted chunk with its text, file path, line range and a commit-pinned GitHub link. Filter on recordType chunk. This is the view to send to an embedding model.

## `files` (type: `string`):

One row per file with its whole text, in files and both modes. Filter on recordType file.

## `summary` (type: `string`):

How many repositories were processed, failed or had no readable files, how many files, chunks and tokens were written, and why the run stopped.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "repositories": [
        "https://github.com/expressjs/express"
    ],
    "mode": "chunks",
    "chunkSize": 512,
    "chunkOverlap": 64,
    "paths": [
        "lib",
        "Readme.md"
    ],
    "redactSecrets": true,
    "includeMetadata": true,
    "buildContextFile": true,
    "contextTokenBudget": 100000,
    "maxRepositories": 100,
    "maxFilesPerRepository": 500,
    "maxFileKb": 256,
    "maxArchiveMb": 100,
    "maxChunksPerRepository": 5000
};

// Run the Actor and wait for it to finish
const run = await client.actor("mmdincer/github-repo-rag-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "repositories": ["https://github.com/expressjs/express"],
    "mode": "chunks",
    "chunkSize": 512,
    "chunkOverlap": 64,
    "paths": [
        "lib",
        "Readme.md",
    ],
    "redactSecrets": True,
    "includeMetadata": True,
    "buildContextFile": True,
    "contextTokenBudget": 100000,
    "maxRepositories": 100,
    "maxFilesPerRepository": 500,
    "maxFileKb": 256,
    "maxArchiveMb": 100,
    "maxChunksPerRepository": 5000,
}

# Run the Actor and wait for it to finish
run = client.actor("mmdincer/github-repo-rag-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "repositories": [
    "https://github.com/expressjs/express"
  ],
  "mode": "chunks",
  "chunkSize": 512,
  "chunkOverlap": 64,
  "paths": [
    "lib",
    "Readme.md"
  ],
  "redactSecrets": true,
  "includeMetadata": true,
  "buildContextFile": true,
  "contextTokenBudget": 100000,
  "maxRepositories": 100,
  "maxFilesPerRepository": 500,
  "maxFileKb": 256,
  "maxArchiveMb": 100,
  "maxChunksPerRepository": 5000
}' |
apify call mmdincer/github-repo-rag-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,mmdincer/github-repo-rag-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/QMyN5NG9nvbaMYndu/builds/rdsmvMjEO8dctskm7/openapi.json