Pricing

from $0.10 / 1,000 results

Try for free

Go to Apify Store

GitHub Repo Scraper - Code & Docs to LLM-Ready Text

Try for free

Chat with a repo: turn any GitHub repository into LLM-ready text - README, file tree and source files as clean text with token counts, plus repo metadata and optional issues. Feed a codebase to ChatGPT/Claude or a RAG pipeline. Official GitHub API, no token required. Export JSON, CSV or Markdown.

Pricing

from $0.10 / 1,000 results

Rating

0.0

(0)

Developer

Get Anything

Actor stats

Bookmarked

Total users

Monthly active users

20 hours ago

Last modified

GitHub Repo Scraper — Code & Docs to LLM-Ready Text

Chat with a repo. Turn any GitHub repository into clean, LLM-ready text: the README, the file tree, and the contents of selected source/doc files — each with a token count — plus repo metadata and (optionally) recent issues. Built for feeding a codebase into ChatGPT / Claude or a RAG pipeline.

Why this one

AI coding assistants and RAG-over-code are one of the biggest LLM use cases. This Actor gives you the whole repo as structured text in one run — no cloning, no git, no parsing HTML. It uses GitHub's official REST API plus raw.githubusercontent.com, so it's fast and reliable. Works without a token (60 req/h); add one for 5,000 req/h.

What it does

Scrape multiple repos by owner/repo or URL.
README + file tree + file contents (filtered by glob patterns).
Sensible include/exclude defaults (code + docs in; node_modules, lockfiles, binaries out).
Caps for max files, file size, and chars per file to control token cost.
Optional issues as text.
Export to JSON, CSV, or Markdown, or pull via the Apify API.

Input

Field	Description
`repos`	Repositories as `owner/repo` or GitHub URLs.
`includeReadme` / `includeFileTree` / `includeFileContents`	What to return.
`includePatterns` / `excludePatterns`	Glob filters (defaults provided).
`maxFiles` / `maxFileSizeKB` / `maxCharsPerFile`	Token-budget caps.
`includeIssues` / `maxIssues`	Optionally pull issues.
`githubToken`	Optional PAT for a higher rate limit.

Example

{
  "repos": ["apify/apify-sdk-python"],
  "includeReadme": true,
  "includeFileContents": true,
  "includePatterns": ["*.py", "*.md"],
  "maxFiles": 50,
  "maxFileSizeKB": 100
}

Output

One repo record (metadata + README + tree), then one file record per selected file:

{ "resultType": "file", "repo": "apify/apify-sdk-python", "path": "src/apify/actor.py",
  "content": "from __future__ ...", "tokensApprox": 1820,
  "url": "https://github.com/apify/apify-sdk-python/blob/master/src/apify/actor.py" }

Use cases

RAG over a codebase — chunk file.content into a vector DB, ask questions about the repo.
LLM code review / onboarding — paste README + key files into ChatGPT/Claude.
Docs generation — summarise a repo from its README and source.
Dependency / license audits — pull metadata across many repos.

FAQ

Do I need a token? No — but a token raises the limit to 5,000 req/h for large runs.

How are binary files handled? Skipped (size cap + a binary guard); only text is returned.

YouTube Transcript & Subtitles Scraper API

johnvc/YoutubeTranscripts

Scrape YouTube transcripts, subtitles, and captions in bulk, the cheapest pay-per-video YouTube transcript API on Apify. Callable from any MCP client (Claude, Cursor, ChatGPT). Supports YouTube videos, Shorts, and every URL format.

John

3.2K

4.5

(10)

AI Context Scraper

wheat_tourist/ai-context-scraper

AI Context Scraper is a production-grade Apify Actor that gathers high-quality coding context from the Web, GitHub, and StackOverflow for AI agents and RAG systems. It uses NVIDIA Nemotron 3 Super to synthesize documents, code snippets, and patterns into actionable implementation guidance.

Varun Chopra

GitHub Scraper - Repos, Developers & Contact Leads

scrapesage/github-scraper

Scrape GitHub via the official API: search repositories & developers, get full repo metadata, README, languages, topics, stars & activity, plus developer/org profiles and contributor & stargazer leads with emails. Developer lead-gen + monitoring. No browser.

Scrape Sage

GitHub Repo Scraper | 24 Fields, README & Release, No Token

themineworks/github-repo-intelligence

Deep metadata for any public GitHub repository: stars, forks, language breakdown, topics, license, README, contributors count, latest release, commit recency. No token required. MCP ready for AI agents doing repo due diligence.

The Mine Works

GitHub Repository README Scraper

parseforge/github-repo-readme-scraper

Scrape README content from GitHub repositories by search query or direct URL. Returns full Markdown body, rendered text, stars, language, license, and topics for each repo.

ParseForge

Website Content Crawler for AI — Clean Markdown, 4x Cheaper

joyouscam35875/website-content-crawler

Crawl any website and extract clean text/markdown for LLMs, RAG pipelines, vector databases. BFS crawl with depth control, robots.txt support, boilerplate removal. Perfect for feeding AI models. $0.001/page — 4x cheaper than the official Apify crawler.

Ken Digital

GitHub Repository Scraper - Stars, Topics, Trending

logiover/github-repository-scraper

Scrape GitHub repos by search query and export stars, topics, forks & license to CSV/JSON. GitHub data export without an API key - trending repos scraper.

Logiover

Google News Scraper - Articles, Sources & Monitoring

scrapesage/google-news-scraper

Scrape Google News by keyword, topic, top headlines or city. Get real publisher URLs (decoded, not redirect links), source, date, snippet, related coverage and full article text, author & image. Monitor mode returns only new articles. Export JSON, CSV, Excel.

Scrape Sage

Bing Copilot AI Answers API | Microsoft AEO Monitoring

johnvc/bing-copilot-api

Track Bing Copilot AI answers for any query: Microsoft AI-generated answers, structured text blocks, cited sources, and brand mentions as clean JSON. The fourth engine in the AEO suite with Google AI Overview, Naver AI Overview, and Brave AI Mode. Pay per query. MCP-ready for AI agents.

John

5.0

(1)

Site Crawler: Website → Markdown Corpus for LLM/RAG

boxbox10/site-crawler

Crawl a whole website or docs site and get one clean, LLM-ready Markdown + JSON record per page (title, headings, content, links, token count). Built for RAG ingestion and AI knowledge bases.

Marvin Eguilos

PDF Extractor: PDF → Clean Markdown + JSON for LLM/RAG

boxbox10/pdf-extractor

Turn any PDF URL into clean, LLM-ready Markdown + structured JSON (title, metadata, per-page text, page count, word count, token count). Perfect for RAG pipelines, AI agents, and LLM document ingestion.

Marvin Eguilos