🤖 GitHub Documentation Intelligence

AI-powered documentation extraction and analysis for GitHub repositories

Extract, structure, and analyze documentation from any GitHub repository in seconds. Perfect for building RAG systems, onboarding developers, and auditing documentation quality.

---

🎯 What It Does

Automatically extracts and structures:

• ✅ README files - Main project documentation
• ✅ Documentation folders - All markdown files from docs/, documentation/, etc.
• ✅ Code documentation - Docstrings from Python, JavaScript, TypeScript files
• ✅ Metadata - Repository info, stars, language, topics
• ✅ Statistics - Word counts, file counts, documentation coverage

---

🚀 Quick Start

Input Example

{
  "url": "https://github.com/pallets/flask",
  "maxFiles": 20,
  "extractCodeDocs": true
}

Output Example

{
  "status": "success",
  "metadata": {
    "name": "flask",
    "description": "The Python micro framework",
    "language": "Python",
    "stars": 65000,
    "url": "https://github.com/pallets/flask"
  },
  "readme": {
    "filename": "README.md",
    "content": "...",
    "sections": [...],
    "word_count": 450
  },
  "documentation_files": [...],
  "code_documentation": [...],
  "combined_markdown": "...",
  "statistics": {
    "has_readme": true,
    "documentation_files_count": 23,
    "code_files_with_docs": 15,
    "total_words": 12500,
    "total_docstrings": 87
  }
}

---

⭐ Key Features

📊 Comprehensive Extraction

• Extracts README, docs folders, and code docstrings
• Supports Python, JavaScript, TypeScript
• Handles nested documentation structures
• Preserves markdown formatting and sections

🎯 Structured Output

• Clean JSON format ready for processing
• Pydantic models for type safety
• Combined markdown for easy reading
• Detailed statistics and metadata

🛡️ Robust & Reliable

• Proper error handling
• Rate limit management
• Partial success handling
• Detailed logging

⚡ Fast & Efficient

• Async operations
• Smart file filtering
• Configurable limits
• Optimized API usage

---

💡 Use Cases

🤖 RAG Systems

Extract clean documentation for training AI models:

# Use extracted docs for embeddings
docs = result['combined_markdown']
chunks = create_embeddings(docs)

👨‍💻 Developer Onboarding

Generate comprehensive repo overviews:

• Understand project structure
• Find key documentation
• Identify important files

📈 Documentation Audits

Analyze documentation quality:

• Check completeness
• Identify gaps
• Track improvements

🔍 Code Search

Enable semantic search over codebases:

• Search through docstrings
• Find relevant code examples
• Understand APIs

---

🔧 Configuration

GitHub Token (Recommended)

For private repos and higher rate limits (5,000 vs 60 requests/hour):

1. Go to https://github.com/settings/tokens
2. Generate new token (classic)
3. Select scopes: repo or public_repo
4. Add to input: "githubToken": "ghp_your_token"

Options

Option	Type	Default	Description
maxFiles	integer	100	Maximum files to process
extractCodeDocs	boolean	true	Extract code docstrings

---

📊 Statistics Provided

• has_readme: Whether README exists
• documentation_files_count: Number of doc files found
• code_files_with_docs: Number of code files with docstrings
• total_words: Total documentation words
• total_lines: Total documentation lines
• total_docstrings: Total docstrings extracted

---

🛠️ Development

Local Testing

# Install dependencies
pip install -r requirements.txt

# Run locally
apify run

Project Structure

.
├── src/
│   ├── main.py           # Actor entry point
│   ├── extractor.py      # Extraction logic
│   ├── models.py         # Data models
│   └── utils.py          # Helper functions
├── .actor/
│   ├── actor.json        # Actor configuration
│   └── input_schema.json # Input schema
├── requirements.txt      # Dependencies
└── Dockerfile           # Container config

---

🤝 Contributing

Issues and pull requests welcome! This is an active project participating in the Apify $1M Challenge.

---

📝 License

Apache 2.0

---

💬 Support

• Questions? Join Apify Discord
• Issues? Open a GitHub issue
• Need help? Check Apify documentation

---

🎯 Coming Soon

• 🔜 Documentation quality scoring (A-F grades)
• 🔜 MCP server for AI agents
• 🔜 Change detection and tracking
• 🔜 Multi-repo comparison
• 🔜 PDF documentation support
• 🔜 Website documentation scraping

##FAQs Q: Why did extraction fail? A: Common reasons: 1.Repository doesn't exist (check URL) 2.Repository is private (add GitHub token) 3.Rate limit exceeded (add token for 5000/hour) 4.Repository is too large (reduce maxFiles)

Q: What if I hit rate limits? A:

Without token: 60 requests/hour With token: 5,000 requests/hour Get token: https://github.com/settings/tokens

Q: Can I extract from private repos? A: Yes! Add your GitHub token in the input: json{ "source": { "url": "...", "githubToken": "ghp_your_token" } } Q: What's the maximum repository size? A:

1.Max 500 files per run 2.Max 5MB per file 3.Max 50MB total data 4.Adjust maxFiles if needed

Q: Why are some files skipped? A: Files are skipped if they: 1.Are too large (>5MB) 2.Can't be decoded (binary files) 3.Cause encoding errors

Q: How long does extraction take? A: 1.Small repos (<100 files): 2-5 seconds 2.Medium repos (100-500 files): 10-30 seconds 3.Large repos (500+ files): 30-60 seconds 4.Max timeout: 4 minutes

---

Built with ❤️ for the Apify $1M Challenge

⭐ If you find this useful, please star the Actor!