# Website to Text & Markdown β€” AI / RAG Content Crawler (`inexhaustible_glass/rag-website-crawler`) Actor Scrape any website into clean text & Markdown with RAG-ready chunks and token counts for LLMs, vector databases (Pinecone, Qdrant) and AI chatbots. Also extracts linked PDF/Word/Excel. Anti-block, robots.txt-aware. Website-to-text for beginners, full RAG pipeline for pros. CPU only, no API key. - **URL**: https://apify.com/inexhaustible\_glass/rag-website-crawler.md - **Developed by:** [Hitman studio](https://apify.com/inexhaustible_glass) (community) - **Categories:** SEO tools, Lead generation, Social media - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, NaN bookmarks - **User rating**: No ratings yet ## Pricing from $5.00 / 1,000 results This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## πŸ•·οΈ RAG Website Crawler β€” Markdown + Chunks + PDFs for AI Turn any website into **clean, LLM-ready data** in one run. Built for RAG pipelines, AI chatbots, and vector databases (Pinecone, Qdrant, Weaviate…). ### Why this one is better | Feature | Plain content crawlers | **RAG Website Crawler** | |---|---|---| | Clean Markdown | βœ… | βœ… | | **Auto chunks + token counts** | ❌ (extra step) | βœ… built-in | | **PDF / Word / Excel extraction** | ❌ skipped | βœ… included | | **Anti-block fetching** | sometimes | βœ… browser TLS + proxy | | **AI summary per page** | ❌ | βœ… optional, **your own key** | | robots.txt + trap protection | varies | βœ… built-in | | GPU needed | β€” | ❌ 100% CPU | ### What you get per page ```json { "url": "https://site.com/docs/intro", "title": "Introduction", "markdown": "## Introduction\n\n...", "word_count": 812, "token_count": 1043, "chunk_count": 3, "chunks": [{ "index": 0, "text": "...", "tokens": 500 }], "is_document": false, "depth": 1, "content_hash": "…", "crawled_at": "2026-06-08T07:00:00Z" } ```` Chunks are ready to embed straight into a vector DB. ### Robust by design Handles the classic crawler traps automatically: - **Infinite loops / calendar traps** β†’ depth + page caps, trap heuristics - **Duplicate URLs / content** β†’ URL normalisation + content-hash dedup - **robots.txt & crawl-delay** β†’ respected (toggle) - **Rate limits / blocks** β†’ polite delay + jitter + proxy + 429 backoff - **Huge pages / memory** β†’ size cap, HTTP-only (no heavy browser) - **Dead URLs** β†’ limited retries, never re-queued ### Input (key options) - `startUrls` β€” where to begin - `maxPages`, `maxDepth`, `sameDomainOnly`, `allowSubdomains` - `chunkSizeTokens`, `chunkOverlapTokens` - `includeDocuments` β€” also crawl linked PDFs/Office files - `respectRobotsTxt`, `crawlDelaySeconds`, `useProxy` - `aiProvider` + `aiApiKey` (**BYOK**) β€” optional per-page AI summary ### Privacy The AI summary uses **your own** key (`isSecret`, encrypted, never logged). The Actor never ships any built-in key, so nothing of ours can be exposed. ### What people use this for (search terms) Whether you are a beginner who just wants to **copy a website's text**, or a developer building a production **RAG pipeline**, this Actor fits: - website to text Β· website to markdown Β· scrape website content Β· copy all pages of a website Β· website content downloader Β· website reader Β· extract text from a website Β· web page to text - data for AI Β· LLM-ready data Β· RAG crawler Β· vector database ingestion Β· embeddings input Β· knowledge base builder Β· AI chatbot training data Β· documentation scraper Β· docs to markdown - works with ChatGPT, Claude, Gemini, LangChain, LlamaIndex, Pinecone, Qdrant, Weaviate, Milvus, Supabase Vector - also: PDF scraper Β· crawl PDFs on a website Β· Word/Excel text extraction Β· sitemap crawler Β· whole-site crawler #### Common use cases - Build an **AI chatbot** that answers questions about your website or docs - Feed a **company knowledge base** into a vector database for RAG - Turn **documentation / help centers** into clean Markdown for LLMs - Collect **research content** from many pages into one structured dataset - Extract text from **PDFs and documents** linked across a site # Actor input Schema ## `startUrls` (type: `array`): Website URLs to start crawling from. Example: a docs site, blog, or any site you want as AI/RAG data. ## `maxPages` (type: `integer`): Stop after this many pages are saved (hard cap protects against runaway crawls). ## `maxDepth` (type: `integer`): How many link-hops away from the start URLs to follow. ## `sameDomainOnly` (type: `boolean`): Only follow links on the start URL's domain. ## `allowSubdomains` (type: `boolean`): When staying on-domain, also follow subdomains (blog.site.com). ## `includeDocuments` (type: `boolean`): Also extract text from linked PDF, DOCX, XLSX, PPTX and similar files. ## `crawlSitemaps` (type: `boolean`): Also read the site's sitemap.xml to find every listed page (not just linked ones). ## `includeUrlGlobs` (type: `array`): Optional glob patterns to limit the crawl, e.g. https://site.com/docs/\*\* . Leave empty for all. ## `excludeUrlGlobs` (type: `array`): Optional glob patterns to skip, e.g. \*\*/login or **/tag/** . ## `chunkSizeTokens` (type: `integer`): Target token size of each RAG chunk. ## `chunkOverlapTokens` (type: `integer`): Token overlap between consecutive chunks (helps retrieval). ## `respectRobotsTxt` (type: `boolean`): Honour robots.txt rules and crawl-delay (recommended). ## `crawlDelaySeconds` (type: `integer`): Politeness delay so the target site is not hammered. ## `requestTimeoutSecs` (type: `integer`): Give up on a single page if it does not respond within this many seconds. ## `maxPageSizeMb` (type: `integer`): Skip downloading anything larger than this (protects memory). ## `useProxy` (type: `boolean`): Route requests through Apify Proxy to avoid IP blocks. ## `proxyGroups` (type: `array`): Optional Apify proxy groups, e.g. RESIDENTIAL. ## `aiProvider` (type: `string`): Add a short AI summary per page using YOUR OWN key. Leave blank to skip. ## `aiApiKey` (type: `string`): Your own API key for the chosen provider. Stored encrypted, never logged. ## `aiModel` (type: `string`): Override the default model for the chosen provider. ## Actor input object example ```json { "startUrls": [ { "url": "https://docs.apify.com/academy/web-scraping-for-beginners" } ], "maxPages": 50, "maxDepth": 3, "sameDomainOnly": true, "allowSubdomains": true, "includeDocuments": true, "crawlSitemaps": false, "includeUrlGlobs": [], "excludeUrlGlobs": [], "chunkSizeTokens": 500, "chunkOverlapTokens": 50, "respectRobotsTxt": true, "crawlDelaySeconds": 1, "requestTimeoutSecs": 25, "maxPageSizeMb": 5, "useProxy": false, "proxyGroups": [], "aiProvider": "" } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ { "url": "https://docs.apify.com/academy/web-scraping-for-beginners" } ] }; // Run the Actor and wait for it to finish const run = await client.actor("inexhaustible_glass/rag-website-crawler").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`πŸ’Ύ Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // πŸ“š Want to learn more πŸ“–? Go to β†’ https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": [{ "url": "https://docs.apify.com/academy/web-scraping-for-beginners" }] } # Run the Actor and wait for it to finish run = client.actor("inexhaustible_glass/rag-website-crawler").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("πŸ’Ύ Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # πŸ“š Want to learn more πŸ“–? Go to β†’ https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ { "url": "https://docs.apify.com/academy/web-scraping-for-beginners" } ] }' | apify call inexhaustible_glass/rag-website-crawler --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=inexhaustible_glass/rag-website-crawler", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Website to Text & Markdown β€” AI / RAG Content Crawler", "description": "Scrape any website into clean text & Markdown with RAG-ready chunks and token counts for LLMs, vector databases (Pinecone, Qdrant) and AI chatbots. Also extracts linked PDF/Word/Excel. Anti-block, robots.txt-aware. Website-to-text for beginners, full RAG pipeline for pros. CPU only, no API key.", "version": "1.0", "x-build-id": "vM3eJPfb1K12Th8qb" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/inexhaustible_glass~rag-website-crawler/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-inexhaustible_glass-rag-website-crawler", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/inexhaustible_glass~rag-website-crawler/runs": { "post": { "operationId": "runs-sync-inexhaustible_glass-rag-website-crawler", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/inexhaustible_glass~rag-website-crawler/run-sync": { "post": { "operationId": "run-sync-inexhaustible_glass-rag-website-crawler", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "startUrls" ], "properties": { "startUrls": { "title": "Start URLs", "type": "array", "description": "Website URLs to start crawling from. Example: a docs site, blog, or any site you want as AI/RAG data.", "items": { "type": "object", "required": [ "url" ], "properties": { "url": { "type": "string", "title": "URL of a web page", "format": "uri" } } } }, "maxPages": { "title": "Max pages", "minimum": 1, "maximum": 5000, "type": "integer", "description": "Stop after this many pages are saved (hard cap protects against runaway crawls).", "default": 50 }, "maxDepth": { "title": "Max crawl depth", "minimum": 0, "maximum": 20, "type": "integer", "description": "How many link-hops away from the start URLs to follow.", "default": 3 }, "sameDomainOnly": { "title": "Stay on the same domain", "type": "boolean", "description": "Only follow links on the start URL's domain.", "default": true }, "allowSubdomains": { "title": "Allow subdomains", "type": "boolean", "description": "When staying on-domain, also follow subdomains (blog.site.com).", "default": true }, "includeDocuments": { "title": "Crawl linked documents (PDF/Word/Excel)", "type": "boolean", "description": "Also extract text from linked PDF, DOCX, XLSX, PPTX and similar files.", "default": true }, "crawlSitemaps": { "title": "Discover URLs from sitemap.xml", "type": "boolean", "description": "Also read the site's sitemap.xml to find every listed page (not just linked ones).", "default": false }, "includeUrlGlobs": { "title": "Only crawl URLs matching (glob)", "type": "array", "description": "Optional glob patterns to limit the crawl, e.g. https://site.com/docs/** . Leave empty for all.", "default": [], "items": { "type": "string" } }, "excludeUrlGlobs": { "title": "Skip URLs matching (glob)", "type": "array", "description": "Optional glob patterns to skip, e.g. **/login or **/tag/** .", "default": [], "items": { "type": "string" } }, "chunkSizeTokens": { "title": "Chunk size (tokens)", "minimum": 50, "maximum": 4000, "type": "integer", "description": "Target token size of each RAG chunk.", "default": 500 }, "chunkOverlapTokens": { "title": "Chunk overlap (tokens)", "minimum": 0, "maximum": 1000, "type": "integer", "description": "Token overlap between consecutive chunks (helps retrieval).", "default": 50 }, "respectRobotsTxt": { "title": "Respect robots.txt", "type": "boolean", "description": "Honour robots.txt rules and crawl-delay (recommended).", "default": true }, "crawlDelaySeconds": { "title": "Delay between requests (seconds)", "minimum": 0, "maximum": 30, "type": "integer", "description": "Politeness delay so the target site is not hammered.", "default": 1 }, "requestTimeoutSecs": { "title": "Request timeout (seconds)", "minimum": 5, "maximum": 90, "type": "integer", "description": "Give up on a single page if it does not respond within this many seconds.", "default": 25 }, "maxPageSizeMb": { "title": "Max page size (MB)", "minimum": 1, "maximum": 50, "type": "integer", "description": "Skip downloading anything larger than this (protects memory).", "default": 5 }, "useProxy": { "title": "Use Apify Proxy (anti-block)", "type": "boolean", "description": "Route requests through Apify Proxy to avoid IP blocks.", "default": false }, "proxyGroups": { "title": "Proxy groups", "type": "array", "description": "Optional Apify proxy groups, e.g. RESIDENTIAL.", "default": [], "items": { "type": "string" } }, "aiProvider": { "title": "AI summary provider (optional, BYOK)", "enum": [ "", "nvidia", "openai", "groq" ], "type": "string", "description": "Add a short AI summary per page using YOUR OWN key. Leave blank to skip.", "default": "" }, "aiApiKey": { "title": "AI API key (your own)", "type": "string", "description": "Your own API key for the chosen provider. Stored encrypted, never logged." }, "aiModel": { "title": "AI model (optional)", "type": "string", "description": "Override the default model for the chosen provider." } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```