# Docs & Help Center to RAG JSONL (`orbiscribe/docs-help-center-rag-snapshot`) Actor Paste a docs or help center URL and get clean Markdown, breadcrumbs, page records, and JSONL chunks for RAG. - **URL**: https://apify.com/orbiscribe/docs-help-center-rag-snapshot.md - **Developed by:** [Orbiscribe Labs](https://apify.com/orbiscribe) (community) - **Categories:** AI, Developer tools, Automation - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, NaN bookmarks - **User rating**: No ratings yet ## Pricing $1.00 / 1,000 docs page processeds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Docs & Help Center to RAG JSONL Use this Actor when you need a chatbot-ready snapshot of public documentation, a help center, or a developer docs site. It crawls from known docs URLs, extracts readable article content, keeps breadcrumb-style context, detects common documentation platforms, and writes Markdown plus JSONL chunks. ### Why use this instead of a generic crawler? Generic website crawlers are useful when you need every reachable page. This Actor is for docs and help center ingestion, where the useful output is clean article text, stable chunks, breadcrumbs, and a quick page inventory. - paste a docs or help center URL - keep the crawl focused with `includeUrlPatterns` - start with a small live default run - export `DOCS_RAG_CHUNKS_JSONL` directly to a vector pipeline - pay a low flat page price instead of guessing compute-unit cost ### What you get - Dataset rows for docs pages and chunks. - Clean Markdown, main text, headings, links, canonical URL, content hash, platform hint, and inferred breadcrumbs. - Key-value outputs: `RAG_CHUNKS_JSONL`, `DOCS_RAG_CHUNKS_JSONL`, `DOC_PAGES`, `BREADCRUMB_INDEX`, `DOCS_URL_INVENTORY`, `DOCS_MARKDOWN_BUNDLE`, `BUYER_BRIEF`, and `RUN_SUMMARY`. ### Common workflows - Build a support bot from public help center articles. - Snapshot developer docs before loading them into a vector database. - Export docs pages with breadcrumb metadata for citations. - Schedule repeat docs crawls and compare artifacts downstream. ### Input Provide one or more `docsUrls`. Use `maxDepth` and `sameDomainOnly` to control crawl breadth. Use `platformHint` when you know the docs platform and want that stored with the output. The default input runs a tiny live Apify docs sample: ```json { "docsUrls": [ { "url": "https://docs.apify.com/academy/getting-started" }, { "url": "https://docs.apify.com/academy/web-scraping-for-beginners" }, { "url": "https://docs.apify.com/academy/actor-marketing-playbook/actor-basics/actor-description" } ], "includeUrlPatterns": ["/academy/"], "excludeUrlPatterns": [], "maxPages": 5, "maxDepth": 1, "sameDomainOnly": true, "dryRun": false } ```` Use `dryRun: true` when you want bundled demo records without crawling live pages or calling custom pay-per-event charges. ### Pricing Recommended monetization: Pay per Event at **$0.001** per `docs-rag-page`. That is **$1 per 1,000** processed docs pages, plus normal Apify platform usage. When pay-per-event pricing is enabled, dry runs are uncharged and free-plan callers get the first 25 processed sources without this Actor's custom event charge. Users should still set Apify spending limits before large crawls. ### Limits and compliance Public docs only. This Actor does not bypass logins, paywalls, robots policies, or access controls. Output quality depends on the site HTML structure and navigation. # Actor input Schema ## `docsUrls` (type: `array`): Public documentation or help center start URLs. ## `includeUrlPatterns` (type: `array`): Only crawl docs URLs containing at least one of these strings. Leave empty to allow all discovered docs URLs. ## `excludeUrlPatterns` (type: `array`): Skip docs URLs containing any of these strings before maxPages is applied. ## `platformHint` (type: `string`): Optional docs platform hint. ## `maxPages` (type: `integer`): Maximum number of HTML pages to fetch in one run. ## `maxDepth` (type: `integer`): Maximum link-following depth from the provided start or docs URLs. ## `sameDomainOnly` (type: `boolean`): Restrict discovered links to the same hostname as the current page. ## `chunkSizeChars` (type: `integer`): Target character length for generated text chunks. ## `chunkOverlapChars` (type: `integer`): Character overlap between adjacent chunks. ## `requestTimeoutMs` (type: `integer`): Per-request timeout in milliseconds. ## `dryRun` (type: `boolean`): Emit bundled demo docs records without fetching or charging events. ## Actor input object example ```json { "docsUrls": [ { "url": "https://docs.apify.com/academy/getting-started" }, { "url": "https://docs.apify.com/academy/web-scraping-for-beginners" }, { "url": "https://docs.apify.com/academy/actor-marketing-playbook/actor-basics/actor-description" } ], "includeUrlPatterns": [ "/academy/" ], "excludeUrlPatterns": [], "platformHint": "auto", "maxPages": 5, "maxDepth": 1, "sameDomainOnly": true, "chunkSizeChars": 2500, "chunkOverlapChars": 250, "requestTimeoutMs": 15000, "dryRun": false } ``` # Actor output Schema ## `results` (type: `string`): No description ## `ragChunksJsonl` (type: `string`): No description ## `docsChunksJsonl` (type: `string`): No description ## `docPages` (type: `string`): No description ## `breadcrumbIndex` (type: `string`): No description ## `urlInventory` (type: `string`): No description ## `markdownBundle` (type: `string`): No description ## `buyerBrief` (type: `string`): No description ## `runSummary` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = {}; // Run the Actor and wait for it to finish const run = await client.actor("orbiscribe/docs-help-center-rag-snapshot").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = {} # Run the Actor and wait for it to finish run = client.actor("orbiscribe/docs-help-center-rag-snapshot").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{}' | apify call orbiscribe/docs-help-center-rag-snapshot --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=orbiscribe/docs-help-center-rag-snapshot", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Docs & Help Center to RAG JSONL", "description": "Paste a docs or help center URL and get clean Markdown, breadcrumbs, page records, and JSONL chunks for RAG.", "version": "0.1", "x-build-id": "DWJL6HCFWrfkppO9U" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/orbiscribe~docs-help-center-rag-snapshot/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-orbiscribe-docs-help-center-rag-snapshot", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/orbiscribe~docs-help-center-rag-snapshot/runs": { "post": { "operationId": "runs-sync-orbiscribe-docs-help-center-rag-snapshot", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/orbiscribe~docs-help-center-rag-snapshot/run-sync": { "post": { "operationId": "run-sync-orbiscribe-docs-help-center-rag-snapshot", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "docsUrls": { "title": "Docs or help center URLs", "type": "array", "description": "Public documentation or help center start URLs.", "default": [ { "url": "https://docs.apify.com/academy/getting-started" }, { "url": "https://docs.apify.com/academy/web-scraping-for-beginners" }, { "url": "https://docs.apify.com/academy/actor-marketing-playbook/actor-basics/actor-description" } ], "items": { "type": "object", "required": [ "url" ], "properties": { "url": { "type": "string", "title": "URL of a web page", "format": "uri" } } } }, "includeUrlPatterns": { "title": "Include URL patterns", "type": "array", "description": "Only crawl docs URLs containing at least one of these strings. Leave empty to allow all discovered docs URLs.", "default": [ "/academy/" ], "items": { "type": "string" } }, "excludeUrlPatterns": { "title": "Exclude URL patterns", "type": "array", "description": "Skip docs URLs containing any of these strings before maxPages is applied.", "default": [], "items": { "type": "string" } }, "platformHint": { "title": "Platform hint", "enum": [ "auto", "docusaurus", "gitbook", "mintlify", "readme", "zendesk", "intercom", "generic" ], "type": "string", "description": "Optional docs platform hint.", "default": "auto" }, "maxPages": { "title": "Max pages", "minimum": 1, "maximum": 2000, "type": "integer", "description": "Maximum number of HTML pages to fetch in one run.", "default": 5 }, "maxDepth": { "title": "Max link depth", "minimum": 0, "maximum": 5, "type": "integer", "description": "Maximum link-following depth from the provided start or docs URLs.", "default": 1 }, "sameDomainOnly": { "title": "Same domain only", "type": "boolean", "description": "Restrict discovered links to the same hostname as the current page.", "default": true }, "chunkSizeChars": { "title": "Chunk size characters", "minimum": 500, "maximum": 12000, "type": "integer", "description": "Target character length for generated text chunks.", "default": 2500 }, "chunkOverlapChars": { "title": "Chunk overlap characters", "minimum": 0, "maximum": 2000, "type": "integer", "description": "Character overlap between adjacent chunks.", "default": 250 }, "requestTimeoutMs": { "title": "Request timeout ms", "minimum": 1000, "maximum": 60000, "type": "integer", "description": "Per-request timeout in milliseconds.", "default": 15000 }, "dryRun": { "title": "Dry run", "type": "boolean", "description": "Emit bundled demo docs records without fetching or charging events.", "default": false } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```