# FAQ & Help Center Scraper with Change Monitor (`scrapepilot/faq`) Actor Scrape and monitor documentation, FAQs, and help centers effortlessly. Extract structured Q\&As, body text, breadcrumbs, and metadata. Features automatic MD5 change detection to track page updates over time. Fast, async, and powered by cost-effective Pay Per Event billing - **URL**: https://apify.com/scrapepilot/faq.md - **Developed by:** [Scrape Pilot](https://apify.com/scrapepilot) (community) - **Categories:** Developer tools, Automation, Other - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, NaN bookmarks - **User rating**: No ratings yet ## Pricing from $7.00 / 1,000 successful document scrapeds This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README --- ## 📚 FAQ & Help Center Scraper with Change Monitor – RAG-Ready Documentation Intelligence **Crawl any documentation or help center – extract FAQ Q&A pairs, content text, metadata, breadcrumbs, reading time, code snippets, and detect content changes with hash‑based monitoring.** Perfect for **RAG‑pipeline‑data** ingestion, **change‑detection** workflows, **documentation‑scraper** automation, and **web‑monitor** systems. Built‑in **faq‑scraper** extracts structured question‑answer pairs automatically. --- ### 💡 What is the FAQ & Help Center Scraper with Change Monitor? The **FAQ & Help Center Scraper with Change Monitor** is a powerful Apify actor that crawls any documentation site, knowledge base, or help center – and extracts all valuable content in a clean, structured format. Unlike generic scrapers, this actor is **built for RAG pipelines** and **AI‑ready data extraction**: - **Automatic FAQ extraction** – identifies question‑answer pairs from any page (using headers, buttons, and question patterns). - **Content change detection** – tracks modifications via content hashing, logs `New`, `Modified`, or `Unchanged` status, and records times changed. - **Rich metadata** – word count, reading time, author, published/last modified dates, breadcrumbs, product area, audience (developer vs general). - **Code & media awareness** – counts code snippets, detects images and videos. - **Related articles discovery** – finds internal recommended links. - **Pay‑per‑result pricing** – only charged for successfully scraped pages ($0.007/page). Start cost $0.02. Ideal for: - **RAG‑pipeline‑data** ingestion (e.g., loading into vector databases) - **Documentation‑scraper** for AI training or retrieval - **Change‑detection** monitoring (e.g., alert when a critical help page changes) - **Web‑monitor** for competitor documentation updates - **faq‑scraper** to build automated FAQ datasets --- ### 🚀 Key Features | Feature | Description | |---------|-------------| | **Automatic FAQ extraction** | Identifies question‑answer pairs from common patterns (“how to”, “what is”, “why”, “?”) and HTML structures. | | **Content change detection** | Hashes page content; tracks `New`, `Modified`, `Unchanged` status, times changed, and previous hash. | | **Rich content metadata** | Word count, reading time, author, published/modified dates, breadcrumbs, product area, audience (developer/general). | | **Code & media counters** | Counts code/pre blocks, detects images and videos. | | **Related articles discovery** | Extracts internal recommended or related article links. | | **Concurrent crawling** | Configurable concurrency (default 15) to speed up large documentation sites. | | **Resume & checkpoint** | Saves progress after each page; resumes from last state if interrupted. | | **Pay‑per‑result (PPE)** | Charged only for successfully scraped pages ($0.007/page). Failed pages cost nothing. | | **Residential proxy ready** | Bypasses anti‑bot measures (strongly recommended for large runs). | | **Clean JSON output** | Ready for RAG pipelines, vector databases, or analytics. | --- ### 📥 Input Parameters | Parameter | Type | Required | Default | Description | |----------------------|---------|----------|---------|-------------| | `startUrls` | array | No | `[{"url":"https://docs.stripe.com/"}]` | List of starting URLs (help center root, documentation index). | | `maxItems` | integer | No | `100` | Maximum number of pages to scrape (stop when reached). | | `concurrency` | integer | No | `15` | Number of concurrent HTTP requests (1–20). | | `proxyConfiguration` | object | No | – | Apify proxy configuration. Residential strongly recommended. | #### Example Input ```json { "startUrls": [{"url": "https://docs.stripe.com/"}, {"url": "https://support.google.com/" }], "maxItems": 500, "concurrency": 10, "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"] } } ```` *** ### 📤 Output Fields Each scraped page returns an object with the following fields: | Field | Type | Description | |------------------------|---------|-------------| | `url` | string | Page URL | | `title` | string | Page title (from ``) | | `category` | string | Second breadcrumb or “General” | | `breadcrumb` | string | Breadcrumb trail (if present) | | `content_text` | string | Main content (truncated to 1500 chars in output but full stored) | | `word_count` | integer | Number of words in main content | | `reading_time_min` | integer | Estimated reading time in minutes | | `faq_questions` | array | List of extracted questions | | `faq_answers` | array | Corresponding answers (truncated to 240 chars) | | `question_count` | integer | Number of FAQ questions found | | `code_snippets_count` | integer | Number of `<pre>` or `<code>` elements | | `has_images` | boolean | True if page contains `<img>` | | `has_video` | boolean | True if page contains `<iframe>` or `<video>` | | `author` | string | Extracted author or “Documentation Team” | | `published_date` | string | Publication date from meta tags | | `last_modified` | string | Last modified date from meta tags | | `content_hash` | string | MD5 hash of main content (for change detection) | | `change_status` | string | `New`, `Modified`, or `Unchanged` | | `change_detected_at` | string | ISO timestamp of last change detection | | `change_summary` | string | Human‑readable change description | | `times_changed` | integer | Number of times this page has changed since first scrape | | `product_area` | string | First breadcrumb or “Core API/Platform” | | `audience` | string | `Developers/Merchants` (if code snippets) else `General Users` | | `helpful_yes_no` | boolean | True if page contains “Was this helpful?” widget | | `related_articles` | array | List of related article titles | #### Example Output (FAQ‑rich page) ```json [ { "url": "https://docs.stripe.com/payments/checkout", "title": "Accept a payment with Stripe Checkout", "category": "Payments", "breadcrumb": "Docs > Payments > Checkout", "content_text": "Stripe Checkout is a prebuilt payment page...", "word_count": 1240, "reading_time_min": 6, "faq_questions": [ "What is Stripe Checkout?", "How do I customize Checkout?", "Does Checkout support recurring payments?" ], "faq_answers": [ "Stripe Checkout is a prebuilt, hosted payment page...", "You can customize Checkout by passing `payment_method_options`...", "Yes, Checkout supports subscriptions..." ], "question_count": 3, "code_snippets_count": 2, "has_images": true, "has_video": false, "author": "Stripe Documentation Team", "published_date": "2025-01-15T10:00:00Z", "last_modified": "2026-05-28T14:30:00Z", "content_hash": "a3f5c9e8d2b1f0e9c7a6b5d4e3f2a1b0", "change_status": "Modified", "change_detected_at": "2026-05-28T14:30:00Z", "change_summary": "Content updated. Previous Hash: b2e4d6f8a1c3b5e7d9f0a2c4b6e8d0f2", "times_changed": 3, "product_area": "Payments", "audience": "Developers/Merchants", "helpful_yes_no": true, "related_articles": ["One‑time payments", "Setup Intents", "Payment methods"] } ] ``` *** ### 💰 Pricing | Component | Price | |-------------------------------|-----------------------| | **Actor start (per run)** | **$0.02** | | **Per successful page result**| **$0.007** | | **Per 1,000 successful pages**| **$7.00** | - You are charged **only when a page is successfully scraped** (i.e., content extracted, no error). - Failed pages (blocked, 404, network timeout) cost nothing. - Actor start fee covers infrastructure even if zero pages are scraped. - Example: 500 successful pages = $0.02 + (500 × $0.007) = **$3.52**. - Example: 2,500 successful pages = $0.02 + (2,500 × $0.007) = **$17.52**. **Change detection is free** – the actor computes hashes and tracks changes without extra cost. You only pay for the initial scrape of each page; subsequent runs that re‑scrape the same page (if not changed) still incur a charge (since you are retrieving fresh HTML and processing it). To reduce costs, use `maxItems` and schedule runs only when needed. *** ### 🛠 How to Use on Apify 1. **Create a task** with this actor. 2. **Provide start URLs** – the root of the documentation or help center (e.g., `https://docs.stripe.com/`, `https://support.google.com/`). 3. **Set maxItems** – how many pages to scrape (default 100, maximum limited by website size). 4. **Adjust concurrency** – higher concurrency speeds up crawling but may increase blocking risk. 5. **Enable residential proxies** – strongly recommended to avoid rate limiting. 6. **Run** – the actor will crawl, extract, detect changes, charge per page, and push results to dataset. 7. **Export** – download JSON, CSV, or Excel. > **Change detection across runs**: The actor saves content hashes in key‑value store. When you rerun with the same start URLs, it will compare hashes and set `change_status` accordingly. #### Running via API ```bash curl -X POST "https://api.apify.com/v2/acts/your-username~faq-help-center-scraper/runs" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_API_TOKEN" \ -d '{ "startUrls": [{"url": "https://docs.stripe.com/"}], "maxItems": 200, "concurrency": 10 }' ``` *** ### 🎯 Use Cases | Use Case | How This Actor Helps | |----------|----------------------| | **RAG‑pipeline‑data** | Ingest clean, structured documentation into vector databases (Pinecone, Weaviate, Chroma). | | **Documentation‑scraper** | Automatically sync external help center content into your own knowledge base. | | **Change‑detection** | Monitor competitor or internal docs for updates – get alerts when critical pages change. | | **Web‑monitor** | Track frequency of content updates across large documentation sites. | | **faq‑scraper** | Build FAQ datasets for chatbot training or customer support automation. | | **Content migration** | Extract all pages from an old help center to a new platform. | *** ### ❓ Frequently Asked Questions **1. What is a “successful result” for PPE charging?**\ A page is considered successful if the actor can extract its HTML and parse at least a title and some content. Failed pages (404, Cloudflare block, timeout) are **not charged**. **2. How does change detection work?**\ The actor computes an MD5 hash of the page’s main content. If the hash differs from the previous run’s stored hash, `change_status` becomes `Modified` and `times_changed` increments. If the page was not seen before, `change_status` is `New`. If the hash matches, it is `Unchanged`. **3. Are FAQ questions and answers always accurate?**\ The actor uses heuristic rules (question marks, question‑word lists, HTML structure). Accuracy is high for well‑structured documentation, but you may get false positives. You can filter by `question_count` or manually review. **4. Do I need residential proxies?**\ For small documentation sites (<100 pages), datacenter proxies may work. For large sites (Stripe, Google, etc.) or frequent runs, residential proxies are strongly recommended to avoid `429` and CAPTCHA blocks. **5. What does the `content_text` field contain?**\ It contains the extracted main text from `<main>`, `<article>`, or the body, cleaned and truncated to 1500 characters in the JSON output. The full text is stored in the dataset item, but the displayed output in the console may truncate it. **6. How do I run change detection on a schedule?**\ Create an Apify schedule that triggers this actor every day or week. The actor will load the previous state from the key‑value store and only push pages that have changed (if you want to reduce costs, you can filter out `Unchanged` pages in post‑processing). **7. Can I limit crawling to a specific sub‑path?**\ Yes. The actor only follows links that stay within the same domain. If you want to restrict to a sub‑path (e.g., `/docs/` only), you must provide a start URL that is already within that path and rely on the fact that the actor respects the same‑domain rule. It does not have a path‑prefix filter. **8. What is the `audience` field?**\ If the page contains code snippets (`<pre>` or `<code>`), it is classified as `Developers/Merchants`. Otherwise, `General Users`. **9. How long does a run take?**\ For 1,000 pages, with concurrency set to 15, expect roughly 5–10 minutes, depending on network latency and proxy speed. **10. What happens if the actor hits the spending limit?**\ The actor checks `charge_result.event_charge_limit_reached` after each page and stops gracefully. It saves state so that when you restart (after increasing budget), it will resume without re‑scraping already processed pages. *** *** ### 🔍 SEO Keywords `faq-scraper`, `documentation-scraper`, `rag-pipeline-data`, `change-detection`, `web-monitor`, `help center scraper`, `knowledge base extractor`, `FAQ extraction`, `content change monitoring`, `RAG ingestion`, `documentation crawler`, `Apify doc scraper`, `vector database source` *** ### 🔗 Related Actors - **[Amazon Product Scraper – ASIN, Price, Rating, Reviews](https://apify.com/your-username/amazon-product-scraper)** - **[NHS Job Scraper – Salary, Band, Employer, Location](https://apify.com/your-username/nhs-job-scraper)** - **[Instagram Profile Scraper – Followers, Bio, Posts, Verified](https://apify.com/your-username/instagram-profile-scraper)** *** *** **Start monitoring documentation changes and extracting FAQ data today – $0.02 per run + $7 per 1,000 successful pages. Perfect for RAG pipelines, change detection, and AI training.** ```` # Actor input Schema ## `startUrls` (type: `array`): List of Help Center or Documentation URLs to monitor. ## `maxItems` (type: `integer`): Maximum number of documentation pages to monitor. ## `proxyConfiguration` (type: `object`): Select Residential Proxies to ensure dynamic IP rotation and prevent security blocking by documentation sites. ## Actor input object example ```json { "startUrls": [ { "url": "https://docs.stripe.com/" }, { "url": "https://docs.github.com/en" } ], "maxItems": 100, "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ] } } ```` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '<YOUR_API_TOKEN>' with your token const client = new ApifyClient({ token: '<YOUR_API_TOKEN>', }); // Prepare Actor input const input = { "startUrls": [ { "url": "https://docs.stripe.com/" }, { "url": "https://docs.github.com/en" } ], "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ] } }; // Run the Actor and wait for it to finish const run = await client.actor("scrapepilot/faq").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '<YOUR_API_TOKEN>' with your token. client = ApifyClient("<YOUR_API_TOKEN>") # Prepare the Actor input run_input = { "startUrls": [ { "url": "https://docs.stripe.com/" }, { "url": "https://docs.github.com/en" }, ], "proxyConfiguration": { "useApifyProxy": True, "apifyProxyGroups": ["RESIDENTIAL"], }, } # Run the Actor and wait for it to finish run = client.actor("scrapepilot/faq").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ { "url": "https://docs.stripe.com/" }, { "url": "https://docs.github.com/en" } ], "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ] } }' | apify call scrapepilot/faq --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=scrapepilot/faq", "--header", "Authorization: Bearer <YOUR_API_TOKEN>" ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "FAQ & Help Center Scraper with Change Monitor", "description": "Scrape and monitor documentation, FAQs, and help centers effortlessly. Extract structured Q&As, body text, breadcrumbs, and metadata. Features automatic MD5 change detection to track page updates over time. Fast, async, and powered by cost-effective Pay Per Event billing", "version": "0.0", "x-build-id": "BeaeYprnwW1LUpp9V" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/scrapepilot~faq/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-scrapepilot-faq", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/scrapepilot~faq/runs": { "post": { "operationId": "runs-sync-scrapepilot-faq", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/scrapepilot~faq/run-sync": { "post": { "operationId": "run-sync-scrapepilot-faq", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "startUrls" ], "properties": { "startUrls": { "title": "Start URLs", "type": "array", "description": "List of Help Center or Documentation URLs to monitor.", "items": { "type": "object", "required": [ "url" ], "properties": { "url": { "type": "string", "title": "URL of a web page", "format": "uri" } } } }, "maxItems": { "title": "Max Pages to Crawl", "type": "integer", "description": "Maximum number of documentation pages to monitor.", "default": 100 }, "proxyConfiguration": { "title": "Proxy Configuration", "type": "object", "description": "Select Residential Proxies to ensure dynamic IP rotation and prevent security blocking by documentation sites." } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```