# RAG Markdown Scraper (`tillzero/rag-markdown-scraper`) Actor

Turn any website into a clean Vector Database pipeline. Bypasses bot protections to extract pure GitHub-Flavored Markdown. Features built-in AI Vision OCR for images, Delta Crawling to save costs, Custom Regex, and direct Pinecone upserts for RAG agents.

- **URL**: https://apify.com/tillzero/rag-markdown-scraper.md
- **Developed by:** [K S](https://apify.com/tillzero) (community)
- **Categories:** AI, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 page scrapeds

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md):

```bash
npm install apify-client
```

In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md):

```bash
pip install apify-client
```

In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md):

````bash
# MacOS / Linux
curl -fsSL https://apify.com/install-cli.sh | bash
# Windows
irm https://apify.com/install-cli.ps1 | iex
```bash

In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md).

If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).


# README

## Ultimate RAG Markdown Scraper & AI Data Agent

Transform any website into a pristine, structured Vector Database pipeline. This API provides programmatic access to a god-tier web crawler that perfectly extracts GitHub-Flavored Markdown, automatically chunks text, utilizes AI Vision to read infographics, and upserts directly to your Pinecone Vector Database with zero code.

### 🌟 Why use this Actor?
Building a production-ready RAG pipeline usually requires stitching together multiple unstable microservices. This Actor handles the entire ingestion process in a single, robust API call. 
- **Scale:** Effortlessly bypass modern anti-bot protections using advanced proxy rotation, infinite scrolling, and Playwright JS rendering.
- **Data Enrichment:** Provide a Custom AI Prompt, and the Actor will natively use OpenAI to analyze the extracted Markdown and return structured JSON answers alongside your data.
- **Cost Efficiency:** Features built-in Delta Crawling, saving you 90% in LLM and compute costs by exclusively scraping pages that have changed since the last run.

---

### ⚙️ Core Features & How to Use Them

#### 1. Delta Crawling (Cost Saver)
**How it works:** When enabled, the Actor remembers the exact HTML fingerprint of every page it scraped in previous runs. On subsequent runs, if the page hasn't changed, the Actor skips it instantly.
**When to use it:** If you are running this Actor on a weekly or daily schedule to monitor documentation or blog updates, Delta Crawling will save you massive amounts of OpenAI API tokens and Apify compute units.

#### 2. Browser Engine vs. Cheerio Engine
**How it works:** You have the choice between `useBrowser: true` (Playwright) and `useBrowser: false` (Cheerio). 
- **Cheerio:** Downloads the static HTML. It is blazing fast and incredibly cheap. Use this for standard blogs, WordPress sites, and documentation.
- **Playwright (Browser):** Boots up a real headless Chromium browser. Use this for heavy Single Page Applications (React, Vue, Next.js) where the data doesn't appear until the JavaScript executes.

#### 3. AI Vision OCR
**How it works:** When scraping documentation, standard scrapers completely ignore diagrams and images. By providing an OpenAI API Key and enabling Vision OCR, this Actor will take every `<img>` on the website, send it to `gpt-4o-mini`, and replace the image in the final Markdown with a highly detailed, AI-generated text description of the image's contents.

#### 4. Custom Regex Extractions
**How it works:** Want to pull every email address or API endpoint from a website? Pass an array of Regex strings into `customRegexExtractors`. The Actor will search the raw text and return an array of exact matches in the final JSON output.

---

### 📤 Output Formats

Depending on your pipeline, this Actor can deliver data in multiple formats:

#### 1. Structured JSON (Default)
The raw output is delivered directly to your Apify Dataset in clean JSON, perfect for programmatic consumption.
```json
[
  {
    "url": "https://example.com/docs",
    "title": "Documentation",
    "aiResponse": "The product costs $99/month.",
    "markdown": "# Documentation\\nWelcome to our API.",
    "regexMatches": ["support@example.com"]
  }
]
````

#### 2. Physical .ZIP Export

If you prefer managing files locally or uploading them to Obsidian/Notion, enable `exportAsZip`. The Actor will package every single scraped page into its own `.md` file, inject YAML Frontmatter at the top (containing the URL, Title, and Date), and provide a downloadable `.zip` file in the Apify Key-Value Store upon completion.

#### 3. Direct Pinecone Vector Upsert

Bypass JSON and ZIP entirely! Provide your Pinecone Host URL and API key in the inputs. The Actor will automatically chunk the Markdown, generate Vector Embeddings using OpenAI's `text-embedding-3-small`, and push them directly to your database.

***

### 🚀 Quick Start (API Integration)

You can easily trigger this Actor programmatically using the Apify SDK.

#### Python SDK

```python
from apify_client import ApifyClient

client = ApifyClient("<YOUR_API_TOKEN>")

run_input = {
    "startUrls": [{"url": "https://crawlee.dev"}],
    "useBrowser": True,
    "exportAsZip": True,
    "injectFrontmatter": True
}

run = client.actor("<YOUR_ACTOR_ID>").call(run_input=run_input)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)
```

#### Webhooks

You can configure Apify Webhooks to send a POST request to your own backend the moment this Actor finishes a run, allowing you to build completely automated ingestion pipelines without polling!

***

### 🛠️ Debugging & Memory Limits

As a user of this Actor, you have complete control over the compute resources allocated to your run. **Please note the following debugging tips regarding memory allocation on the Apify platform:**

1. **"Timeout 60000ms exceeded" Error**
   - **Cause:** You are scraping a very heavy Single Page Application (SPA) using the Playwright Browser, and the default 512MB of RAM allocated by Apify is not enough to boot the browser quickly.
   - **The Fix:** Before starting the run, go to the Apify **Run settings** at the bottom of the page and increase your **Memory allocation** from `512 MB` to **`1024 MB` or `2048 MB`**.

2. **"Memory critically overloaded (95%)" Warning**
   - **Cause:** You set the Max Concurrency too high for your allocated memory. Playwright browsers require massive amounts of RAM.
   - **The Fix:** Either decrease your `maxConcurrency` input to `1`, or increase your run's Memory allocation in the Apify settings.

3. **Missing Content in Markdown**
   - **Cause:** You used the static Cheerio engine on a React/JavaScript website.
   - **The Fix:** Toggle the **Use Browser** option to `true`.

4. **Pinecone Vectors Failing to Upsert**
   - **Cause:** Mismatched dimension size.
   - **The Fix:** Ensure your Pinecone Index is configured for exactly **1536 dimensions**, which is the required size for OpenAI's `text-embedding-3-small` model used by this Actor.

***

### 💰 Transparent Pricing

This Actor operates on a highly optimized architecture to keep your compute costs low.

- **Cheerio Engine (No JS):** Ultra-fast. Extract roughly **10,000 pages for $1**.
- **Playwright Engine (Full Browser):** Consumes more memory to render JavaScript. Extract roughly **1,500 pages for $1**.

*Note: For feature requests or highly customized enterprise scraper integrations, please contact me directly through my Apify Developer Profile or drop a ticket in the **Issues** tab!*

# Actor input Schema

## `startUrls` (type: `array`):

The URLs you want to scrape for your RAG pipeline.

## `sitemapUrls` (type: `array`):

Provide links to sitemap.xml files. The crawler will instantly extract all pages and scrape them without clicking through the site.

## `exportAsZip` (type: `boolean`):

If enabled, the scraper will generate a physical .zip file containing perfectly formatted .md files for each page.

## `exportFailedUrls` (type: `boolean`):

Generates a FAILED\_URLS.json file containing all dead or failed links and the exact error reasons.

## `onlyScrapeUpdatedPages` (type: `boolean`):

Saves state between runs. Only scrapes and outputs pages that are brand new or whose content has changed since the last run. Saves huge LLM costs.

## `webhookUrl` (type: `string`):

Send an automated POST request with run details (JSON and ZIP links) to Zapier, Make.com, or your backend when the run finishes.

## `aiApiKey` (type: `string`):

Your API key for OpenAI, Anthropic, OpenRouter, or Groq.

## `aiApiBaseUrl` (type: `string`):

Use https://api.openai.com/v1 for OpenAI, or https://openrouter.ai/api/v1 for Claude/Gemini.

## `aiModel` (type: `string`):

The model to use (e.g. gpt-4o-mini, claude-3-5-sonnet).

## `enableVision` (type: `boolean`):

Warning: Can be expensive. Automatically sends all images to the Vision API to extract text and describe charts, injecting the result into the Markdown.

## `customAiPrompt` (type: `string`):

Ask the AI to extract specific data from every page (e.g. 'Summarize the pricing model').

## `pineconeApiKey` (type: `string`):

The Actor will automatically upsert vector embeddings directly to this database.

## `pineconeIndexHost` (type: `string`):

E.g. index-1234.svc.pinecone.io. Requires Pinecone API key.

## `customRegexExtractors` (type: `array`):

Array of objects like \[{"name": "emails", "regex": "\[a-z]+@\[a-z]+\\.\[a-z]+"}] to scrape specific data points from the Markdown.

## `injectFrontmatter` (type: `boolean`):

Embeds Title, URL, and Author as a YAML block at the top of the Markdown. Perfect for Obsidian/Notion users.

## `initialCookies` (type: `array`):

Paste a JSON array of session cookies to scrape private pages behind a login wall.

## `useBrowser` (type: `boolean`):

Enable this to scrape React/Next.js or sites requiring JavaScript. It is slower and costs more CU, but guarantees perfect rendering.

## `scrollToBottom` (type: `boolean`):

Automatically scrolls to the bottom of the page to trigger lazy-loaded text.

## `waitForSelector` (type: `string`):

A CSS selector (e.g. '.main-content') to wait for before scraping. Ensures lazy-loaded text is captured.

## `clickSelector` (type: `string`):

A CSS selector (e.g. '#accept-cookies') to click before scraping. Useful for closing popups/overlays.

## `chunkSize` (type: `integer`):

Automatically split the text into chunks of this size. Leave empty to output the full page as one string.

## `chunkOverlap` (type: `integer`):

How many characters should overlap between chunks to preserve context.

## `mustIncludeUrls` (type: `array`):

Only crawl URLs that contain at least one of these strings (e.g. \['/docs/', '/help/']). Leave empty to crawl everything.

## `excludeUrls` (type: `array`):

URL patterns to skip (e.g. \['https://example.com/login/\*']).

## `proxyConfiguration` (type: `object`):

Select proxies to be used by the crawler to avoid getting blocked.

## `maxPagesPerCrawl` (type: `integer`):

Set a hard limit to manage your Compute Unit (CU) costs.

## `maxCrawlDepth` (type: `integer`):

How deep into the website to crawl. 0 = only Start URLs, 1 = one click away, etc.

## `maxRequestRetries` (type: `integer`):

How many times to retry a failed/dead link. Set to 1 for fast-fail and cost savings.

## `maxConcurrency` (type: `integer`):

Lower concurrency uses less RAM (saves costs), higher is faster.

## `stripQueryParameters` (type: `boolean`):

Dramatically saves costs by preventing infinite loops on URLs like /page?sort=asc

## `enableDeduplication` (type: `boolean`):

If enabled, pages with the exact same text content will be skipped (saves RAG tokens).

## `removeSelectors` (type: `string`):

Comma-separated CSS selectors to strip out of the page before converting to Markdown.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://crawlee.dev"
    }
  ],
  "sitemapUrls": [],
  "exportAsZip": false,
  "exportFailedUrls": true,
  "onlyScrapeUpdatedPages": false,
  "webhookUrl": "",
  "aiApiKey": "",
  "aiApiBaseUrl": "https://api.openai.com/v1",
  "aiModel": "gpt-4o-mini",
  "enableVision": false,
  "customAiPrompt": "",
  "pineconeApiKey": "",
  "pineconeIndexHost": "",
  "customRegexExtractors": [],
  "injectFrontmatter": false,
  "initialCookies": [],
  "useBrowser": false,
  "scrollToBottom": false,
  "waitForSelector": "",
  "clickSelector": "",
  "chunkSize": 1000,
  "chunkOverlap": 200,
  "mustIncludeUrls": [],
  "excludeUrls": [],
  "proxyConfiguration": {
    "useApifyProxy": true
  },
  "maxPagesPerCrawl": 100,
  "maxCrawlDepth": 20,
  "maxRequestRetries": 1,
  "maxConcurrency": 10,
  "stripQueryParameters": true,
  "enableDeduplication": true,
  "removeSelectors": "nav, footer, aside, script, style, noscript, iframe, svg, .cookie-banner, .ads"
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://crawlee.dev"
        }
    ],
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("tillzero/rag-markdown-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://crawlee.dev" }],
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("tillzero/rag-markdown-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://crawlee.dev"
    }
  ],
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call tillzero/rag-markdown-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=tillzero/rag-markdown-scraper",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

```json
{
    "openapi": "3.0.1",
    "info": {
        "title": "RAG Markdown Scraper",
        "description": "Turn any website into a clean Vector Database pipeline. Bypasses bot protections to extract pure GitHub-Flavored Markdown. Features built-in AI Vision OCR for images, Delta Crawling to save costs, Custom Regex, and direct Pinecone upserts for RAG agents.",
        "version": "0.0",
        "x-build-id": "w29IeIO2jVLufb3Eh"
    },
    "servers": [
        {
            "url": "https://api.apify.com/v2"
        }
    ],
    "paths": {
        "/acts/tillzero~rag-markdown-scraper/run-sync-get-dataset-items": {
            "post": {
                "operationId": "run-sync-get-dataset-items-tillzero-rag-markdown-scraper",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK"
                    }
                }
            }
        },
        "/acts/tillzero~rag-markdown-scraper/runs": {
            "post": {
                "operationId": "runs-sync-tillzero-rag-markdown-scraper",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor and returns information about the initiated run in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK",
                        "content": {
                            "application/json": {
                                "schema": {
                                    "$ref": "#/components/schemas/runsResponseSchema"
                                }
                            }
                        }
                    }
                }
            }
        },
        "/acts/tillzero~rag-markdown-scraper/run-sync": {
            "post": {
                "operationId": "run-sync-tillzero-rag-markdown-scraper",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK"
                    }
                }
            }
        }
    },
    "components": {
        "schemas": {
            "inputSchema": {
                "type": "object",
                "required": [
                    "startUrls"
                ],
                "properties": {
                    "startUrls": {
                        "title": "Start URLs",
                        "type": "array",
                        "description": "The URLs you want to scrape for your RAG pipeline.",
                        "items": {
                            "type": "object",
                            "required": [
                                "url"
                            ],
                            "properties": {
                                "url": {
                                    "type": "string",
                                    "title": "URL of a web page",
                                    "format": "uri"
                                }
                            }
                        }
                    },
                    "sitemapUrls": {
                        "title": "Sitemap URLs (Ultimate Efficiency)",
                        "type": "array",
                        "description": "Provide links to sitemap.xml files. The crawler will instantly extract all pages and scrape them without clicking through the site.",
                        "default": [],
                        "items": {
                            "type": "object",
                            "required": [
                                "url"
                            ],
                            "properties": {
                                "url": {
                                    "type": "string",
                                    "title": "URL of a web page",
                                    "format": "uri"
                                }
                            }
                        }
                    },
                    "exportAsZip": {
                        "title": "Export as ZIP (Markdown Files)",
                        "type": "boolean",
                        "description": "If enabled, the scraper will generate a physical .zip file containing perfectly formatted .md files for each page.",
                        "default": false
                    },
                    "exportFailedUrls": {
                        "title": "Export Audit Log (Failed URLs)",
                        "type": "boolean",
                        "description": "Generates a FAILED_URLS.json file containing all dead or failed links and the exact error reasons.",
                        "default": true
                    },
                    "onlyScrapeUpdatedPages": {
                        "title": "Delta Crawl (Only Updated Pages)",
                        "type": "boolean",
                        "description": "Saves state between runs. Only scrapes and outputs pages that are brand new or whose content has changed since the last run. Saves huge LLM costs.",
                        "default": false
                    },
                    "webhookUrl": {
                        "title": "Webhook URL",
                        "type": "string",
                        "description": "Send an automated POST request with run details (JSON and ZIP links) to Zapier, Make.com, or your backend when the run finishes.",
                        "default": ""
                    },
                    "aiApiKey": {
                        "title": "AI API Key",
                        "type": "string",
                        "description": "Your API key for OpenAI, Anthropic, OpenRouter, or Groq.",
                        "default": ""
                    },
                    "aiApiBaseUrl": {
                        "title": "AI API Base URL",
                        "type": "string",
                        "description": "Use https://api.openai.com/v1 for OpenAI, or https://openrouter.ai/api/v1 for Claude/Gemini.",
                        "default": "https://api.openai.com/v1"
                    },
                    "aiModel": {
                        "title": "AI Model",
                        "type": "string",
                        "description": "The model to use (e.g. gpt-4o-mini, claude-3-5-sonnet).",
                        "default": "gpt-4o-mini"
                    },
                    "enableVision": {
                        "title": "Enable AI Vision (OCR)",
                        "type": "boolean",
                        "description": "Warning: Can be expensive. Automatically sends all images to the Vision API to extract text and describe charts, injecting the result into the Markdown.",
                        "default": false
                    },
                    "customAiPrompt": {
                        "title": "Custom AI Prompt",
                        "type": "string",
                        "description": "Ask the AI to extract specific data from every page (e.g. 'Summarize the pricing model').",
                        "default": ""
                    },
                    "pineconeApiKey": {
                        "title": "Pinecone API Key (Zero-Code RAG)",
                        "type": "string",
                        "description": "The Actor will automatically upsert vector embeddings directly to this database.",
                        "default": ""
                    },
                    "pineconeIndexHost": {
                        "title": "Pinecone Index Host (Zero-Code RAG)",
                        "type": "string",
                        "description": "E.g. index-1234.svc.pinecone.io. Requires Pinecone API key.",
                        "default": ""
                    },
                    "customRegexExtractors": {
                        "title": "Custom Regex Extractors",
                        "type": "array",
                        "description": "Array of objects like [{\"name\": \"emails\", \"regex\": \"[a-z]+@[a-z]+\\\\.[a-z]+\"}] to scrape specific data points from the Markdown.",
                        "default": []
                    },
                    "injectFrontmatter": {
                        "title": "Inject YAML Frontmatter",
                        "type": "boolean",
                        "description": "Embeds Title, URL, and Author as a YAML block at the top of the Markdown. Perfect for Obsidian/Notion users.",
                        "default": false
                    },
                    "initialCookies": {
                        "title": "Initial Cookies (Login Support)",
                        "type": "array",
                        "description": "Paste a JSON array of session cookies to scrape private pages behind a login wall.",
                        "default": []
                    },
                    "useBrowser": {
                        "title": "Use Browser (JS Rendering)",
                        "type": "boolean",
                        "description": "Enable this to scrape React/Next.js or sites requiring JavaScript. It is slower and costs more CU, but guarantees perfect rendering.",
                        "default": false
                    },
                    "scrollToBottom": {
                        "title": "Auto-Scroll to Bottom (Browser Only)",
                        "type": "boolean",
                        "description": "Automatically scrolls to the bottom of the page to trigger lazy-loaded text.",
                        "default": false
                    },
                    "waitForSelector": {
                        "title": "Wait For Selector (Browser Only)",
                        "type": "string",
                        "description": "A CSS selector (e.g. '.main-content') to wait for before scraping. Ensures lazy-loaded text is captured.",
                        "default": ""
                    },
                    "clickSelector": {
                        "title": "Click Selector (Browser Only)",
                        "type": "string",
                        "description": "A CSS selector (e.g. '#accept-cookies') to click before scraping. Useful for closing popups/overlays.",
                        "default": ""
                    },
                    "chunkSize": {
                        "title": "Chunk Size",
                        "type": "integer",
                        "description": "Automatically split the text into chunks of this size. Leave empty to output the full page as one string.",
                        "default": 1000
                    },
                    "chunkOverlap": {
                        "title": "Chunk Overlap",
                        "type": "integer",
                        "description": "How many characters should overlap between chunks to preserve context.",
                        "default": 200
                    },
                    "mustIncludeUrls": {
                        "title": "Whitelist URLs (Must include)",
                        "type": "array",
                        "description": "Only crawl URLs that contain at least one of these strings (e.g. ['/docs/', '/help/']). Leave empty to crawl everything.",
                        "default": [],
                        "items": {
                            "type": "object",
                            "required": [
                                "url"
                            ],
                            "properties": {
                                "url": {
                                    "type": "string",
                                    "title": "URL of a web page",
                                    "format": "uri"
                                }
                            }
                        }
                    },
                    "excludeUrls": {
                        "title": "Exclude URLs",
                        "type": "array",
                        "description": "URL patterns to skip (e.g. ['https://example.com/login/*']).",
                        "default": [],
                        "items": {
                            "type": "object",
                            "required": [
                                "url"
                            ],
                            "properties": {
                                "url": {
                                    "type": "string",
                                    "title": "URL of a web page",
                                    "format": "uri"
                                }
                            }
                        }
                    },
                    "proxyConfiguration": {
                        "title": "Proxy configuration",
                        "type": "object",
                        "description": "Select proxies to be used by the crawler to avoid getting blocked."
                    },
                    "maxPagesPerCrawl": {
                        "title": "Max pages to crawl",
                        "type": "integer",
                        "description": "Set a hard limit to manage your Compute Unit (CU) costs.",
                        "default": 100
                    },
                    "maxCrawlDepth": {
                        "title": "Max Crawl Depth",
                        "type": "integer",
                        "description": "How deep into the website to crawl. 0 = only Start URLs, 1 = one click away, etc.",
                        "default": 20
                    },
                    "maxRequestRetries": {
                        "title": "Max Request Retries",
                        "type": "integer",
                        "description": "How many times to retry a failed/dead link. Set to 1 for fast-fail and cost savings.",
                        "default": 1
                    },
                    "maxConcurrency": {
                        "title": "Max Concurrency",
                        "type": "integer",
                        "description": "Lower concurrency uses less RAM (saves costs), higher is faster.",
                        "default": 10
                    },
                    "stripQueryParameters": {
                        "title": "Strip Query Parameters",
                        "type": "boolean",
                        "description": "Dramatically saves costs by preventing infinite loops on URLs like /page?sort=asc",
                        "default": true
                    },
                    "enableDeduplication": {
                        "title": "Enable Content Deduplication",
                        "type": "boolean",
                        "description": "If enabled, pages with the exact same text content will be skipped (saves RAG tokens).",
                        "default": true
                    },
                    "removeSelectors": {
                        "title": "Remove Elements (CSS Selectors)",
                        "type": "string",
                        "description": "Comma-separated CSS selectors to strip out of the page before converting to Markdown.",
                        "default": "nav, footer, aside, script, style, noscript, iframe, svg, .cookie-banner, .ads"
                    }
                }
            },
            "runsResponseSchema": {
                "type": "object",
                "properties": {
                    "data": {
                        "type": "object",
                        "properties": {
                            "id": {
                                "type": "string"
                            },
                            "actId": {
                                "type": "string"
                            },
                            "userId": {
                                "type": "string"
                            },
                            "startedAt": {
                                "type": "string",
                                "format": "date-time",
                                "example": "2025-01-08T00:00:00.000Z"
                            },
                            "finishedAt": {
                                "type": "string",
                                "format": "date-time",
                                "example": "2025-01-08T00:00:00.000Z"
                            },
                            "status": {
                                "type": "string",
                                "example": "READY"
                            },
                            "meta": {
                                "type": "object",
                                "properties": {
                                    "origin": {
                                        "type": "string",
                                        "example": "API"
                                    },
                                    "userAgent": {
                                        "type": "string"
                                    }
                                }
                            },
                            "stats": {
                                "type": "object",
                                "properties": {
                                    "inputBodyLen": {
                                        "type": "integer",
                                        "example": 2000
                                    },
                                    "rebootCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "restartCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "resurrectCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "computeUnits": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            },
                            "options": {
                                "type": "object",
                                "properties": {
                                    "build": {
                                        "type": "string",
                                        "example": "latest"
                                    },
                                    "timeoutSecs": {
                                        "type": "integer",
                                        "example": 300
                                    },
                                    "memoryMbytes": {
                                        "type": "integer",
                                        "example": 1024
                                    },
                                    "diskMbytes": {
                                        "type": "integer",
                                        "example": 2048
                                    }
                                }
                            },
                            "buildId": {
                                "type": "string"
                            },
                            "defaultKeyValueStoreId": {
                                "type": "string"
                            },
                            "defaultDatasetId": {
                                "type": "string"
                            },
                            "defaultRequestQueueId": {
                                "type": "string"
                            },
                            "buildNumber": {
                                "type": "string",
                                "example": "1.0.0"
                            },
                            "containerUrl": {
                                "type": "string"
                            },
                            "usage": {
                                "type": "object",
                                "properties": {
                                    "ACTOR_COMPUTE_UNITS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_WRITES": {
                                        "type": "integer",
                                        "example": 1
                                    },
                                    "KEY_VALUE_STORE_LISTS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_INTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_EXTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_SERPS": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            },
                            "usageTotalUsd": {
                                "type": "number",
                                "example": 0.00005
                            },
                            "usageUsd": {
                                "type": "object",
                                "properties": {
                                    "ACTOR_COMPUTE_UNITS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_WRITES": {
                                        "type": "number",
                                        "example": 0.00005
                                    },
                                    "KEY_VALUE_STORE_LISTS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_INTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_EXTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_SERPS": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}
```