# RAG Markdown Scraper (`litezero/rag-markdown-scraper`) Actor

Turn any website into a clean Vector Database pipeline. Bypasses bot protections to extract pure GitHub-Flavored Markdown. Features built-in AI Vision OCR for images, Delta Crawling to save costs, Custom Regex, and direct Pinecone upserts for RAG agents.

- **URL**: https://apify.com/litezero/rag-markdown-scraper.md
- **Developed by:** [LiteZero](https://apify.com/litezero) (community)
- **Categories:** AI, Developer tools, Automation
- **Stats:** 1 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.70 / 1,000 result extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Ultimate RAG Markdown Scraper & AI Data Agent

Transform any website into a pristine, structured Vector Database pipeline. This API provides programmatic access to a god-tier web crawler that perfectly extracts GitHub-Flavored Markdown, automatically chunks text, utilizes AI Vision to read infographics, and upserts directly to your Pinecone Vector Database with zero code.

### 🌟 Why use this Actor?

Building a production-ready RAG pipeline usually requires stitching together multiple unstable microservices. This Actor handles the entire ingestion process in a single, robust API call.

- **Scale:** Effortlessly bypass modern anti-bot protections using advanced proxy rotation, infinite scrolling, and Playwright JS rendering.
- **Data Enrichment:** Provide a Custom AI Prompt, and the Actor will natively use OpenAI to analyze the extracted Markdown and return structured JSON answers alongside your data.
- **Cost Efficiency:** Features built-in Delta Crawling, saving you 90% in LLM and compute costs by exclusively scraping pages that have changed since the last run.

***

### ⚙️ Core Features & How to Use Them

#### 1. Delta Crawling (Cost Saver)

**How it works:** When enabled, the Actor remembers the exact HTML fingerprint of every page it scraped in previous runs. On subsequent runs, if the page hasn't changed, the Actor skips it instantly.
**When to use it:** If you are running this Actor on a weekly or daily schedule to monitor documentation or blog updates, Delta Crawling will save you massive amounts of OpenAI API tokens and Apify compute units.

#### 2. Browser Engine vs. Cheerio Engine

**How it works:** You have the choice between `useBrowser: true` (Playwright) and `useBrowser: false` (Cheerio).

- **Cheerio:** Downloads the static HTML. It is blazing fast and incredibly cheap. Use this for standard blogs, WordPress sites, and documentation.
- **Playwright (Browser):** Boots up a real headless Chromium browser. Use this for heavy Single Page Applications (React, Vue, Next.js) where the data doesn't appear until the JavaScript executes.

#### 3. AI Vision OCR

**How it works:** When scraping documentation, standard scrapers completely ignore diagrams and images. By providing an OpenAI API Key and enabling Vision OCR, this Actor will take every `<img>` on the website, send it to `gpt-4o-mini`, and replace the image in the final Markdown with a highly detailed, AI-generated text description of the image's contents.

#### 4. Custom Regex Extractions

**How it works:** Want to pull every email address or API endpoint from a website? Pass an array of Regex strings into `customRegexExtractors`. The Actor will search the raw text and return an array of exact matches in the final JSON output.

***

### 📤 Output Formats

Depending on your pipeline, this Actor can deliver data in multiple formats:

#### 1. Structured JSON (Default)

The raw output is delivered directly to your Apify Dataset in clean JSON, perfect for programmatic consumption.

```json
[
  {
    "url": "https://example.com/docs",
    "title": "Documentation",
    "aiResponse": "The product costs $99/month.",
    "markdown": "# Documentation\\nWelcome to our API.",
    "regexMatches": ["support@example.com"]
  }
]
```

#### 2. Physical .ZIP Export

If you prefer managing files locally or uploading them to Obsidian/Notion, enable `exportAsZip`. The Actor will package every single scraped page into its own `.md` file, inject YAML Frontmatter at the top (containing the URL, Title, and Date), and provide a downloadable `.zip` file in the Apify Key-Value Store upon completion.

#### 3. Direct Pinecone Vector Upsert

Bypass JSON and ZIP entirely! Provide your Pinecone Host URL and API key in the inputs. The Actor will automatically chunk the Markdown, generate Vector Embeddings using OpenAI's `text-embedding-3-small`, and push them directly to your database.

***

### 🚀 Quick Start (API Integration)

You can easily trigger this Actor programmatically using the Apify SDK.

#### Python SDK

```python
from apify_client import ApifyClient

client = ApifyClient("<YOUR_API_TOKEN>")

run_input = {
    "startUrls": [{"url": "https://crawlee.dev"}],
    "useBrowser": True,
    "exportAsZip": True,
    "injectFrontmatter": True
}

run = client.actor("<YOUR_ACTOR_ID>").call(run_input=run_input)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)
```

#### Webhooks

You can configure Apify Webhooks to send a POST request to your own backend the moment this Actor finishes a run, allowing you to build completely automated ingestion pipelines without polling!

***

### 🛠️ Debugging & Memory Limits

As a user of this Actor, you have complete control over the compute resources allocated to your run. **Please note the following debugging tips regarding memory allocation on the Apify platform:**

1. **"Timeout 60000ms exceeded" Error**
   - **Cause:** You are scraping a very heavy Single Page Application (SPA) using the Playwright Browser, and the default 512MB of RAM allocated by Apify is not enough to boot the browser quickly.
   - **The Fix:** Before starting the run, go to the Apify **Run settings** at the bottom of the page and increase your **Memory allocation** from `512 MB` to **`1024 MB` or `2048 MB`**.

2. **"Memory critically overloaded (95%)" Warning**
   - **Cause:** You set the Max Concurrency too high for your allocated memory. Playwright browsers require massive amounts of RAM.
   - **The Fix:** Either decrease your `maxConcurrency` input to `1`, or increase your run's Memory allocation in the Apify settings.

3. **Missing Content in Markdown**
   - **Cause:** You used the static Cheerio engine on a React/JavaScript website.
   - **The Fix:** Toggle the **Use Browser** option to `true`.

4. **Pinecone Vectors Failing to Upsert**
   - **Cause:** Mismatched dimension size.
   - **The Fix:** Ensure your Pinecone Index is configured for exactly **1536 dimensions**, which is the required size for OpenAI's `text-embedding-3-small` model used by this Actor.

***

### 💰 Transparent Pricing

This Actor operates on a highly optimized architecture to keep your compute costs low.

- **Cheerio Engine (No JS):** Ultra-fast. Extract roughly **10,000 pages for $1**.
- **Playwright Engine (Full Browser):** Consumes more memory to render JavaScript. Extract roughly **1,500 pages for $1**.

*Note: For feature requests or highly customized enterprise scraper integrations, please contact me directly through my Apify Developer Profile or drop a ticket in the **Issues** tab!*

# Actor input Schema

## `startUrls` (type: `array`):

The URLs you want to scrape for your RAG pipeline.

## `sitemapUrls` (type: `array`):

Provide links to sitemap.xml files. The crawler will instantly extract all pages and scrape them without clicking through the site.

## `exportAsZip` (type: `boolean`):

If enabled, the scraper will generate a physical .zip file containing perfectly formatted .md files for each page.

## `exportFailedUrls` (type: `boolean`):

Generates a FAILED_URLS.json file containing all dead or failed links and the exact error reasons.

## `onlyScrapeUpdatedPages` (type: `boolean`):

Saves state between runs. Only scrapes and outputs pages that are brand new or whose content has changed since the last run. Saves huge LLM costs.

## `webhookUrl` (type: `string`):

Send an automated POST request with run details (JSON and ZIP links) to Zapier, Make.com, or your backend when the run finishes.

## `aiApiKey` (type: `string`):

Your API key for OpenAI, Anthropic, OpenRouter, or Groq.

## `aiApiBaseUrl` (type: `string`):

Use https://api.openai.com/v1 for OpenAI, or https://openrouter.ai/api/v1 for Claude/Gemini.

## `aiModel` (type: `string`):

The model to use (e.g. gpt-4o-mini, claude-3-5-sonnet).

## `enableVision` (type: `boolean`):

Warning: Can be expensive. Automatically sends all images to the Vision API to extract text and describe charts, injecting the result into the Markdown.

## `customAiPrompt` (type: `string`):

Ask the AI to extract specific data from every page (e.g. 'Summarize the pricing model').

## `pineconeApiKey` (type: `string`):

The Actor will automatically upsert vector embeddings directly to this database.

## `pineconeIndexHost` (type: `string`):

E.g. index-1234.svc.pinecone.io. Requires Pinecone API key.

## `customRegexExtractors` (type: `array`):

Array of objects like \[{"name": "emails", "regex": "\[a-z]+@\[a-z]+\\.\[a-z]+"}] to scrape specific data points from the Markdown.

## `injectFrontmatter` (type: `boolean`):

Embeds Title, URL, and Author as a YAML block at the top of the Markdown. Perfect for Obsidian/Notion users.

## `initialCookies` (type: `array`):

Paste a JSON array of session cookies to scrape private pages behind a login wall.

## `useBrowser` (type: `boolean`):

Enable this to scrape React/Next.js or sites requiring JavaScript. It is slower and costs more CU, but guarantees perfect rendering.

## `scrollToBottom` (type: `boolean`):

Automatically scrolls to the bottom of the page to trigger lazy-loaded text.

## `waitForSelector` (type: `string`):

A CSS selector (e.g. '.main-content') to wait for before scraping. Ensures lazy-loaded text is captured.

## `clickSelector` (type: `string`):

A CSS selector (e.g. '#accept-cookies') to click before scraping. Useful for closing popups/overlays.

## `chunkSize` (type: `integer`):

Automatically split the text into chunks of this size. Leave empty to output the full page as one string.

## `chunkOverlap` (type: `integer`):

How many characters should overlap between chunks to preserve context.

## `mustIncludeUrls` (type: `array`):

Only crawl URLs that contain at least one of these strings (e.g. \['/docs/', '/help/']). Leave empty to crawl everything.

## `excludeUrls` (type: `array`):

URL patterns to skip (e.g. \['https://example.com/login/\*']).

## `proxyConfiguration` (type: `object`):

Select proxies to be used by the crawler to avoid getting blocked.

## `maxPagesPerCrawl` (type: `integer`):

Set a hard limit to manage your Compute Unit (CU) costs.

## `maxCrawlDepth` (type: `integer`):

How deep into the website to crawl. 0 = only Start URLs, 1 = one click away, etc.

## `maxRequestRetries` (type: `integer`):

How many times to retry a failed/dead link. Set to 1 for fast-fail and cost savings.

## `maxConcurrency` (type: `integer`):

Lower concurrency uses less RAM (saves costs), higher is faster.

## `stripQueryParameters` (type: `boolean`):

Dramatically saves costs by preventing infinite loops on URLs like /page?sort=asc

## `enableDeduplication` (type: `boolean`):

If enabled, pages with the exact same text content will be skipped (saves RAG tokens).

## `removeSelectors` (type: `string`):

Comma-separated CSS selectors to strip out of the page before converting to Markdown.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://crawlee.dev"
    }
  ],
  "sitemapUrls": [],
  "exportAsZip": false,
  "exportFailedUrls": true,
  "onlyScrapeUpdatedPages": false,
  "webhookUrl": "",
  "aiApiKey": "",
  "aiApiBaseUrl": "https://api.openai.com/v1",
  "aiModel": "gpt-4o-mini",
  "enableVision": false,
  "customAiPrompt": "",
  "pineconeApiKey": "",
  "pineconeIndexHost": "",
  "customRegexExtractors": [],
  "injectFrontmatter": false,
  "initialCookies": [],
  "useBrowser": false,
  "scrollToBottom": false,
  "waitForSelector": "",
  "clickSelector": "",
  "chunkSize": 1000,
  "chunkOverlap": 200,
  "mustIncludeUrls": [],
  "excludeUrls": [],
  "proxyConfiguration": {
    "useApifyProxy": true
  },
  "maxPagesPerCrawl": 100,
  "maxCrawlDepth": 20,
  "maxRequestRetries": 1,
  "maxConcurrency": 10,
  "stripQueryParameters": true,
  "enableDeduplication": true,
  "removeSelectors": "nav, footer, aside, script, style, noscript, iframe, svg, .cookie-banner, .ads"
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://crawlee.dev"
        }
    ],
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("litezero/rag-markdown-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://crawlee.dev" }],
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("litezero/rag-markdown-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://crawlee.dev"
    }
  ],
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call litezero/rag-markdown-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,litezero/rag-markdown-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/qUyc5LegosBmkcoow/builds/H8vaAKddD54r01NKW/openapi.json