Ultimate RAG Markdown Scraper & AI Data Agent

Transform any website into a pristine, structured Vector Database pipeline. This API provides programmatic access to a god-tier web crawler that perfectly extracts GitHub-Flavored Markdown, automatically chunks text, utilizes AI Vision to read infographics, and upserts directly to your Pinecone Vector Database with zero code.

🌟 Why use this Actor?

Building a production-ready RAG pipeline usually requires stitching together multiple unstable microservices. This Actor handles the entire ingestion process in a single, robust API call.

• Scale: Effortlessly bypass modern anti-bot protections using advanced proxy rotation, infinite scrolling, and Playwright JS rendering.
• Data Enrichment: Provide a Custom AI Prompt, and the Actor will natively use OpenAI to analyze the extracted Markdown and return structured JSON answers alongside your data.
• Cost Efficiency: Features built-in Delta Crawling, saving you 90% in LLM and compute costs by exclusively scraping pages that have changed since the last run.

---

⚙️ Core Features & How to Use Them

1. Delta Crawling (Cost Saver)

How it works: When enabled, the Actor remembers the exact HTML fingerprint of every page it scraped in previous runs. On subsequent runs, if the page hasn't changed, the Actor skips it instantly. When to use it: If you are running this Actor on a weekly or daily schedule to monitor documentation or blog updates, Delta Crawling will save you massive amounts of OpenAI API tokens and Apify compute units.

2. Browser Engine vs. Cheerio Engine

How it works: You have the choice between useBrowser: true (Playwright) and useBrowser: false (Cheerio).

• Cheerio: Downloads the static HTML. It is blazing fast and incredibly cheap. Use this for standard blogs, WordPress sites, and documentation.
• Playwright (Browser): Boots up a real headless Chromium browser. Use this for heavy Single Page Applications (React, Vue, Next.js) where the data doesn't appear until the JavaScript executes.

3. AI Vision OCR

How it works: When scraping documentation, standard scrapers completely ignore diagrams and images. By providing an OpenAI API Key and enabling Vision OCR, this Actor will take every <img> on the website, send it to gpt-4o-mini, and replace the image in the final Markdown with a highly detailed, AI-generated text description of the image's contents.

4. Custom Regex Extractions

How it works: Want to pull every email address or API endpoint from a website? Pass an array of Regex strings into customRegexExtractors. The Actor will search the raw text and return an array of exact matches in the final JSON output.

---

📤 Output Formats

Depending on your pipeline, this Actor can deliver data in multiple formats:

1. Structured JSON (Default)

The raw output is delivered directly to your Apify Dataset in clean JSON, perfect for programmatic consumption.

[
  {
    "url": "https://example.com/docs",
    "title": "Documentation",
    "aiResponse": "The product costs $99/month.",
    "markdown": "# Documentation\\nWelcome to our API.",
    "regexMatches": ["support@example.com"]
  }
]

2. Physical .ZIP Export

If you prefer managing files locally or uploading them to Obsidian/Notion, enable exportAsZip. The Actor will package every single scraped page into its own .md file, inject YAML Frontmatter at the top (containing the URL, Title, and Date), and provide a downloadable .zip file in the Apify Key-Value Store upon completion.

3. Direct Pinecone Vector Upsert

Bypass JSON and ZIP entirely! Provide your Pinecone Host URL and API key in the inputs. The Actor will automatically chunk the Markdown, generate Vector Embeddings using OpenAI's text-embedding-3-small, and push them directly to your database.

---

🚀 Quick Start (API Integration)

You can easily trigger this Actor programmatically using the Apify SDK.

Python SDK

from apify_client import ApifyClient

client = ApifyClient("<YOUR_API_TOKEN>")

run_input = {
    "startUrls": [{"url": "https://crawlee.dev"}],
    "useBrowser": True,
    "exportAsZip": True,
    "injectFrontmatter": True
}

run = client.actor("<YOUR_ACTOR_ID>").call(run_input=run_input)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

Webhooks

You can configure Apify Webhooks to send a POST request to your own backend the moment this Actor finishes a run, allowing you to build completely automated ingestion pipelines without polling!

---

🛠️ Debugging & Memory Limits

As a user of this Actor, you have complete control over the compute resources allocated to your run. Please note the following debugging tips regarding memory allocation on the Apify platform:

1. "Timeout 60000ms exceeded" Error

   • Cause: You are scraping a very heavy Single Page Application (SPA) using the Playwright Browser, and the default 512MB of RAM allocated by Apify is not enough to boot the browser quickly.
   • The Fix: Before starting the run, go to the Apify Run settings at the bottom of the page and increase your Memory allocation from 512 MB to 1024 MB or 2048 MB.

2. "Memory critically overloaded (95%)" Warning

   • Cause: You set the Max Concurrency too high for your allocated memory. Playwright browsers require massive amounts of RAM.
   • The Fix: Either decrease your maxConcurrency input to 1, or increase your run's Memory allocation in the Apify settings.

3. Missing Content in Markdown

   • Cause: You used the static Cheerio engine on a React/JavaScript website.
   • The Fix: Toggle the Use Browser option to true.

4. Pinecone Vectors Failing to Upsert

   • Cause: Mismatched dimension size.
   • The Fix: Ensure your Pinecone Index is configured for exactly 1536 dimensions, which is the required size for OpenAI's text-embedding-3-small model used by this Actor.

---

💰 Transparent Pricing

This Actor operates on a highly optimized architecture to keep your compute costs low.

• Cheerio Engine (No JS): Ultra-fast. Extract roughly 10,000 pages for $1.
• Playwright Engine (Full Browser): Consumes more memory to render JavaScript. Extract roughly 1,500 pages for $1.

Note: For feature requests or highly customized enterprise scraper integrations, please contact me directly through my Apify Developer Profile or drop a ticket in the Issues tab!