# Web to Markdown Converter for AI & RAG (`rbbscrap/web-to-markdown-scraper`) Actor Convert web pages, documentation, and blog posts into clean, LLM-ready Markdown. Strips scripts, ads, and cookie banners for OpenAI, Clau.... Compatible Make.com, Zapier, n8n, Sheets & Webhooks. 256MB RAM & sub-second speed. - **URL**: https://apify.com/rbbscrap/web-to-markdown-scraper.md - **Developed by:** [Hector](https://apify.com/rbbscrap) (community) - **Categories:** Developer tools - **Features:** Limited permissions - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 1 bookmarks - **User rating**: No ratings yet ## Pricing $0.50 / 1,000 scraped markdown pages This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README

Web to Markdown Converter for AI & RAG Official Logo

## Web to Markdown Converter for AI & RAG **Web to Markdown Converter for AI & RAG Scraper** extracts structured Web to Markdown records, metadata, metrics, and direct links in real time. Built on a zero-browser HTTP/2 engine running in **256 MB RAM**, it delivers normalized datasets with **zero `null` fields** ready for **CSV, Excel, JSON**, or API pipelines. > **Looking for related scrapers?** Check out [Reddit Scraper](https://apify.com/rbbscrap/reddit-scraper), [LinkedIn Jobs Scraper](https://apify.com/rbbscrap/linkedin-jobs-scraper), [Google Autocomplete & SEO Scraper](https://apify.com/rbbscrap/google-autocomplete-keyword-scraper), [Hacker News Scraper](https://apify.com/rbbscrap/hacker-news-scraper), or browse our [full catalog of 118+ dedicated scrapers](https://apify.com/rbbscrap). *** ### What does Web to Markdown Converter for AI & RAG Scraper do? Convert web pages, documentation, and blog posts into clean, LLM-ready Markdown. Strips scripts, ads, and cookie banners for OpenAI, Clau.... Compatible Make.com, Zapier, n8n, Sheets & Webhooks. 256MB RAM & sub-second speed. Whether you need to monitor Web to Markdown at scale, enrich B2B/B2C databases, or feed structured records into analytics and AI workflows, **Web to Markdown Converter for AI & RAG Scraper** handles pagination, field normalization, and schema validation automatically: - **Bulk Data Extraction:** Collect Web to Markdown listings, items, and detailed attributes in batches using keywords, filters, or direct URLs. - **Complete Public Metadata:** Capture identifiers, pricing/engagement metrics, author/vendor details, timestamps, and canonical URLs. - **Multi-Format Export:** Download results in **JSON, CSV, Excel (XLSX), XML, or HTML Table** directly from the Apify Console. - **Ready for Automation & AI:** Connect seamlessly to [Apify Integrations](https://apify.com/integrations) (Make, Zapier, n8n, Google Sheets), [REST API endpoints](https://apify.com/rbbscrap/web-to-markdown-scraper/api), or the [Apify MCP Server](https://apify.com/rbbscrap/web-to-markdown-scraper/api/mcp). #### Beyond the Official Web to Markdown API - **No API Keys or Approval Delays:** Start extracting public Web to Markdown data immediately without registering developer apps or waiting for quota approvals. - **No Restrictive Rate Quotas:** Bypass strict per-minute API limits and expensive enterprise tiers while capturing the complete public data points displayed on Web to Markdown. - **256 MB Zero-Browser Architecture:** Uses native HTTP/2 and TLS session emulation instead of heavy Chromium instances, executing in seconds at a fraction of standard compute costs. *** ### What data can you scrape from Web to Markdown? **Web to Markdown Converter for AI & RAG Scraper** extracts the following data points from Web to Markdown:
🔗 URLs (urls) 📝 Only Main Content (onlyMainContent) 🖼️ Include Images (includeImages)
🔗 Include Links (includeLinks) 🔗 URL (url) 📝 Title (title)
📝 Description (description) 🖼️ Featured Image (featuredImage) 📝 Favicon (favicon)
📝 Word Count (wordCount) 📝 Character Count (characterCount) 📝 Markdown (markdown)
| Data Category | Extracted Fields & Data Points | | :--- | :--- | | **Identity & Core Identifiers** | `title` (Title) | | **Metrics, Pricing & Engagement** | `wordCount` (Word Count), `characterCount` (Character Count) | | **Details, Attributes & Context** | `onlyMainContent` (Only Main Content), `description` (Description), `favicon` (Favicon), `markdown` (Markdown) | | **Direct Links, Media & Timestamps** | `urls` (URLs), `includeImages` (Include Images), `includeLinks` (Include Links), `url` (URL), `featuredImage` (Featured Image), `scrapedAt` (Scraped At) | *** ### Why scrape Web to Markdown? (Typical Use Cases & Guides) You can use **Web to Markdown Converter for AI & RAG Scraper** for a wide range of data and business workflows: - **RAG (Retrieval-Augmented Generation):** Clean web content before embedding into vector databases (Pinecone, Qdrant). - **AI Agent Web Browsing:** Give AI agents (LangChain, CrewAI, AutoGen) ultra-fast web reading capabilities. - **Content Archiving & Research:** Convert documentation, articles, and research whitepapers into markdown. - **AI Agents, RAG & LLM Knowledge Pipelines:** Stream clean, structured Web to Markdown JSON records into vector databases, Claude, GPT-4, or LangChain workflows. - **Automated BI Dashboards & Scheduled Alerts:** Schedule daily or hourly runs and push live Web to Markdown updates directly to Google Sheets, Looker Studio, Slack, or webhooks. - **[Market Research & Competitor Analysis](https://apify.com/use-cases/market-research):** Benchmark Web to Markdown offerings, pricing strategies, and market share across regions. - **[Lead Generation & CRM Enrichment](https://blog.apify.com/marketing-leads-scraping-with-apify/):** Build targeted prospect lists and sync verified records straight into HubSpot, Salesforce, or Clay. - **[AI Agents & RAG Pipelines via MCP](https://blog.apify.com/how-to-use-mcp/):** Connect live Web to Markdown data directly to Claude Desktop, Cursor, or LangChain agents. *** ### How to scrape Web to Markdown data **Web to Markdown Converter for AI & RAG Scraper** is designed for a zero-configuration start, even if you have never scraped data before: 1. [Create a free Apify account](https://console.apify.com/sign-up) using your email (includes **$5.00 in free monthly credits**). 2. Open **[Web to Markdown Converter for AI & RAG Scraper](https://apify.com/rbbscrap/web-to-markdown-scraper)** and click on the **[Input tab](https://apify.com/rbbscrap/web-to-markdown-scraper/input)** — all required fields are already pre-filled with a working example. 3. Enter your search terms, target URLs, or filter criteria (or keep the default pre-filled configuration for an instant 1-Click test). 4. Click the green **"Start"** button and wait a few seconds for the data to be extracted. 5. Download your dataset in **JSON, CSV, Excel (XLSX), or HTML**, or fetch it via the [API tab](https://apify.com/rbbscrap/web-to-markdown-scraper/api). > For more technical details on every parameter, check out the [Input Schema tab](https://apify.com/rbbscrap/web-to-markdown-scraper/input-schema) or read the [Apify Platform Documentation](https://docs.apify.com/platform/actors/running). #### 🔗 How to scrape Web to Markdown by Search Term or Direct URL
Scraping by Search Term / Filter 🔑 Enter keywords, categories, or filter parameters just as you would search on Web to Markdown. Set the maximum number of items you want to extract and click Start.
Scraping by Direct URL 🔗 Paste one or more direct Web to Markdown links (or import a list of URLs from a Google Sheet / CSV) to extract targeted item or profile details in bulk.
*** ### ⬇️ Input Configuration & Examples To start scraping Web to Markdown, fill in the visual input form in the Apify Console or pass a JSON payload via the API. Full technical specifications are available on the [Input Schema tab](https://apify.com/rbbscrap/web-to-markdown-scraper/input-schema). #### Input Parameters Reference | Parameter | Type | Default / Prefill | Description | | :--- | :---: | :---: | :--- | | `urls` | `array` | `["https://en.wikipedia.org/wiki/Web...` | List of web page URLs to crawl and convert to Markdown. | | `onlyMainContent` | `boolean` | `true` | Automatically remove navbars, footers, ads, and sidebars before converting to Markdown. | | `includeImages` | `boolean` | `true` | Keep image markdown syntax `![alt](url)` or remove images to save tokens. | | `includeLinks` | `boolean` | `true` | Keep markdown links `[text](url)` or strip them to plain text to save LLM tokens. | | `extractMetadata` | `boolean` | `true` | Include title, description, OG images, and word counts for vector database and RAG pipelines. | | `maxHeadingLevel` | `integer` | `6` | Maximum heading depth to preserve (e.g. 3 to convert H4-H6 into bold text). | | `maxPages` | `integer` | `10` | Maximum number of pages to process. | | `proxyConfiguration` | `object` | — | Apify Proxy configuration. | #### Example 1: Standard Extraction (1-Click Default) ```json { "urls": [ "https://en.wikipedia.org/wiki/Web_scraping" ], "onlyMainContent": true, "includeImages": true, "includeLinks": true } ``` #### Example 2: Filtered & Batch Extraction ```json { "onlyMainContent": true, "includeImages": true, "includeLinks": true, "extractMetadata": true, "maxHeadingLevel": 6, "maxPages": 20 } ``` *** ### ⬆️ Output Example (Table View, JSON & Fields Reference) The scraped results are stored in a structured Apify Dataset available in the **Storage** tab. Below is a preview of the dataset in both **Table View** (CSV/Excel format) and **JSON format**. #### Table View (CSV / Excel Dataset Preview) | URLs | Only Main Content | Include Images | Include Links | | :--- | :--- | :--- | :--- | | `https://en.wikipedia.org/wiki...` | `true` | `true` | `true` | | `https://en.wikipedia.org/wiki...` | `true` | `true` | `true` | | `https://en.wikipedia.org/wiki...` | `true` | `true` | `true` | #### JSON Output Example ```json { "urls": [ "https://en.wikipedia.org/wiki/Web_scraping" ], "onlyMainContent": true, "includeImages": true, "includeLinks": true } ``` #### Output Fields Reference | Field Name | Type | Description | | :--- | :---: | :--- | | `urls` | `array` | Extracted urls value. | | `onlyMainContent` | `boolean` | Extracted only main content value. | | `includeImages` | `boolean` | Extracted include images value. | | `includeLinks` | `boolean` | Extracted include links value. | | `url` | `string` | Extracted url value. | | `title` | `string` | Extracted title value. | | `description` | `string` | Extracted description value. | | `featuredImage` | `string` | Extracted featured image value. | | `favicon` | `string` | Extracted favicon value. | | `wordCount` | `integer / number` | Extracted word count value. | | `characterCount` | `integer / number` | Extracted character count value. | | `markdown` | `string` | Extracted markdown value. | | `scrapedAt` | `string` | Extracted scraped at value. | *** ### 🔗 Scrape any data you need with dedicated scrapers Combine **Web to Markdown Converter for AI & RAG Scraper** with our other dedicated scrapers from [@RBBscrap](https://apify.com/rbbscrap) if you want to extract specialized data across platforms:
🔹 Reddit Scraper 🔹 LinkedIn Jobs Scraper 🔹 Google Autocomplete & SEO Scraper
🔹 Hacker News Scraper 🔹 Glassdoor Scraper 🔹 YouTube Scraper
| Scraper | Extracted Data & Primary Use Case | | :--- | :--- | | **[Reddit Scraper](https://apify.com/rbbscrap/reddit-scraper)** | Posts, Comments & Subreddits | | **[LinkedIn Jobs Scraper](https://apify.com/rbbscrap/linkedin-jobs-scraper)** | B2B Leads & Salaries | | **[Google Autocomplete & SEO Scraper](https://apify.com/rbbscrap/google-autocomplete-keyword-scraper)** | Search Ideas | | **[Hacker News Scraper](https://apify.com/rbbscrap/hacker-news-scraper)** | Stories, Comments & News | | **[Glassdoor Scraper](https://apify.com/rbbscrap/glassdoor-scraper)** | Salaries, Reviews & Jobs | | **[YouTube Scraper](https://apify.com/rbbscrap/youtube-scraper)** | Videos, Shorts & Channels | *** ### ⚠️ Error Items & Resilience Reference **Web to Markdown Converter for AI & RAG Scraper** is engineered to process multi-query and batch runs resiliently without crashing if a single query or URL fails: | Status / Code | Meaning & Automatic Handling | | :--- | :--- | | `OK` | Record successfully extracted, validated, and normalized with zero `null` fields. | | `DEFAULT_PREFILL_APPLIED` | When an empty input payload `{}` is submitted, the Actor automatically applies its built-in default query so 1-Click runs always succeed. | | `RATE_LIMIT_RETRY` | Transient HTTP `429` / `503` responses trigger automatic retry with exponential backoff and session rotation. | | `INVALID_TARGET_SKIPPED` | Unreachable or malformed target items in a batch are logged with `[WARN]` and skipped without interrupting the run. | *** ### ❓ Frequently Asked Questions (FAQ) #### How much does scraping Web to Markdown cost? **Web to Markdown Converter for AI & RAG Scraper** uses transparent **Pay-Per-Event (`PAY_PER_EVENT`)** pricing set **10x cheaper than the market leader** ($\div 10$): - **Event Rate:** **$0.00025** per *Scraped Web to Markdown Converter for AI & RAG Record* (**$0.25 per 1,000 items**). - **🎁 1st Run 100% FREE:** Your very first execution is **100% free** (`$0.00` event charges are applied on your initial run). - **Apify Free Plan ($5.00/month included):** Every Apify account receives **$5.00 in free monthly credits**, which is enough to extract up to **~20,000 items per month for free**. - View full details on the [Pricing tab](https://apify.com/rbbscrap/web-to-markdown-scraper/pricing) and the [Apify Pricing page](https://apify.com/pricing). #### Can I integrate Web to Markdown Converter for AI & RAG Scraper with Make, Zapier, n8n, or Google Sheets? Yes. **Web to Markdown Converter for AI & RAG Scraper** can be connected with almost any cloud service or web app thanks to [integrations on the Apify platform](https://apify.com/integrations), including [Make](https://docs.apify.com/platform/integrations/make), [Zapier](https://docs.apify.com/platform/integrations/zapier), [n8n](https://docs.apify.com/platform/integrations/n8n), [Slack](https://docs.apify.com/platform/integrations/slack), [Airbyte](https://docs.apify.com/platform/integrations/airbyte), [Google Sheets](https://docs.apify.com/platform/integrations/google-sheets), [Google Drive](https://docs.apify.com/platform/integrations/drive), and [webhooks](https://docs.apify.com/platform/integrations/webhooks). #### Can I use Web to Markdown Converter for AI & RAG Scraper with the Apify API (Python & Node.js)? Yes. The Apify API gives you programmatic access to run **Web to Markdown Converter for AI & RAG Scraper**, schedule jobs, and fetch datasets. Use the `apify-client` [PyPI package](https://apify.com/rbbscrap/web-to-markdown-scraper/api/python) for Python or the `apify-client` [NPM package](https://apify.com/rbbscrap/web-to-markdown-scraper/api/javascript) for Node.js, or check the [Apify API Reference](https://docs.apify.com/api/v2): **Python (`apify-client`)** ```python from apify_client import ApifyClient client = ApifyClient("YOUR_APIFY_API_TOKEN") run = client.actor("rbbscrap/web-to-markdown-scraper").call(run_input={ "urls": ["https://en.wikipedia.org/wiki/Web_scraping"], "onlyMainContent": True, "includeImages": True, "includeLinks": True }) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) ``` **Node.js (`apify-client`)** ```javascript import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: 'YOUR_APIFY_API_TOKEN' }); const run = await client.actor('rbbscrap/web-to-markdown-scraper').call({ urls: ["https://en.wikipedia.org/wiki/Web_scraping"], onlyMainContent: true, includeImages: true, includeLinks: true }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); console.log(items); ``` #### Can I use Web to Markdown Converter for AI & RAG Scraper through an MCP Server (Claude Desktop, Cursor, AI Agents)? Yes. With the Apify API, you can use **Web to Markdown Converter for AI & RAG Scraper** in conjunction with an MCP server to connect directly to **Claude Desktop, Cursor, Windsurf, or LibreChat**. Read the official guide on [how to set up Apify Actors with MCP](https://blog.apify.com/how-to-use-mcp/), or go to the [MCP tab](https://apify.com/rbbscrap/web-to-markdown-scraper/api/mcp): 1. Start a Server-Sent Events (SSE) session to receive a `sessionId`. 2. Send API messages using that `sessionId` to trigger **Web to Markdown Converter for AI & RAG Scraper** with your input parameters. 3. Receive structured Web to Markdown dataset records directly inside your AI agent conversation. #### Do I need residential proxies or login cookies to scrape Web to Markdown? No. **Web to Markdown Converter for AI & RAG Scraper** operates on a zero-browser HTTP/2 protocol engine (256 MB RAM) and works out-of-the-box without requiring login cookies or external proxies for standard workloads. For high-volume enterprise runs, you can optionally use [Apify Proxy](https://docs.apify.com/proxy) in the input configuration. #### Is it legal to scrape data from Web to Markdown? Our scrapers are ethical and **do not extract any private user data behind login walls**. They only extract what is shared publicly on the web. However, please keep in mind that personal data is protected by **GDPR ([EU Regulation 2016/679](https://eur-lex.europa.eu/eli/reg/2016/679/oj))** and other regulations worldwide. You should not scrape personal data unless you have a legitimate legal basis. Read more in Apify's guide on the [legality of web scraping](https://blog.apify.com/is-web-scraping-legal/). #### Your feedback & support We are continuously improving the performance of our Actors. If you have any technical feedback for **Web to Markdown Converter for AI & RAG Scraper**, need a custom feature, or found a bug, please create an issue on the Actor's [Issues tab](https://apify.com/rbbscrap/web-to-markdown-scraper/issues/open) in the Apify Console. # Actor input Schema ## `urls` (type: `array`): List of web page URLs to crawl and convert to Markdown. ## `onlyMainContent` (type: `boolean`): Automatically remove navbars, footers, ads, and sidebars before converting to Markdown. ## `includeImages` (type: `boolean`): Keep image markdown syntax `![alt](url)` or remove images to save tokens. ## `includeLinks` (type: `boolean`): Keep markdown links `[text](url)` or strip them to plain text to save LLM tokens. ## `extractMetadata` (type: `boolean`): Include title, description, OG images, and word counts for vector database and RAG pipelines. ## `maxHeadingLevel` (type: `integer`): Maximum heading depth to preserve (e.g. 3 to convert H4-H6 into bold text). ## `maxPages` (type: `integer`): Maximum number of pages to process. ## `proxyConfiguration` (type: `object`): Apify Proxy configuration. ## `maxExecutionTimeSecs` (type: `integer`): Fail-Fast safety ceiling timeout in seconds (15 to 90s). Prevents long-running runs from wasting compute units. ## Actor input object example ```json { "urls": [ "https://en.wikipedia.org/wiki/Web_scraping" ], "onlyMainContent": true, "includeImages": true, "includeLinks": true, "extractMetadata": true, "maxHeadingLevel": 6, "maxPages": 10, "maxExecutionTimeSecs": 75 } ``` # Actor output Schema ## `results` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "urls": [ "https://en.wikipedia.org/wiki/Web_scraping" ], "onlyMainContent": true, "includeImages": true, "includeLinks": true, "extractMetadata": true, "maxHeadingLevel": 6, "maxPages": 10, "maxExecutionTimeSecs": 75 }; // Run the Actor and wait for it to finish const run = await client.actor("rbbscrap/web-to-markdown-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "urls": ["https://en.wikipedia.org/wiki/Web_scraping"], "onlyMainContent": True, "includeImages": True, "includeLinks": True, "extractMetadata": True, "maxHeadingLevel": 6, "maxPages": 10, "maxExecutionTimeSecs": 75, } # Run the Actor and wait for it to finish run = client.actor("rbbscrap/web-to-markdown-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "urls": [ "https://en.wikipedia.org/wiki/Web_scraping" ], "onlyMainContent": true, "includeImages": true, "includeLinks": true, "extractMetadata": true, "maxHeadingLevel": 6, "maxPages": 10, "maxExecutionTimeSecs": 75 }' | apify call rbbscrap/web-to-markdown-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,rbbscrap/web-to-markdown-scraper" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/fmxg63aIqfhEFeojn/builds/eO2JcAmVHImhLSQQW/openapi.json