Pricing

from $0.60 / 1,000 results

Try for free

Go to Apify Store

Website to Markdown Crawler for LLM & RAG (fast, no browser)

Try for free

Website to Markdown Crawler turns any site into clean Markdown per page for LLMs, RAG pipelines and vector DBs — no headless browser, $1 per 1,000 pages.

Pricing

from $0.60 / 1,000 results

Rating

0.0

(0)

Developer

Murat Uzun

Actor stats

Bookmarked

Total users

Monthly active users

18 hours ago

Last modified

What is Website to Markdown Crawler?

Website to Markdown Crawler is an Apify Actor that turns any website into clean, LLM-ready Markdown — one row per page, with navigation, footers, sidebars and cookie banners stripped, headings and code blocks preserved, and every link made absolute. It runs on plain HTTP with Crawlee's CheerioCrawler — no headless browser — which is what makes it fast and roughly 10× cheaper than browser-based content crawlers. Feed it a start URL (or a whole /sitemap.xml) and it does a breadth-first crawl up to your page budget, ready to feed an LLM, a RAG pipeline, LangChain/LlamaIndex loader, or a vector DB (Pinecone, Qdrant, Weaviate, pgvector) in minutes.

Why use Website to Markdown Crawler?

Feed an LLM or RAG pipeline in minutes. Point it at your docs site, a competitor's help center, or your own product site and get clean Markdown ready for chunking and embedding — no HTML soup to clean up first.
10× cheaper, no browser. apify/website-content-crawler and similar Actors spin up a browser per page. This Actor fetches raw HTML over HTTP, so it's faster and priced at $1 per 1,000 pages.
Predictable cost. maxPages is the billed unit and the crawl stops at exactly that count — no surprise runs.
Sitemap-aware BFS crawl. Optionally seeds from /sitemap.xml (including nested sitemap indexes) so you reach real content pages immediately instead of waiting for link discovery.

How to use Website to Markdown Crawler

Paste one or more pages into Start URLs, e.g. https://docs.apify.com/platform or your own docs/blog root.
Set Max pages — this is the billed unit ($1 per 1,000 pages) and the crawl stops exactly there.
Optionally set Include path prefixes (e.g. /docs) to stay inside one section of a larger site.
Choose Output format: Markdown, plain text, or both.
Click Start, then download the dataset as JSON, CSV, Excel or HTML, or pull it straight into your RAG pipeline via the API.

Example input

{
  "startUrls": [{ "url": "https://docs.apify.com/platform" }],
  "maxPages": 50,
  "maxDepth": 3,
  "sameDomainOnly": true,
  "useSitemap": true,
  "outputFormat": "markdown"
}

Example output

{
  "url": "https://crawlee.dev/docs",
  "finalUrl": "https://crawlee.dev/js/docs/quick-start",
  "statusCode": 200,
  "title": "Quick Start | Crawlee for JavaScript",
  "description": "Crawlee helps you build reliable scrapers. Fast.",
  "lang": "en",
  "canonical": "https://crawlee.dev/js/docs/quick-start",
  "markdown": "# Quick Start\n\nWith this short tutorial you can start scraping with Crawlee in a minute or two...\n\n## Choose your crawler\n\n### CheerioCrawler\n\nThis is a plain HTTP crawler...",
  "text": null,
  "wordCount": 1484,
  "headings": ["Quick Start", "Choose your crawler", "CheerioCrawler", "PuppeteerCrawler"],
  "links": ["https://crawlee.dev/js/docs/introduction", "https://crawlee.dev/js/docs/guides"],
  "depth": 0,
  "contentHash": "3f9a1c2e...",
  "crawledAt": "2026-09-12T18:21:07.000Z",
  "error": null
}

What data does Website to Markdown Crawler extract?

One row per crawled page:

Field	Type	Description
`url` / `finalUrl`	string	Requested URL and URL after redirects
`statusCode`	integer	HTTP status code, `null` on a failed request
`title` / `description`	string	Page `<title>` (or first `<h1>`) and meta description
`lang`	string	`<html lang>` attribute
`canonical`	string	`<link rel=canonical>`, resolved to an absolute URL
`markdown`	string	Main content as Markdown — headings, lists, code blocks, tables, absolute links/images
`text`	string	Main content as plain text
`wordCount`	integer	Word count of the extracted content
`headings`	array	H1-H3 text, in order, up to 50
`links`	array	Same-domain absolute links found on the page, up to 200
`depth`	integer	Link depth from the nearest start URL
`contentHash`	string	SHA-1 of the plain text — diff two runs to detect changed pages
`crawledAt`	string	ISO timestamp
`error`	string	Error message on a failed page, otherwise `null`

markdown/text are populated according to the Output format input (markdown, text, or both); the unused one is null.

Input parameters

Parameter	Type	Default	Description
`startUrls`	array	`docs.apify.com/platform`	Page(s) to start crawling from
`maxPages`	integer	50	Billed unit; the crawl stops at exactly this count (1-5000)
`maxDepth`	integer	3	Link depth from a start URL (0-10)
`sameDomainOnly`	boolean	true	Only follow links on the same domain (`www.` counts as the same)
`includePathPrefixes`	array	none	Only crawl paths starting with one of these, e.g. `/docs`
`excludePathPatterns`	array	images/PDF/CSS/JS, `/login`, `/signup`, `?replytocom=`	Regexes tested against the full URL; a match is skipped
`useSitemap`	boolean	true	Also seed from `/sitemap.xml` (follows one nested sitemap-index level)
`outputFormat`	string	`markdown`	`markdown`, `text`, or `both`
`removeSelectors`	array	none	Extra CSS selectors to strip before extraction
`maxConcurrency`	integer	10	Parallel requests (1-50)
`respectRobots`	boolean	true	Skip URLs disallowed by `robots.txt`
`proxyConfiguration`	object	none	Optional Apify Proxy configuration

Pricing

Website to Markdown Crawler uses pay-per-event pricing: $1 per 1,000 pages ($0.001 per page), plus a negligible actor-start fee. There is no separate compute-unit or browser billing because no browser is used. Set Max pages (and, on the platform, Maximum cost per run) to cap spend — the Actor trims its own crawl to stay within budget and never bills more pages than it delivers.

Website to Markdown Crawler vs. browser-based content crawlers

Browser-based crawlers such as apify/website-content-crawler load every page in a real (often headless) browser before extracting content — that's necessary for JavaScript-rendered single-page apps, but it's slow and expensive for the huge majority of documentation sites, blogs and marketing pages that are plain server-rendered HTML. This Actor skips the browser entirely: a page that takes a browser-based crawler several seconds and a full render cycle is a sub-second HTTP fetch here, at a fraction of the price. The trade-off is explicit: sites that only render content client-side with JavaScript (pure SPAs) will come back with little or no text — see Limitations below.

Using Website to Markdown Crawler with AI agents, LangChain and LlamaIndex

Website to Markdown Crawler runs on pay-per-event pricing with limited permissions, so it's callable through the Apify MCP server directly from an AI agent — pass startUrls and maxPages and get back one Markdown row per page. The dataset also drops straight into a RAG pipeline: use Apify's LangChain or LlamaIndex ApifyDatasetLoader to turn the dataset into Document objects, then chunk and embed into Pinecone, Qdrant, Weaviate or pgvector. It also connects through n8n, Make and Zapier via Apify's standard integrations.

Limitations

JavaScript-rendered single-page apps are the main limitation. This Actor fetches raw HTML over HTTP and never executes page JavaScript, so a site that renders its content entirely client-side (no content in the initial HTML) will come back with an empty or near-empty row. It does follow <meta http-equiv="refresh"> redirect stubs (one hop) and ordinary HTTP redirects, which covers a common class of "empty landing page" issue, but not full client-side rendering. If you hit this, look for a browser-based crawler instead.
Interactive widgets that swap content via JavaScript (tabs, accordions with lazy content) are captured in whatever state they render in the raw HTML — usually just the first/default tab.
The main-content heuristic (no @mozilla/readability) is tuned for documentation and blog layouts (article, main, #content, .content); unusual layouts may need removeSelectors to clean up.

FAQ

Is this legal? The Actor reads only public HTML. You're responsible for crawling sites you're allowed to and respecting their terms of service.

Why is a row empty? Almost always a JavaScript-rendered page — see Limitations above. Check the statusCode and error fields; a 200 with an empty markdown usually means client-side rendering.

How do I stay inside one section of a site? Set includePathPrefixes, e.g. ["/docs"], and/or turn off useSitemap if the sitemap covers more than you want.

How do I control cost? Set maxPages; the Actor never crawls or bills past it. On the platform, also set Maximum cost per run.

Can I detect changed pages between runs? Yes — compare contentHash (SHA-1 of the extracted text) across runs to see which pages changed.

Does this work with n8n, Make or Zapier? Yes, through Apify's standard integrations, and through LangChain/LlamaIndex document loaders for RAG pipelines.

Part of the webdatatools web-intelligence suite — every Actor is pay-per-event, runs without proxies or a headless browser, and returns one clean row per entity:

Browse the whole suite at webdatatools, or call ten of these Actors straight from Claude, Cursor or Cline with the webdatatools MCP server.

Website & domain intelligence

Website Contact & Social Extractor — e-mails, phones and social profiles per domain
Website Tech Stack Detector — CMS, e-commerce, analytics, pixels and payments per domain
Domain DNS & Email Security Checker — SPF, DKIM, DMARC, MX provider, registrar and domain age
Domain Security Audit — TLS expiry, security headers, redirect chain, robots and llms.txt
Subdomain Finder (Certificate Transparency) — every subdomain seen in CT logs, with a live DNS check
Bulk Core Web Vitals & PageSpeed Audit — Lighthouse scores, LCP, CLS, INP and top fixes per URL
On-Page SEO Audit — title, meta, headings, links, images and schema issues per page
Sitemap URL Extractor & Change Monitor — every sitemap URL, or new and removed pages between runs
Wayback Machine Snapshot & Page Change Tracker — how a page changed over time, or every archived snapshot

Content for AI, LLMs and RAG

AI Web Search & Read — a query turned into clean Markdown from the top search results
Article & News Extractor — clean article text, author, date and Markdown per URL
Structured Data & JSON-LD Extractor — Schema.org and Open Graph data from any page
Google News Scraper — news results by keyword, topic or site
Press Release Monitor — PR Newswire, Business Wire and GlobeNewswire releases

Search, video and social

Google Search Results Scraper — organic SERP results per keyword and country
YouTube Comments Scraper — comments and replies with likes, no API key
YouTube Channel Latest Videos — the latest 15 videos of any channel from RSS
YouTube Channel Videos Scraper — a channel's full video, shorts and stream list
YouTube Search Results Scraper — videos, channels and playlists per query
YouTube Video Details Scraper — views, likes, description, tags and chapters per video
Apple Podcasts Lookup & Episodes Scraper — podcast metadata and episodes from iTunes and RSS
Bluesky Scraper — posts, profiles, followers and threads from the AT Protocol API

Leads, jobs and company data

Company 360 — one row per domain: contacts, tech, security, hiring and company facts
Hiring Signals Scraper — open jobs and hiring velocity from 10 public ATS boards
Y Combinator Companies & Founders Scraper — YC startups by batch, industry and hiring status
Wikidata Entity & Company Enrichment — HQ, founders, employees, revenue and social IDs per company
Bulk Email Validator — syntax, MX, disposable, role and free-provider checks
OpenStreetMap POI Extractor — shops and amenities by radius, bbox or area

Developer, app and research data

npm, PyPI & Crates.io Package Health Checker — releases, downloads, deprecation and a health score
GitHub Repository Health & Activity Report — stars, commits, contributors and risk flags per repo
VS Code Marketplace Extension Scraper — installs, ratings and versions per extension
Chrome Web Store Extension Scraper — users, rating, version and developer per extension
Google Play Store Scraper — apps, ratings, installs, developer contact and reviews
App Store (iOS) App Metadata & Top Charts — ratings, price, version and charts per app
CrossRef DOI & Citation Metadata Lookup — papers, authors, journals and citation counts
FDA Recalls & Adverse Events Monitor — food, drug and device recalls from openFDA
iCal / ICS Calendar Feed to Events Extractor — any public calendar feed as event rows
Shopify Store Products Scraper — catalog, prices, variants and stock per store

Support and feedback

Found a bug or want a feature? Open an issue on the Issues tab.

Website Contact & Social Extractor

webdatatools/contact-extractor

Crawl any website and get one clean row per domain with every e-mail, phone number and social profile it exposes.

Murat Uzun

Website to Markdown Crawler for LLM & RAG

logiover/website-text-markdown-crawler

Crawl any website to clean Markdown and plain text for LLM training and RAG. HTML to Markdown, no API or login. Export website text to CSV or JSON.

Logiover

Website to Markdown Scraper — LLM & RAG Ready

perforated_hummingbird/url-to-markdown

Scrape any website into clean, LLM-ready Markdown. This URL-to-Markdown converter strips ads, nav, and boilerplate with Mozilla Readability — feed your AI models and RAG pipelines only real content. Batch URLs, optional JavaScript rendering, pay only for pages scraped successfully.

Damon Williams

Website to Markdown — Content Extractor for LLMs & RAG

insight.solutions/website-content-extractor

Turn any website into clean Markdown and plain text for RAG pipelines, vector stores and LLM agents. Reads the sitemap or crawls links, strips navigation, cookie bars and footers, and returns one row per page with metadata, headings, links and optional chunks. Static HTML, no browser.

Insight Solutions

Website Tech Stack Detector

webdatatools/tech-stack-detector

Find out which CMS, e-commerce platform, analytics, ad pixels, chat widget and payment provider any website runs — one clean row per domain.

Murat Uzun

Sitemap to URL Crawler — Extract Sitemap.xml URLs

logiover/sitemap-to-url-crawler

Extract all URLs from any sitemap.xml recursively. Export sitemap URLs to CSV/JSON for RAG pipelines, SEO audits, and LLM training datasets.

Logiover

Website to Markdown Crawler - Full-Site Text for LLMs & RAG

entranced_gelato/website-to-markdown-crawler

Crawl any website from a start URL and get every page as clean text + Markdown for LLMs, RAG, and AI agents. Follows internal links with depth and page limits, strips nav and ads, and returns one structured record per page. A fast, no-config site-to-Markdown crawler.

AIDevs

LLM Markdown Crawler

sleek_waveform/llm-markdown-crawler

Crawl any website and extract clean, boilerplate-free Markdown optimized for LLMs, RAG pipelines, and AI training datasets. Uses Mozilla Readability to strip navigation and ads, then converts to clean Markdown. No browser required — fast and cheap.

Daniel Dimitrov

Website Content Crawler Pro — Markdown for AI & RAG

hipersoft/web-content-crawler-pro

Crawl any website into clean Markdown or text for LLMs, RAG and AI. HTTP + JS-browser modes, main-content (readability) extraction, sitemaps, URL glob filters, cookie-banner removal and CSS selectors. No setup.

hiper soft

Website Content Crawler - Markdown & Text for LLM / RAG

pear_fight/website-content-crawler-markdown-text-for-llm-rag

Crawl any website and extract clean article text and Markdown, ready to feed into LLMs, ChatGPT, vector databases and RAG pipelines. Removes navigation, ads and boilerplate. Configurable crawl depth and page limits. Export to JSON, CSV, Excel.