Website to Markdown — Content Crawler for LLM & RAG avatar

Website to Markdown — Content Crawler for LLM & RAG

Pricing

from $0.60 / 1,000 results

Try for free
Go to Apify Store
Website to Markdown — Content Crawler for LLM & RAG

Website to Markdown — Content Crawler for LLM & RAG

Try for free

Content Crawler turns any site into clean Markdown per page for LLMs, RAG pipelines and vector DBs — no headless browser, $1 per 1,000 pages.

Pricing

from $0.60 / 1,000 results

Rating

0.0

(0)

Developer

Murat Uzun

Murat Uzun

Maintained by Community

Actor stats

0

Bookmarked

5

Total users

2

Monthly active users

2 days ago

Last modified

Categories

AI

Developer tools

Agents

Share

What is Website to Markdown Crawler?

Website to Markdown Crawler is an Apify Actor that turns any website into clean, LLM-ready Markdown — one row per page, with navigation, footers, sidebars and cookie banners stripped, headings and code blocks preserved, and every link made absolute. It runs on plain HTTP with Crawlee's CheerioCrawler — no headless browser — which is what makes it fast and roughly 10× cheaper than browser-based content crawlers. Feed it a start URL (or a whole /sitemap.xml) and it does a breadth-first crawl up to your page budget, ready to feed an LLM, a RAG pipeline, LangChain/LlamaIndex loader, or a vector DB (Pinecone, Qdrant, Weaviate, pgvector) in minutes.

Pricing: Website to Markdown costs $1.00 per 1,000 results (pay per result, no subscription; Apify's free plan credit covers small runs).

Why use Website to Markdown Crawler?

  • Feed an LLM or RAG pipeline in minutes. Point it at your docs site, a competitor's help center, or your own product site and get clean Markdown ready for chunking and embedding — no HTML soup to clean up first.
  • 10× cheaper, no browser. apify/website-content-crawler and similar Actors spin up a browser per page. This Actor fetches raw HTML over HTTP, so it's faster and priced at $1 per 1,000 pages.
  • Predictable cost. maxPages is the billed unit and the crawl stops at exactly that count — no surprise runs.
  • Sitemap-aware BFS crawl. Optionally seeds from /sitemap.xml (including nested sitemap indexes) so you reach real content pages immediately instead of waiting for link discovery.

How to use Website to Markdown Crawler

  1. Paste one or more pages into Start URLs, e.g. https://docs.apify.com/platform or your own docs/blog root.
  2. Set Max pages — this is the billed unit ($1 per 1,000 pages) and the crawl stops exactly there.
  3. Optionally set Include path prefixes (e.g. /docs) to stay inside one section of a larger site.
  4. Choose Output format: Markdown, plain text, or both.
  5. Click Start, then download the dataset as JSON, CSV, Excel or HTML, or pull it straight into your RAG pipeline via the API.

Example input

{
  "startUrls": [{ "url": "https://docs.apify.com/platform" }],
  "maxPages": 50,
  "maxDepth": 3,
  "sameDomainOnly": true,
  "useSitemap": true,
  "outputFormat": "markdown"
}

Example output

{
  "url": "https://crawlee.dev/docs",
  "finalUrl": "https://crawlee.dev/js/docs/quick-start",
  "statusCode": 200,
  "title": "Quick Start | Crawlee for JavaScript",
  "description": "Crawlee helps you build reliable scrapers. Fast.",
  "lang": "en",
  "canonical": "https://crawlee.dev/js/docs/quick-start",
  "markdown": "# Quick Start\n\nWith this short tutorial you can start scraping with Crawlee in a minute or two...\n\n## Choose your crawler\n\n### CheerioCrawler\n\nThis is a plain HTTP crawler...",
  "text": null,
  "wordCount": 1484,
  "headings": ["Quick Start", "Choose your crawler", "CheerioCrawler", "PuppeteerCrawler"],
  "links": ["https://crawlee.dev/js/docs/introduction", "https://crawlee.dev/js/docs/guides"],
  "depth": 0,
  "contentHash": "3f9a1c2e...",
  "crawledAt": "2026-09-12T18:21:07.000Z",
  "error": null
}

What data does Website to Markdown Crawler extract?

One row per crawled page:

FieldTypeDescription
url / finalUrlstringRequested URL and URL after redirects
statusCodeintegerHTTP status code, null on a failed request
title / descriptionstringPage <title> (or first <h1>) and meta description
langstring<html lang> attribute
canonicalstring<link rel=canonical>, resolved to an absolute URL
markdownstringMain content as Markdown — headings, lists, code blocks, tables, absolute links/images
textstringMain content as plain text
wordCountintegerWord count of the extracted content
headingsarrayH1-H3 text, in order, up to 50
linksarraySame-domain absolute links found on the page, up to 200
depthintegerLink depth from the nearest start URL
contentHashstringSHA-1 of the plain text — diff two runs to detect changed pages
crawledAtstringISO timestamp
errorstringAlways null: pages that fail are not charged and are listed in the run's ERRORS record instead
chunksarrayWith Add RAG chunks: { text, tokens, headingPath } per chunk, otherwise null
changeStatusstringWith Detect changes: new, changed or unchanged, otherwise null

markdown/text are populated according to the Output format input (markdown, text, or both); the unused one is null.

Input parameters

ParameterTypeDefaultDescription
startUrlsarraydocs.apify.com/platformPage(s) to start crawling from
maxPagesinteger50Billed unit; the crawl stops at exactly this count (1-5000)
maxDepthinteger3Link depth from a start URL (0-10)
sameDomainOnlybooleantrueOnly follow links on the same domain (www. counts as the same)
includePathPrefixesarraynoneOnly crawl paths starting with one of these, e.g. /docs
excludePathPatternsarrayimages/PDF/CSS/JS, /login, /signup, ?replytocom=Regexes tested against the full URL; a match is skipped
useSitemapbooleantrueAlso seed from /sitemap.xml (follows one nested sitemap-index level)
outputFormatstringmarkdownmarkdown, text, or both
removeSelectorsarraynoneExtra CSS selectors to strip before extraction
maxConcurrencyinteger10Parallel requests (1-50)
respectRobotsbooleantrueSkip URLs disallowed by robots.txt
proxyConfigurationobjectnoneOptional Apify Proxy configuration
chunkForRagbooleanfalseAdd a chunks array (heading-aware, exact o200k token counts) to every page
chunkSizeinteger500Max tokens per chunk (50-8000)
changeDetectionbooleanfalseAdd changeStatus (new / changed / unchanged) versus the previous run with the same start URLs
skipUnchangedContentbooleantrueWith change detection on, unchanged pages get empty markdown/text/chunks

Pricing

Website to Markdown Crawler uses pay-per-event pricing: $1 per 1,000 pages ($0.001 per page), plus a negligible actor-start fee. There is no separate compute-unit or browser billing because no browser is used. Set Max pages (and, on the platform, Maximum cost per run) to cap spend — the Actor trims its own crawl to stay within budget and never bills more pages than it delivers.

Website to Markdown Crawler vs. browser-based content crawlers

Browser-based crawlers such as apify/website-content-crawler load every page in a real (often headless) browser before extracting content — that's necessary for JavaScript-rendered single-page apps, but it's slow and expensive for the huge majority of documentation sites, blogs and marketing pages that are plain server-rendered HTML. This Actor skips the browser entirely: a page that takes a browser-based crawler several seconds and a full render cycle is a sub-second HTTP fetch here, at a fraction of the price. The trade-off is explicit: sites that only render content client-side with JavaScript (pure SPAs) will come back with little or no text — see Limitations below.

RAG-ready chunks in the same run

Turn on Add RAG chunks and every page row gets a chunks array: heading-aware pieces of the Markdown, each with its heading path and an exact GPT (o200k) token count — embed them straight into Pinecone, Qdrant, pgvector or any vector DB. Code blocks stay whole and every chunk is at most Chunk size tokens. No extra charge: chunks are part of the page result.

Keep a knowledge base fresh: change detection

Turn on Detect changes since last run and schedule the Actor. Each page gets changeStatus — new, changed or unchanged — compared with the previous run that used the same start URLs (the state is kept in a key-value store in your own account). With Leave content empty for unchanged pages on, unchanged rows carry no Markdown, so you only re-embed what changed. Every crawled page still counts as one result.

Using Website to Markdown Crawler with AI agents, LangChain and LlamaIndex

Website to Markdown Crawler runs on pay-per-event pricing with limited permissions, so it's callable through the Apify MCP server directly from an AI agent — pass startUrls and maxPages and get back one Markdown row per page. The dataset also drops straight into a RAG pipeline: use Apify's LangChain or LlamaIndex ApifyDatasetLoader to turn the dataset into Document objects, then chunk and embed into Pinecone, Qdrant, Weaviate or pgvector. It also connects through n8n, Make and Zapier via Apify's standard integrations.

Limitations

  • JavaScript-rendered single-page apps are the main limitation. This Actor fetches raw HTML over HTTP and never executes page JavaScript, so a site that renders its content entirely client-side (no content in the initial HTML) will come back with an empty or near-empty row. It does follow <meta http-equiv="refresh"> redirect stubs (one hop) and ordinary HTTP redirects, which covers a common class of "empty landing page" issue, but not full client-side rendering. If you hit this, look for a browser-based crawler instead.
  • Interactive widgets that swap content via JavaScript (tabs, accordions with lazy content) are captured in whatever state they render in the raw HTML — usually just the first/default tab.
  • Short run timeouts end cleanly. If the run timeout is reached before maxPages, the crawler stops shortly before the deadline and the run finishes as SUCCEEDED with every page crawled so far (you pay only for those). Raise the timeout or lower maxPages for a full crawl.
  • The main-content heuristic (no @mozilla/readability) is tuned for documentation and blog layouts (article, main, #content, .content); unusual layouts may need removeSelectors to clean up.

FAQ

Is this legal? The Actor reads only public HTML. You're responsible for crawling sites you're allowed to and respecting their terms of service.

Why is a row empty? Almost always a JavaScript-rendered page — see Limitations above. Check the statusCode field (failed pages are in the ERRORS record); a 200 with an empty markdown usually means client-side rendering.

How do I stay inside one section of a site? Set includePathPrefixes, e.g. ["/docs"], and/or turn off useSitemap if the sitemap covers more than you want.

How do I control cost? Set maxPages; the Actor never crawls or bills past it. On the platform, also set Maximum cost per run.

Can I detect changed pages between runs? Yes — compare contentHash (SHA-1 of the extracted text) across runs to see which pages changed.

Does this work with n8n, Make or Zapier? Yes, through Apify's standard integrations, and through LangChain/LlamaIndex document loaders for RAG pipelines.

Part of the webdatatools web-intelligence suite — every Actor is pay-per-event, reads public data without a login, and returns one clean row per entity:

Browse the whole suite at webdatatools, or call ten of these Actors straight from Claude, Cursor or Cline with the webdatatools MCP server.

Website & domain intelligence

Content for AI, LLMs and RAG

Search, video and social

Leads, jobs and company data

Developer, app and research data

Support and feedback

Found a bug or want a feature? Open an issue on the Issues tab.

You might also like

Website Content Extractor - Markdown for LLM, RAG, Vector DB avatar

Website Content Extractor - Markdown for LLM, RAG, Vector DB

snow_leo_data/website-to-markdown

Readable text from 73.1% of 52 live pages over plain HTTP, including 10 of 10 documentation sites. Website to LLM text and a website to markdown converter in one run: clean Markdown, chunks with overlap for a vector database, 46 flat fields, and what changed since the last run.

User avatar

Snow Leo Data

3

AI Web Search Scraper - SERP & Page Content for LLM RAG avatar

AI Web Search Scraper - SERP & Page Content for LLM RAG

get_anything/ai-web-search-scraper

Web search API for LLMs: query -> clean results (title, URL, snippet, position, domain) and OPTIONALLY each page's full content as LLM-ready Markdown. Ground ChatGPT/Claude and RAG pipelines with live web results. Google (browser-rendered) or DuckDuckGo engine. Export JSON, CSV or Markdown.

User avatar

Get Anything

3

Google Search Results (SERP) & Ads Scraper avatar

Google Search Results (SERP) & Ads Scraper

scrapeforgehq/google-serp-scraper

Scrape Google SERPs at scale: paid ads (headline, copy, displayed/final URL), organic results, People Also Ask, and related searches — by keyword, country, language, and device. Clean JSON output for SEO, PPC, and competitor ad research.

User avatar

ScrapeForge

3

Markdown to HTML Converter — Bulk avatar

Markdown to HTML Converter — Bulk

hipersoft/markdown-to-html

Convert Markdown to clean, styled HTML with GitHub-flavored tables, code highlighting and an optional full HTML document wrapper. For emails, CMS content and static sites.

User avatar

hiper soft

3

WT

Website to Markdown — Clean Page Text for LLMs

sourcedirect/website-to-markdown

Fetch any URL and return clean markdown/text plus links. No headless browser for most pages. Free demo: https://vaytric.com/page-to-markdown/ — bulk URL lists here.

User avatar

Kurt Landman

2

Web Page to Markdown for AI: $1/1K Pages, No Login avatar

Web Page to Markdown for AI: $1/1K Pages, No Login

conserving_celerytop/web-page-to-markdown

Turn any list of web pages into clean Markdown for LLMs, RAG and AI agents: main content without menus and ads, tables kept, plus title, description, language, dates and links. Plain HTTP first, JavaScript only when a page needs it. $1 per 1,000 pages, failures free.

User avatar

Don Mangu

2

WC

Website Content to Markdown & llms.txt Crawler - AI & RAG Ready

abi4/website-content-llmstxt-crawler

High-performance, lightweight asynchronous crawler converting web pages into clean Markdown, computing token estimates, and generating standard /llms.txt manifests for AI and RAG ingestion.

User avatar

Abi

2

Web Crawler: Website to Markdown for LLM & RAG avatar

Web Crawler: Website to Markdown for LLM & RAG

scrape-lads/website-content-crawler-rag

Website content crawler for AI. Crawl any site into clean Markdown, text and ready-to-embed RAG chunks for LLMs and vector databases. Removes navigation, footers and cookie banners, renders JavaScript only when needed. $2 per 1,000 pages.

User avatar

Scrape Lads

2

UT

URL to Agent Markdown

encouraged_offspring/url-to-agent-markdown

User avatar

Lav Chaudhary

2

Website to Markdown Crawler for AI, RAG & LLMs avatar

Website to Markdown Crawler for AI, RAG & LLMs

dima_kadirovich/website-to-markdown

Crawl any website and get clean main-content Markdown for every page: no menus or footers. RAG-ready chunks, llms.txt, and an 'only changed pages' mode for cheap scheduled refreshes. $1 per 1,000 pages, all included.

User avatar

Cronexa Data Tools

2