Website to Markdown — LLM-Ready Content Crawler for RAG

Crawl any website and get clean, LLM-ready Markdown for every page: main content only (no menus, footers or cookie banners), absolute links, tables, title, description, author and publish date. For RAG, vector databases, AI agents and training data. Flat $1 per 1,000 pages.

Turn any website into clean, LLM-ready Markdown. Give it a start URL and it crawls the site (following links and the sitemap, staying on the site and under the path you gave) and returns one row per page with the main content only. Menus, sidebars, footers, cookie banners and buttons are removed using Firefox Reader View's algorithm, and the page body is the fallback when there is no article. You get Markdown with absolute links, images and tables, plus title, description, language, author, publish date, canonical URL and word count, optionally with plain text, cleaned HTML and the page's links. Use it to feed RAG pipelines and vector databases (Pinecone, Qdrant, Weaviate), build chatbot knowledge bases from documentation, give AI agents a fast web-to-text tool, create fine-tuning datasets, migrate content, or monitor competitors' docs and blogs. Flat, predictable pricing: $1 per 1,000 pages, with no compute bill on top and no 8 GB browser runs. Duplicate pages, non-HTML files, JavaScript-only pages and pages blocked by robots.txt are never charged.

What data does it return?

Every run outputs clean, schema-validated rows like these:

url	title	wordCount	language	markdown
https://docs.apify.com/actors/development	Actor development - Platform - Apify Documentation	412	en	## Actor development\n\nActors are serverless programs…
https://simonwillison.net/2024/Dec/31/llms-in-2024/	Things we learned about LLMs in 2024	7262	en-gb	31st December 2024\n\nA lot has happened in the world of Large Language Models…
https://en.wikipedia.org/wiki/Retrieval-augmented_generation	Retrieval-augmented generation - Wikipedia	2382	en	Retrieval-augmented generation (RAG) is a technique that enables large language models…

How much does it cost?

Transparent pay-per-event pricing — you only pay for what you get, and the actor always respects your run budget: set a maximum total charge and it stops cleanly at the cap instead of overrunning.

Event	Price	Charged	Worked example
page-result	$0.001	per page converted and delivered (all output formats included)	100 pages = $0.10; a 1,000-page documentation site = $1.00
apify-actor-start	$0.00005	per run per GB of memory (platform start fee)	One run at up to 1 GB = $0.00005; at 4 GB = $0.0002

Input options

Field	Type	Required	Default	Description
startUrls	array	yes	—	Where to start crawling. The crawl stays on the same site and under each URL's path: https://docs.example.com/guides crawls only the guides; a bare domain crawls the whole site. Use include globs to widen or narrow it.
maxPages	integer	no	100	Stop after converting this many pages (across all start URLs). To convert only the start URLs themselves, set Max link depth to 0.
maxDepth	integer	no	10	How many links away from a start URL to follow. 0 = only the start URLs themselves (no crawling, no sitemaps).
includeSubdomains	boolean	no	false	Also crawl subdomains of the start site (e.g. blog.example.com when starting at example.com). Pages on another host are only reached when you also set include globs that cover them, e.g. https://*.example.com/**.
includeGlobs	array	no	[]	Only crawl URLs matching one of these, e.g. https://example.com/blog/** (** = anything, * = anything except /). Replaces the default 'stay under the start path' rule.
excludeGlobs	array	no	[]	Never crawl URLs matching these, e.g. /tag/ or */login.
useSitemap	boolean	no	true	Also discover pages from the site's sitemap.xml (and sitemaps listed in robots.txt). Finds pages that links miss.
onlyMainContent	boolean	no	true	Keep just the article or main content (Firefox Reader View's algorithm), dropping menus, sidebars, footers and banners. Turn off to keep the whole page body (navigation is still removed).
removeSelectors	array	no	[]	Extra elements to strip before conversion, e.g. .newsletter-signup or #comments.
outputFormats	array	no	["markdown"]	Which content fields to return for each page. Same price whatever you pick.
includeLinks	boolean	no	false	Add the list of absolute links found on each page.
minWords	integer	no	0	Skip thin pages (tag lists, empty archives) with fewer words than this. Skipped pages are not charged.
respectRobotsTxt	boolean	no	true	Skip pages that the site's robots.txt disallows for crawlers.

Limits & honest notes

• Pages are fetched as static HTML without a browser, which is fast, cheap and works for most documentation, blogs, news, knowledge bases and e-commerce sites. Pages whose content is rendered only by JavaScript (single-page apps) are detected, reported and not charged.
• The crawl stays on the start URL's host and under its path, unless you set include globs. To cover subdomains or both the www and bare domain, add include globs such as https://*.example.com/**.
• Pages disallowed by robots.txt are skipped by default. If a site's robots.txt can't be fetched (HTTP 5xx or a network error, after one retry), the whole site is skipped (RFC 9309) and its start URL is reported as skipped, not charged. Login-protected content is not supported. Some sites block datacenter IPs; those start URLs are reported with their HTTP error and not charged.
• Duplicates are removed by normalised URL (tracking parameters and fragments ignored), redirect target and identical content, and are never charged twice.
• Very large sites: results stream as they are crawled, and runs resume after a platform migration without re-charging pages already delivered.
• Respects your run budget: set a maximum total charge and the crawl stops cleanly at the cap.

Related ActorForge actors

• Sitemap URL Extractor — list every URL of a site first
• PDF to Markdown — the same clean Markdown for PDFs
• SEO Audit Crawler
• Website Change Monitor

Changelog

• 2026-10-18 — Initial release: same-site crawling with sitemaps, path scoping and globs; Readability main-content extraction; Markdown/text/HTML outputs; metadata; JavaScript-page detection; dedupe; migration-safe resume.

---

Built and maintained by ActorForge. Reliability first: every actor in the portfolio is monitored by an automated nightly self-test, and breakages are typically fixed the same day.