Website Content Crawler to Markdown for RAG & LLMs

Website content crawler to Markdown for RAG and LLMs: crawl any site into clean Markdown, RAG chunks and llms.txt. A low-cost Website Content Crawler alternative: flat $0.30 per 1,000 pages, fast HTTP crawling, no compute bill to estimate.

What you get

For every page of the site:

• Clean Markdown of the main content: no menus, sidebars, footers or cookie banners. GitHub-flavoured tables, code blocks and absolute links.
• Optional RAG chunks split at headings, paragraphs and code blocks, with token estimates and the heading path.
• Metadata: title, description, H1, language, last modified date, canonical URL and a content hash for change detection.
• An llms.txt index and an llms-full.txt file per domain in the key-value store.

It is a fast HTTP crawler with no browser, so a 200 page documentation site takes about a minute. Pages that are only an empty JavaScript shell can be rendered in headless Chrome on request. Duplicates, robots.txt blocks, errors and empty pages are never charged.

Typical uses:

• Knowledge base for a chatbot or AI agent. Crawl docs, a help center or a blog and load the chunks into a vector database.
• LLM fine-tuning data. Clean Markdown, one record per page, with language and word count.
• llms.txt for your own site. A ready index and a full text file that AI assistants can read.
• Keep an index fresh. Schedule weekly runs. contentHash and lastModified show which pages changed.

Why pick this over a browser-based crawler

	This Actor	Apify Website Content Crawler
Pricing	Flat $0.30 per 1,000 HTTP pages, $6 per 1,000 browser pages	Free Actor, you pay platform compute for the run
Default fetching	Plain HTTP, browser only as an opt-in fallback	Adaptive, can use a headless browser
Cost to plan	Pages x price, capped by Max pages	Depends on memory, browser use and run time
Best for	Docs, blogs, help centers, marketing sites, most server-rendered sites	Heavy JavaScript apps and extras such as file downloads

Most documentation and content sites, including Docusaurus, GitBook, WordPress, MkDocs and Next.js sites, ship HTML and work over plain HTTP. For those, a flat price per page is the cheapest way to get RAG data.

Quick start

1. Open the Input tab and paste one or more start URLs, for example https://docs.example.com/.
2. Set Max pages. This is also your cost cap.
3. Optional: turn on Split into RAG chunks and pick a chunk size.
4. Click Start. The prefilled example saves 10 pages of Apify Academy in under 30 seconds.
5. Download the dataset as JSON, CSV, Excel or HTML, or open the llms.txt files in the Storage tab.

Single page mode: set Max crawl depth to 0. Only the start URLs are fetched, which turns the Actor into a URL to Markdown converter for a list of pages.

By default the crawler stays under the start URL path: https://example.com/docs crawls /docs and everything below it. Redirects are followed, and the crawl stays under the new path.

From code, one HTTP call runs the crawl and returns the pages:

curl -X POST "https://api.apify.com/v2/acts/rod_analytics~website-to-markdown/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"startUrls": [{"url": "https://docs.apify.com/academy"}], "maxPages": 50, "includeChunks": true}'

Load the chunks into LangChain documents:

from apify_client import ApifyClient
from langchain_core.documents import Document

client = ApifyClient("APIFY_TOKEN")
run = client.actor("rod_analytics/website-to-markdown").call(
    run_input={"startUrls": [{"url": "https://docs.example.com/"}], "maxPages": 500, "includeChunks": True}
)
docs = [
    Document(page_content=c["text"], metadata={"source": page["url"], "title": page["title"], "headings": c["headings"]})
    for page in client.dataset(run["defaultDatasetId"]).iterate_items()
    for c in page.get("chunks") or []
]

Input example

{
    "startUrls": [{ "url": "https://docs.apify.com/academy" }],
    "maxPages": 10,
    "includeChunks": true,
    "chunkSize": 500,
    "chunkOverlap": 50
}

Field	What it does	Default
startUrls	Where the crawl starts
maxPages	Stop after this many saved pages	100
maxCrawlDepth	Links away from a start URL. 0 fetches only the start URLs	20
includeGlobs / excludeGlobs	URL patterns, ** matches any path. .../blog/** also matches .../blog	none
sameDomainOnly / stayWithinStartPath	Crawl scope	on / on
useSitemaps	Also queue URLs from sitemap.xml	off
respectRobotsTxt	Skip pages disallowed by robots.txt	on
extractionMode	readability, main or full page	readability
removeSelectors	Extra CSS selectors to drop	none
includeChunks, chunkSize, chunkOverlap	RAG chunks, sizes in estimated tokens	off, 500, 50
dedupeContent	Skip same canonical URL or identical content	on
jsRenderingFallback	Render near-empty pages in headless Chrome	off
maxConcurrency, maxConcurrencyPerDomain, delayBetweenRequestsMs	Politeness per domain	10, 5, 0

Output sample

One dataset item per page, shortened. The Output tab has four views: Overview, Markdown, Metadata and RAG chunks.

{
    "url": "https://docs.apify.com/academy/api-scraping",
    "canonicalUrl": "https://docs.apify.com/academy/api-scraping",
    "title": "API scraping | Academy | Apify Documentation",
    "description": "Learn all about how the professionals scrape various types of APIs with various configurations, parameters, and requirements.",
    "language": "en",
    "h1": "API scraping",
    "markdown": "# API scraping\n\nAPI scraping is locating a website's API endpoints, and fetching the desired data directly from their API...",
    "wordCount": 868,
    "chunkCount": 6,
    "chunks": [
        {
            "index": 0,
            "headings": "API scraping",
            "text": "# API scraping\n\nAPI scraping is locating a website's API endpoints...",
            "charCount": 1790,
            "tokenEstimate": 448
        }
    ],
    "linksCount": 73,
    "depth": 1,
    "statusCode": 200,
    "fetchMode": "http",
    "contentHash": "3f1c0a9e5b7d2c4e8a61",
    "crawledAt": "2026-09-27T11:32:57.120Z",
    "warning": null
}

The key-value store also holds:

• llms-<domain>.txt: an llms.txt index with the title, URL and description of every page.
• llms-full-<domain>.txt: all Markdown of the domain in one file, each page wrapped in <page url="..." title="...">. Split into parts above 8 MB.
• RUN-SUMMARY: saved, duplicate, blocked and failed counts, plus the first 500 problem URLs.

Pricing

Pay per event, no compute charges.

Event	Free tier	Bronze	Silver	Gold
Page crawled over HTTP	$0.0003	$0.00027	$0.00024	$0.00021
Page rendered in Chrome, opt-in	$0.006	$0.0054	$0.0048	$0.0042
Run start, per GB of memory	$0.001	$0.001	$0.001	$0.001

• 1,000 pages cost $0.30 on the free tier and $0.21 on Gold. 10,000 pages cost $3.
• Duplicates, robots.txt blocks, errors and empty pages are not charged.
• The Apify free plan's $5 monthly credit covers about 16,000 pages.
• Your discount tier follows your Apify subscription plan.
• Set Maximum cost per run in Run options and the crawler stops exactly at that budget.

FAQ

Does it work on JavaScript sites? Most sites ship HTML and work over plain HTTP. Pure single page apps return an empty shell. Enable JavaScript rendering fallback for them, at the browser page price.

Does it respect robots.txt? Yes, by default. Disallowed pages are skipped and listed in RUN-SUMMARY. You can turn it off only at your own responsibility, and the run then logs a warning.

How are duplicates detected? By final URL, by rel=canonical and by a hash of the Markdown.

Which proxies can I use? No proxy, which is the default, Apify datacenter proxy, or your own proxy URLs. Residential and SERP proxies are not supported. A run that asks for them stops at the start with a clear message.

What if nothing is saved? The run still ends as succeeded. The dataset holds one free row with errorCode and howToFix, for example for an empty input, a 404 start page or a JavaScript-only site.

Tips. Use include globs such as https://example.com/docs/** to skip blogs and tags. Turn on sitemaps to find pages not linked from the menu. Keep 1024 MB memory for HTTP runs and 2048 MB with the JavaScript fallback. 300 to 800 token chunks with 10% overlap suit most embedding models.

Known limits. No login or forms. PDF and other files are skipped. Very large pages are cut at 5 MB of HTML and 3.5 million characters of Markdown, with a warning. Hash-routed single page apps (/#/page) show as one URL. Token counts are estimates, about 4 characters per token.

Is crawling legal? Crawling public pages is generally allowed, but you are responsible for each site's terms, copyright and privacy law. Do not crawl personal data without a legal basis.

Found a bug or need an export to your vector database? Open an issue in the Issues tab.

Related Actors

Actor	What it does
PDF to Markdown & Text Extractor with OCR	Turn PDF, DOCX and scans into Markdown or text
SEO Audit Tool: Site Crawler, Broken Links & Score	Crawl a site for broken links and SEO issues
PageSpeed Insights & Lighthouse Bulk Checker	Core Web Vitals and Lighthouse scores for many URLs
Tech Stack Detector: Wappalyzer & BuiltWith Alternative	See which CMS, analytics and frameworks a site uses
Bulk Domain Lookup: DNS, WHOIS, SPF, DMARC & SSL	DNS, mail security and SSL checks for domain lists
Speech to Text: Audio, Video & Podcast Transcriber	Transcribe audio, video and podcasts
Email Validator & Verifier: Bulk MX & Disposable Check	Clean email lists before you send
Website Contact Scraper: Emails, Phones & Social Links	Emails, phones and social links from company websites
VIES VAT Checker & EU Company Registry Lookup	Validate EU VAT numbers and look up companies
Companies House Scraper: UK Company Search & Officers	UK company profiles, officers and filings
EU TED Tenders Scraper & Alerts: Public Procurement	EU public tenders with daily alerts
Greenhouse, Lever & Ashby Job Scraper: 7 ATS in One Feed	Jobs from 7 ATS platforms in one feed
GitHub Scraper: Repo Stats, Stars, Trending & Search	Stars, activity and trending GitHub repos