Website to Markdown for AI & RAG – Fast Content Crawler avatar

Website to Markdown for AI & RAG – Fast Content Crawler

Pricing

from $0.56 / 1,000 page scrapeds

Go to Apify Store
Website to Markdown for AI & RAG – Fast Content Crawler

Website to Markdown for AI & RAG – Fast Content Crawler

Crawl any website and get clean, LLM-ready Markdown, text or HTML for RAG, chatbots and AI agents. Optional chunking, token estimates and content hashes. Fast HTTP or browser crawling, $0.80 per 1,000 pages all-in.

Pricing

from $0.56 / 1,000 page scrapeds

Rating

0.0

(0)

Developer

kata Kuri

kata Kuri

Maintained by Community

Actor stats

1

Bookmarked

43

Total users

1

Monthly active users

a day ago

Last modified

Features

Limited permissions

Share

Website to Markdown for AI & RAG - Fast Content Crawler

Crawl any website and get clean, LLM-ready Markdown (or plain text / HTML) for RAG pipelines, chatbots, AI agents and vector databases. Built to be fast and cheap: $0.80 per 1,000 pages, all-in, with a plain HTTP mode that is roughly 10x faster than a browser for static sites and docs.

Why use it

  • LLM-ready output - Markdown with real tables, code blocks that keep their line breaks, and absolute links and image URLs.
  • RAG features built in - optional chunking (by characters or approximate tokens, with overlap), a token count estimate and a content hash (SHA-256) on every page for change detection and deduplication.
  • Cheaper and faster by design - HTTP (Cheerio) mode for static sites, a lean Chromium mode for JavaScript sites that skips images, fonts and media.
  • You only pay for real content - empty pages and anti-bot / captcha challenge pages are never saved, so never charged. They are listed in the FAILED_URLS record.
  • Honest run status - a crash ends the run as Failed; a run that hits its time budget stops gracefully, keeps everything crawled and reports Partial: stopped at time budget, N pages.

Key features

  • Crawler engines - Adaptive uses the Chromium browser and falls back to HTTP if the browser gets nothing (blocked). Or pick Chromium or Cheerio (HTTP) yourself.
  • Clean content extraction - removes navigation, the site header, footer, cookie banners, ads, modals and similar noise, and picks the main content area. A <header> inside an article (title, byline) is kept.
  • Output formats - Markdown, plain text or HTML.
  • Smart URL scoping - stays within the start URL path (on path-segment boundaries, www and apex are the same site). Include/exclude globs for fine control.
  • Sitemaps - found through robots.txt, /sitemap.xml and /sitemap_index.xml; sitemap indexes and .gz files are supported.
  • Canonical URL deduplication - the same page under two URLs is saved once, links are still followed.
  • Dynamic content - wait for JavaScript, scroll for lazy loading, open accordions and tabs, dismiss cookie banners (visible buttons only, never links).
  • Document files - optionally download linked PDF, DOC, DOCX, XLS, XLSX and CSV files into the key-value store (one dataset item per file with a file link; PDFs also get extracted text).
  • Metadata - title, description, author, keywords, language and canonical URL.

Use cases

  • RAG and vector search - build a knowledge base from docs sites, help centers and blogs for Pinecone, Qdrant, Weaviate and others. Turn on chunking and load the chunks straight into your index.
  • Custom GPTs and AI assistants - export crawled data as JSON and use it as knowledge files.
  • Content processing at scale - summarization, translation or classification with LLMs.
  • LangChain and LlamaIndex - use the Apify integrations to feed pages into your pipeline.

Input

Only Start URLs is required. Start URLs can be plain URLs or a remote list (requestsFromUrl).

SettingDescriptionDefault
Start URLsURLs to begin crawling from(required)
Crawler typeAdaptive, Chromium browser, or Cheerio (HTTP)Adaptive
Max pagesMaximum number of pages saved25
Max crawling depthLink depth from start URLs (0 = start URLs only, no sitemap expansion)20
Output formatMarkdown, plain text, or HTMLMarkdown
Exclude URLs (globs)Glob patterns for URLs to skip(none)
Include URLs (globs)Only crawl URLs matching these globs (same website unless cross-domain is enabled)(none)
Allow cross-domain include URLsLet include globs match other domainsNo
Remove elements (CSS)Additional CSS selectors to remove(none, defaults always applied)
Extract elements (CSS)Only keep content from these elements(none)
Remove cookie warningsDismiss and strip cookie bannersYes
Wait for dynamic contentTime to wait for JS rendering (ms), browser only1000
Scroll heightScroll to trigger lazy loading (px), browser only0
Expand clickablesOpen accordions/tabs, browser onlyNo
Save filesDownload linked PDF/DOC/XLS filesNo
Use sitemapsDiscover URLs from sitemapsYes
Chunk size / overlap / unitSplit each page into RAG chunks (characters or approx. tokens). 0 = off0

Output

Each saved page is one dataset item:

{
"url": "https://example.com/docs/getting-started",
"crawl": {
"loadedUrl": "https://example.com/docs/getting-started",
"loadedTime": "2026-01-15T10:30:00.000Z",
"depth": 1
},
"metadata": {
"canonicalUrl": "https://example.com/docs/getting-started",
"title": "Getting Started | Example Docs",
"description": "Learn how to get started with Example.",
"languageCode": "en"
},
"markdown": "# Getting Started\n\nWelcome to Example...",
"tokenCountEstimate": 412,
"contentHash": "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08",
"chunks": [
{ "index": 0, "text": "# Getting Started\n\nWelcome...", "tokenCountEstimate": 198 }
]
}
  • The content field is markdown, text or html, depending on the output format. Only that field is present.
  • metadata fields that a page does not have (author, keywords, ...) are left out.
  • chunks is only present when Chunk size is set.
  • Downloaded documents (Save files) have a file object with a key-value store link; PDFs also contain their extracted text.
  • The FAILED_URLS record of the run's key-value store lists URLs that failed, were blocked or had no content.

Changes to be aware of

  • Include URL globs now stay on the start URLs' website (subdomains such as docs.example.com for example.com are allowed). To match completely unrelated domains, enable Allow cross-domain include URLs.
  • Linked PDFs that could not be text-extracted are still saved (file link in the key-value store) with file.textExtracted: false.
  • A run that stops at its time limit with 0 pages is reported as Failed.

Pricing

$0.80 per 1,000 saved pages ($0.0008 per page) plus $0.0005 per GB at start (rising to $0.005 per GB on 2026-12-01, already scheduled on Apify) — all-in: platform usage and proxy are included, nothing else is billed. Paid Apify plans get up to 30% off. You only pay for pages that were actually saved to the dataset: empty and challenge pages are skipped. The crawl stops by itself when your maximum charge limit is reached.

Integration examples

Python (LangChain)

from langchain_core.documents import Document
from langchain_community.document_loaders import ApifyDatasetLoader # pip install langchain-community apify-client
from apify_client import ApifyClient
client = ApifyClient("YOUR_TOKEN")
run = client.actor("worshipful_knife/website-content-crawler").call(run_input={
"startUrls": [{"url": "https://docs.example.com/"}],
"maxCrawlPages": 50,
})
loader = ApifyDatasetLoader(
dataset_id=run["defaultDatasetId"],
dataset_mapping_function=lambda item: Document(
page_content=item.get("markdown") or item.get("text") or "",
metadata={"source": item["url"], "title": item["metadata"].get("title")},
),
)
docs = loader.load()

Node.js (Apify Client)

import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: 'YOUR_TOKEN' });
const run = await client.actor('worshipful_knife/website-content-crawler').call({
startUrls: [{ url: 'https://docs.example.com/' }],
maxCrawlPages: 50,
outputFormat: 'markdown',
chunkSize: 1000,
chunkOverlap: 100,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Troubleshooting

  • Missing content? Switch to a browser crawler type, which renders JavaScript.
  • Too much noise? Use "Remove elements" or "Extract elements" CSS selectors.
  • Crawler too slow? Use the Cheerio crawler for static sites and raise "Max concurrency".
  • Getting blocked? Use the browser crawler with residential proxies. Blocked pages are listed in FAILED_URLS.
  • Run ended as "Partial"? The run reached its time limit. Everything crawled so far is in the dataset; raise the run timeout or lower "Max pages".

Support

If you have any questions or feedback, open an issue on the Actor's page or contact us through Apify support.