Website to Markdown for AI & RAG – Fast Content Crawler
Pricing
from $0.56 / 1,000 page scrapeds
Website to Markdown for AI & RAG – Fast Content Crawler
Crawl any website and get clean, LLM-ready Markdown, text or HTML for RAG, chatbots and AI agents. Optional chunking, token estimates and content hashes. Fast HTTP or browser crawling, $0.80 per 1,000 pages all-in.
Pricing
from $0.56 / 1,000 page scrapeds
Rating
0.0
(0)
Developer
kata Kuri
Maintained by CommunityActor stats
1
Bookmarked
43
Total users
1
Monthly active users
a day ago
Last modified
Categories
Features
Share
Website to Markdown for AI & RAG - Fast Content Crawler
Crawl any website and get clean, LLM-ready Markdown (or plain text / HTML) for RAG pipelines, chatbots, AI agents and vector databases. Built to be fast and cheap: $0.80 per 1,000 pages, all-in, with a plain HTTP mode that is roughly 10x faster than a browser for static sites and docs.
Why use it
- LLM-ready output - Markdown with real tables, code blocks that keep their line breaks, and absolute links and image URLs.
- RAG features built in - optional chunking (by characters or approximate tokens, with overlap), a token count estimate and a content hash (SHA-256) on every page for change detection and deduplication.
- Cheaper and faster by design - HTTP (Cheerio) mode for static sites, a lean Chromium mode for JavaScript sites that skips images, fonts and media.
- You only pay for real content - empty pages and anti-bot / captcha challenge pages are never saved, so never charged. They are listed in the
FAILED_URLSrecord. - Honest run status - a crash ends the run as Failed; a run that hits its time budget stops gracefully, keeps everything crawled and reports
Partial: stopped at time budget, N pages.
Key features
- Crawler engines - Adaptive uses the Chromium browser and falls back to HTTP if the browser gets nothing (blocked). Or pick Chromium or Cheerio (HTTP) yourself.
- Clean content extraction - removes navigation, the site header, footer, cookie banners, ads, modals and similar noise, and picks the main content area. A
<header>inside an article (title, byline) is kept. - Output formats - Markdown, plain text or HTML.
- Smart URL scoping - stays within the start URL path (on path-segment boundaries,
wwwand apex are the same site). Include/exclude globs for fine control. - Sitemaps - found through
robots.txt,/sitemap.xmland/sitemap_index.xml; sitemap indexes and.gzfiles are supported. - Canonical URL deduplication - the same page under two URLs is saved once, links are still followed.
- Dynamic content - wait for JavaScript, scroll for lazy loading, open accordions and tabs, dismiss cookie banners (visible buttons only, never links).
- Document files - optionally download linked PDF, DOC, DOCX, XLS, XLSX and CSV files into the key-value store (one dataset item per file with a file link; PDFs also get extracted text).
- Metadata - title, description, author, keywords, language and canonical URL.
Use cases
- RAG and vector search - build a knowledge base from docs sites, help centers and blogs for Pinecone, Qdrant, Weaviate and others. Turn on chunking and load the
chunksstraight into your index. - Custom GPTs and AI assistants - export crawled data as JSON and use it as knowledge files.
- Content processing at scale - summarization, translation or classification with LLMs.
- LangChain and LlamaIndex - use the Apify integrations to feed pages into your pipeline.
Input
Only Start URLs is required. Start URLs can be plain URLs or a remote list (requestsFromUrl).
| Setting | Description | Default |
|---|---|---|
| Start URLs | URLs to begin crawling from | (required) |
| Crawler type | Adaptive, Chromium browser, or Cheerio (HTTP) | Adaptive |
| Max pages | Maximum number of pages saved | 25 |
| Max crawling depth | Link depth from start URLs (0 = start URLs only, no sitemap expansion) | 20 |
| Output format | Markdown, plain text, or HTML | Markdown |
| Exclude URLs (globs) | Glob patterns for URLs to skip | (none) |
| Include URLs (globs) | Only crawl URLs matching these globs (same website unless cross-domain is enabled) | (none) |
| Allow cross-domain include URLs | Let include globs match other domains | No |
| Remove elements (CSS) | Additional CSS selectors to remove | (none, defaults always applied) |
| Extract elements (CSS) | Only keep content from these elements | (none) |
| Remove cookie warnings | Dismiss and strip cookie banners | Yes |
| Wait for dynamic content | Time to wait for JS rendering (ms), browser only | 1000 |
| Scroll height | Scroll to trigger lazy loading (px), browser only | 0 |
| Expand clickables | Open accordions/tabs, browser only | No |
| Save files | Download linked PDF/DOC/XLS files | No |
| Use sitemaps | Discover URLs from sitemaps | Yes |
| Chunk size / overlap / unit | Split each page into RAG chunks (characters or approx. tokens). 0 = off | 0 |
Output
Each saved page is one dataset item:
{"url": "https://example.com/docs/getting-started","crawl": {"loadedUrl": "https://example.com/docs/getting-started","loadedTime": "2026-01-15T10:30:00.000Z","depth": 1},"metadata": {"canonicalUrl": "https://example.com/docs/getting-started","title": "Getting Started | Example Docs","description": "Learn how to get started with Example.","languageCode": "en"},"markdown": "# Getting Started\n\nWelcome to Example...","tokenCountEstimate": 412,"contentHash": "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08","chunks": [{ "index": 0, "text": "# Getting Started\n\nWelcome...", "tokenCountEstimate": 198 }]}
- The content field is
markdown,textorhtml, depending on the output format. Only that field is present. metadatafields that a page does not have (author, keywords, ...) are left out.chunksis only present when Chunk size is set.- Downloaded documents (Save files) have a
fileobject with a key-value store link; PDFs also contain their extracted text. - The
FAILED_URLSrecord of the run's key-value store lists URLs that failed, were blocked or had no content.
Changes to be aware of
- Include URL globs now stay on the start URLs' website (subdomains such as
docs.example.comforexample.comare allowed). To match completely unrelated domains, enable Allow cross-domain include URLs. - Linked PDFs that could not be text-extracted are still saved (file link in the key-value store) with
file.textExtracted: false. - A run that stops at its time limit with 0 pages is reported as Failed.
Pricing
$0.80 per 1,000 saved pages ($0.0008 per page) plus $0.0005 per GB at start (rising to $0.005 per GB on 2026-12-01, already scheduled on Apify) — all-in: platform usage and proxy are included, nothing else is billed. Paid Apify plans get up to 30% off. You only pay for pages that were actually saved to the dataset: empty and challenge pages are skipped. The crawl stops by itself when your maximum charge limit is reached.
Integration examples
Python (LangChain)
from langchain_core.documents import Documentfrom langchain_community.document_loaders import ApifyDatasetLoader # pip install langchain-community apify-clientfrom apify_client import ApifyClientclient = ApifyClient("YOUR_TOKEN")run = client.actor("worshipful_knife/website-content-crawler").call(run_input={"startUrls": [{"url": "https://docs.example.com/"}],"maxCrawlPages": 50,})loader = ApifyDatasetLoader(dataset_id=run["defaultDatasetId"],dataset_mapping_function=lambda item: Document(page_content=item.get("markdown") or item.get("text") or "",metadata={"source": item["url"], "title": item["metadata"].get("title")},),)docs = loader.load()
Node.js (Apify Client)
import { ApifyClient } from 'apify-client';const client = new ApifyClient({ token: 'YOUR_TOKEN' });const run = await client.actor('worshipful_knife/website-content-crawler').call({startUrls: [{ url: 'https://docs.example.com/' }],maxCrawlPages: 50,outputFormat: 'markdown',chunkSize: 1000,chunkOverlap: 100,});const { items } = await client.dataset(run.defaultDatasetId).listItems();console.log(items);
Troubleshooting
- Missing content? Switch to a browser crawler type, which renders JavaScript.
- Too much noise? Use "Remove elements" or "Extract elements" CSS selectors.
- Crawler too slow? Use the Cheerio crawler for static sites and raise "Max concurrency".
- Getting blocked? Use the browser crawler with residential proxies. Blocked pages are listed in
FAILED_URLS. - Run ended as "Partial"? The run reached its time limit. Everything crawled so far is in the dataset; raise the run timeout or lower "Max pages".
Support
If you have any questions or feedback, open an issue on the Actor's page or contact us through Apify support.