Website Content to Markdown & llms.txt Crawler - AI & RAG Ready
Pricing
from $0.70 / 1,000 scraped results
High-performance, lightweight asynchronous crawler converting web pages into clean Markdown, computing token estimates, and generating standard /llms.txt manifests for AI and RAG ingestion.
List of URLs from which the crawl begins. Enter one or more URLs.
Maximum number of pages to crawl before completing the run.
Maximum distance in clicks from start URLs (0 = start URLs only, 1 = 1 click away, etc.).
Automatically synthesize standard /llms.txt and /llms-full.txt files and save them in the Key-Value Store.
CSS selectors of elements to strip from the HTML DOM before converting to Markdown.
When enabled, executes the Actor as a Model Context Protocol (MCP) server for direct integration with Cursor, Claude Desktop, and Windsurf.
Transport protocol for MCP tool execution. Use stdio for local CLI/IDE execution and sse for Apify Standby web integration.
Enable Playwright headless browser rendering for client-rendered SPAs (React, Vue, Angular). Note: If enabled, ensure your Actor run memory is set to at least 1,024 MB.
Optional CSS selector to wait for when renderJs is enabled before extracting content (e.g. 'article, .docs-content').
Scope Markdown extraction exclusively to matching element(s) (e.g. 'main, article, .markdown-body') before pruning boilerplate.
Optional JSON map of field names to CSS selectors to extract specific elements into structured output (e.g. {"author": ".author-name", "published": "time"}).
{}