Fast Website Content Crawler (AI/RAG Markdown & Sitemap)
Pricing
from $5.00 / 1,000 crawled web page (clean ai/rag markdown & chunks)s
Fast Website Content Crawler (AI/RAG Markdown & Sitemap)
Ultra-fast HTTP/2 website content & sitemap crawler for AI agents, MCP servers, and RAG pipelines. Converts any webpage or website into clean LLM-ready Markdown, semantic RAG chunks, JSON-LD schema, headings outline, and links with zero browser overhead.
Pricing
from $5.00 / 1,000 crawled web page (clean ai/rag markdown & chunks)s
Rating
0.0
(0)
Developer
H H
Maintained by CommunityActor stats
0
Bookmarked
1
Total users
0
Monthly active users
7 days ago
Last modified
Categories
Features
Share
What can Fast Website Content Crawler do?
Fast Website Content Crawler converts single URLs or entire documentation sites, blogs, help centers, and B2B websites into clean LLM-ready Markdown, semantic RAG chunks, heading outlines, JSON-LD structured data, and internal/external link graphs — using async HTTP/2 + sitemap.xml discovery inside 256 MB RAM with zero browser overhead:
| 📦 What you can extract | ⚙️ Features & integrations |
|---|---|
|
📄 Clean LLM Markdown stripped of navbars, footers, scripts & cookie banners ( |
🧭 Automatic |
Headless browser crawlers consume 2–4 GB of RAM and spend 80% of runtime booting Chromium just to read static HTML documentation, blogs, SaaS pricing pages, and news articles. Fast Website Content Crawler delivers clean Markdown + pre-split RAG chunks at $0.002–$0.0009 per page.
What data can I extract with Fast Website Content Crawler?
Designed from the ground up for AI engineering, RAG ingestion, competitive intelligence, and SEO technical audits, each crawled page emits a complete, self-contained document object:
-
Boilerplate-free Markdown & plain text: preserve structure without wasting LLM tokens. Converts headings, code blocks, tables, lists, and inline links into clean
markdownalongside strippedtext, pluswordCount,charCount,estimatedTokens, andreadingTimeMinutes. -
Pre-chunked RAG vectors: skip writing custom Markdown splitters. When
includeRagChunks: true(default), splitsmarkdownalong natural paragraph/section boundaries up tochunkSizeWords(default250) intoragChunks(ragChunksCount). -
Document outline & structured JSON-LD: semantic context for agents and SEO. Extracts the complete
headingsarray ({level, text}) so agents see page hierarchy immediately, plus parsedjsonLd(Schema.orgobjects embedded in<script type="application/ld+json">). -
Internal & outbound link topology: map site architecture or citation graphs. Lists deduplicated
internalLinks(internalLinksCount) andexternalLinks(externalLinksCount) found on every crawled URL.
Need more specialized workflows? Combine this Actor with these dedicated tools from our suite:
- 📧 Website Email, Phone & Tech Stack Finder: extract verified company emails, DNS/MX mail providers, phones & 50+ tech stack signals from the same domains
- 📝 YouTube Transcript Scraper: ingest YouTube video transcripts into the exact same RAG chunk pipeline
- 📰 Google News Scraper Fast: crawl breaking Google News coverage into decoded publisher Markdown + RAG chunks
- 💬 Reddit Scraper & B2B Buyer-Intent API: combine official docs Markdown with community troubleshooting threads on Reddit
How much will Fast Website Content Crawler cost?
Fast Website Content Crawler uses a transparent Pay-Per-Event (PPE) pricing model, where you pay a tiny $0.05 Actor Start event plus a per-item fee for each Crawled Web Page (Clean AI/RAG Markdown & Chunks). On the Apify Free plan ($5.00 monthly free credit included), it costs $0.015 per item ($15.00 per 1,000 results) — letting you extract up to 330 results every month for free.
Paid Apify plans automatically unlock volume discounts up to 67% off ($0.007 on Starter/Bronze, $0.006 on Scale/Silver, and $0.005 on Business/Gold, Platinum & Diamond) with zero residential proxy surcharges:
| Apify Plan / Tier | Price per extracted dataset item | Price per 1,000 results | Batch Depth & Unmasking | Approx. Monthly Capacity |
|---|---|---|---|---|
Free (FREE Trial Preview) | $0.015 | $15.00 | 5 items/run trial (items 1-2 100% unmasked, items 3-5 preview-masked) | ~330 trial results |
Starter (BRONZE — $29/mo) | $0.007 | $7.00 | Unlimited batch runs & 100% unmasked | ~4,135 results |
Scale (SILVER — $199/mo) | $0.006 | $6.00 | Unlimited batch runs & 100% unmasked | ~33,158 results |
Business (GOLD — $999/mo) | $0.005 | $5.00 | Unlimited batch runs & 100% unmasked | ~199,790 results |
Enterprise (PLATINUM / DIAMOND) | $0.005 / $0.005 | $5.00 | Unlimited batch runs & 100% unmasked | Custom high-volume SLA |
Free Trial vs. Paid Plan Unlock: On the Apify Free tier, runs execute in 5-item Trial Preview mode (items
1and2are returned100%unmasked so you can verify live email/data quality firsthand; items3to5mask emails, phones, and deep RAG/financial blobs). Upgrading to Apify Starter ($29/mo) or higher immediately unlocks unlimited batch extraction (1,000+items/run),100%unmasked emails and payloads, and up to 67% lower per-item pricing ($0.007on Starter down to$0.005on Business). Check the pricing tab for full details.
How do I use Fast Website Content Crawler?
Fast Website Content Crawler was designed to be effortless to start with — even if you have never configured proxies, cookies, or web scrapers before. Here is how to extract data in under a minute:
- Create a free Apify account using your email (includes
$5.00in monthly free usage credit). - Open Fast Website Content Crawler in Apify Console.
- Configure your target inputs (
startUrls,maxPagesPerSite,useSitemap) in the Input tab (use Bulk edit to paste lists at once). - Adjust your results limit or optional enrichment filters to match your workflow.
- Leave Proxy Configuration on its default zero-proxy setting (
useApifyProxy: false) for lowest cost. - Click Save & Start and watch structured records stream into your Dataset in real time.
- Open the Storage tab to inspect your results as a clean table or raw JSON.
- Download your dataset in JSON, CSV, Excel (
.xlsx), or XML, or connect it directly to Clay, n8n, Make, Zapier, Google Sheets, or MCP.
⬇️ Input
Pass one or more starting URLs or bare domains in startUrls. Set maxPagesPerSite: 1 to extract only those exact URLs, or maxPagesPerSite > 1 to crawl across the site using sitemap.xml + internal breadth-first link discovery.
Mode 1: Multi-Page Documentation / Website Sitemap Crawl (maxPagesPerSite > 1)
Automatically inspects /sitemap.xml, robots.txt sitemap directives, and internal links to convert up to maxPagesPerSite pages per domain into Markdown + RAG chunks.
| Input / Configuration | What it returns |
|---|---|
Bare domain ("crawlee.dev", "docs.stripe.com") | Upgraded to HTTPS; discovers subpages via sitemap + BFS |
Subdirectory URL ("https://apify.com/blog") | Crawls starting URL + discovered internal pages on the domain |
Input JSON example:
{"startUrls": ["https://crawlee.dev","https://apify.com"],"maxPagesPerSite": 10,"useSitemap": true,"includeRagChunks": true,"chunkSizeWords": 250}
Mode 2: Exact URL Reader (maxPagesPerSite: 1)
When an AI agent, n8n workflow, or Clay table already has exact article/pricing URLs and wants 1-to-1 clean Markdown extraction with zero link following.
| Input / Configuration | What it returns |
|---|---|
Specific page URLs (maxPagesPerSite: 1) | Exactly 1 clean Markdown + JSON-LD + RAG record per input URL |
Input JSON example:
{"startUrls": ["https://crawlee.dev/js","https://docs.apify.com/platform/integrations/mcp"],"maxPagesPerSite": 1,"useSitemap": false,"includeRagChunks": true,"chunkSizeWords": 300}
Complete Input Parameter Reference
| Parameter | Type | Default | Required | Description |
|---|---|---|---|---|
startUrls | array | ["https://crawlee.dev", "https://apify.com"] | Yes | List of URLs or bare domains to crawl (e.g. ['https://crawlee.dev', 'apify.com']). |
maxPagesPerSite | integer | 5 | No | Maximum number of pages to extract per starting website/domain (1 = only the start URL itself; >1 = also crawls internal links or sitemap pages). |
useSitemap | boolean | true | No | Automatically check /sitemap.xml and robots.txt alongside internal page links when maxPagesPerSite > 1. |
includeRagChunks | boolean | true | No | Split extracted Markdown into token-bounded chunks with metadata for direct vector database ingestion. |
chunkSizeWords | integer | 250 | No | Target word count per RAG chunk. |
proxyConfiguration | object | {"useApifyProxy": false} | No | Optional proxy settings. Works out-of-the-box with direct connections at zero proxy cost. |
⬆️ Output and setup
All extracted records stream incrementally into your run's Dataset under the Storage tab in Apify Console. Preview results as a clean interactive table, download in JSON, JSONL, CSV, Excel (.xlsx), XML, or RSS, or consume items directly via the Apify REST API, Python/Node.js SDKs, or MCP.
📦 Verified dataset output sample #1
{"url": "https://crawlee.dev","domain": "crawlee.dev","httpStatus": 200,"title": "Crawlee · Build reliable crawlers. Fast.","description": null,"author": null,"publishedAt": null,"canonicalUrl": "https://crawlee.dev","language": null,"ogImage": null,"wordCount": 151,"charCount": 1219,"estimatedTokens": 201,"readingTimeMinutes": 0.76,"headingsCount": 2,"headings": [{"level": "h1","text": "Build reliable web scrapers. Fast."},{"level": "h2","text": "Get started now!"}],"markdown": "# Crawlee · Build reliable crawlers. Fast.\n\n# Build reliable web scrapers. Fast.\n\n```\nimport { PlaywrightCrawler } from 'crawlee';// PlaywrightCrawler crawls the web using a headless browser controlled by the Playwright library.const crawler = new PlaywrightCrawler({ // Use the requestHandler to process each of the crawled pages. async requestHandle...","text": "Build reliable web scrapers. Fast.\n\nimport { PlaywrightCrawler } from 'crawlee';// PlaywrightCrawler crawls the web using a headless browser controlled by the Playwright library.const crawler = new PlaywrightCrawler({ // Use the requestHandler to process each of the crawled pages. async requestHandler({ request, page, enqueueLinks, pushData, log }) ...","ragChunksCount": 1,"ragChunks": [{"chunkIndex": 0,"wordCount": 164,"estimatedTokens": 218,"title": "Crawlee · Build reliable crawlers. Fast.","url": "https://crawlee.dev","text": "# Crawlee · Build reliable crawlers. Fast. # Build reliable web scrapers. Fast. ``` import { PlaywrightCrawler } from 'crawlee';// PlaywrightCrawler crawls the web using a headless browser controlled by the Playwright library.const crawler = new PlaywrightCrawler({ // Use the requestHandler to process each of the crawled pages. async requestHandler({ requ..."}],"internalLinksCount": 7,"internalLinks": ["https://crawlee.dev/js","https://crawlee.dev/python"],"externalLinksCount": 9,"externalLinks": ["https://github.com/apify/crawlee","https://github.com/apify/crawlee-python"],"jsonLd": [],"scrapedAt": "2026-11-02T08:15:08.606501+00:00"}
📦 Verified dataset output sample #2
{"url": "https://crawlee.dev/js","domain": "crawlee.dev","httpStatus": 200,"title": "Crawlee for JavaScript · Build reliable crawlers. Fast.","description": null,"author": null,"publishedAt": null,"canonicalUrl": "https://crawlee.dev/js","language": null,"ogImage": null,"wordCount": 436,"charCount": 2854,"estimatedTokens": 580,"readingTimeMinutes": 2.18,"headingsCount": 15,"headings": [{"level": "h1","text": "Build reliable web scrapers. Fast."},{"level": "h3","text": "Work with your favorite tools"}],"markdown": "# Crawlee for JavaScript · Build reliable crawlers. Fast.\n\n# Build reliable web scrapers. Fast.\n\nCrawlee is a web scraping library for JavaScript and Python. It handles blocking, crawling, proxies, and browsers for you. Get started Star Run on import { PlaywrightCrawler } from 'crawlee' ; const crawler = new PlaywrightCrawler ( { async requestHandler ( { ...","text": "Build reliable web scrapers. Fast.\n\nCrawlee is a web scraping library for JavaScript and Python. It handles blocking, crawling, proxies, and browsers for you. Get started Star Run on import { PlaywrightCrawler } from 'crawlee' ; const crawler = new PlaywrightCrawler ( { async requestHandler ( { request , page , enqueueLinks , pushData , log } ) { const ti...","ragChunksCount": 2,"ragChunks": [{"chunkIndex": 0,"wordCount": 250,"estimatedTokens": 332,"title": "Crawlee for JavaScript · Build reliable crawlers. Fast.","url": "https://crawlee.dev/js","text": "# Crawlee for JavaScript · Build reliable crawlers. Fast. # Build reliable web scrapers. Fast. Crawlee is a web scraping library for JavaScript and Python. It handles blocking, crawling, proxies, and browsers for you. Get started Star Run on import { PlaywrightCrawler } from 'crawlee' ; const crawler = new PlaywrightCrawler ( { async requestHandler ( { re..."},{"chunkIndex": 1,"wordCount": 218,"estimatedTokens": 290,"title": "Crawlee for JavaScript · Build reliable crawlers. Fast.","url": "https://crawlee.dev/js","text": "to a shared API. Or even let Adaptive crawler decide if JS rendering is needed. ``` const crawler = new AdaptivePlaywrightCrawler({ renderingTypeDetectionRatio: 0.1, async requestHandler({ querySelector, enqueueLinks }) { // The crawler detects if JS rendering is needed // to extract this data. If not, it will use HTTP // for follow-up requests to save ti..."}],"internalLinksCount": 16,"internalLinks": ["https://crawlee.dev/python","https://crawlee.dev"],"externalLinksCount": 10,"externalLinks": ["https://github.com/apify/crawlee","https://console.apify.com/actors/6i5QsHBMtm3hKph70?runConfig=eyJ1IjoiRWdQdHczb2VqNlRhRHQ1cW4iLCJ2IjoxfQ.eyJpbnB1dCI6IntcbiAgICBcImNvZGVcIjogXCJpbXBvcnQgeyBQbGF5d3JpZ2h0Q3Jhd2xlciB9IGZyb20gJ2NyYXdsZWUnO1xcblxcbi8vIENyYXdsZXIgc2V0dXAgZnJvbSB0aGUgcHJldmlvdXMgZXhhbXBsZS5cXG5jb25zdCBjcmF3bGVyID0gbmV3IFBsYXl3cmlnaHRDcmF3bGVyKHtcXG4gICAgLy8gVXNlIHRoZSByZXF1ZXN0S..."],"jsonLd": [],"scrapedAt": "2026-11-02T08:15:08.658505+00:00"}
Output Field Dictionary
| Output Field | Type | Description |
|---|---|---|
url / canonicalUrl / domain | string | Fetched page URL, canonical <link rel="canonical"> URL, and domain. |
httpStatus | integer | HTTP status code (e.g., 200). |
title / description / author / publishedAt | string | null | Extracted OpenGraph, meta, and article metadata. |
language / ogImage | string | null | HTML lang attribute and social preview image URL. |
wordCount / charCount / estimatedTokens | integer | Exact content volume and estimated LLM token count (~0.75 words/token). |
readingTimeMinutes | number | Estimated human reading time in minutes. |
headingsCount / headings | int / array | Ordered {level, text} heading outline. |
markdown | string | Clean GitHub-flavored Markdown representation of the page's main content. |
text | string | Plain-text body stripped of Markdown formatting. |
ragChunksCount / ragChunks | int / array | Token-bounded {chunkIndex, text, wordCount} chunks. |
internalLinksCount / internalLinks | int / array | Same-domain links discovered on the page. |
externalLinksCount / externalLinks | int / array | Outbound third-party URLs discovered on the page. |
jsonLd | array | Parsed application/ld+json Schema.org objects. |
scrapedAt | string | ISO-8601 UTC extraction timestamp. |
Notes, tips, and exceptions
- HTTP/2 HTML vs. client-only SPAs: 95%+ of documentation sites (Docusaurus, Mintlify, GitBook, ReadTheDocs, Next.js SSR, Astro), blogs, WordPress/Webflow/Shopify pages, and news sites render full HTML server-side and extract cleanly in milliseconds. Purely client-rendered CSR apps behind login walls will return their initial HTML shell.
useSitemap: truespeeds up deep discovery: Checking/sitemap.xml(and nested sitemap indexes) surfaces deep documentation and product pages immediately without waiting for multi-hop link traversal.- Canonical & trailing-slash deduplication: Normalized URLs prevent crawling the same page twice via
?utm_...or#anchorfragments. - Binary asset skipping:
.pdf,.zip,.png,.jpg,.mp4,.css, and.jslinks are automatically ignored during BFS link queuing. - Zero proxy needed: Direct connections are used by default (
useApifyProxy: false), keeping your run cost purely Pay-Per-Event.
What can you do with data from Fast Website Content Crawler?
Fast Website Content Crawler powers high-ROI automation, lead generation, and AI agent pipelines. Here are four proven step-by-step recipes:
🧠 Index technical docs or help centers into a customer-support RAG bot
Keep your AI support agent grounded on your latest product docs, changelogs, and API guides.
Recipe:
-
Point at your docs root. Run
startUrls: ["https://docs.yourproduct.com"]withmaxPagesPerSite: 100,useSitemap: true, andincludeRagChunks: true. -
Upsert
ragChunksinto Qdrant/Pinecone/pgvector. Storeurl,title, andheadings[0].textalongside each chunk's vector embedding. -
Re-sync weekly. Schedule an Apify cron + webhook to refresh changed pages automatically.
Total cost to convert 500 documentation pages on the Free tier: $1.01 ($0.46 on Gold/Diamond).
🕵️ Automated competitor pricing & changelog monitor
Know the same day a competitor changes pricing tiers, removes a free plan, or launches an enterprise feature.
Recipe:
-
List exact competitor pages. Pass
/pricing,/changelog, and/securityURLs withmaxPagesPerSite: 1. -
Diff
markdownsnapshots. Compare today'smarkdownagainst last week's dataset run inside n8n or Python. -
Summarize diffs with Claude/GPT. Send a concise pricing/feature change alert to Slack.
Run daily or weekly across your top 10 competitors.
🔍 Technical SEO, heading & Schema.org JSON-LD audit
Audit client websites for missing h1 tags, thin word counts, broken canonical tags, and Schema.org structured data coverage.
Recipe:
-
Crawl client domain. Set
maxPagesPerSite: 50anduseSitemap: true. -
Inspect
headings,wordCount, andjsonLd. Flag pages wherewordCount < 250, multipleh1tags exist, orjsonLdis empty.
Export directly to Excel or Google Sheets for client SEO deliverables.
🤝 Pre-call AI SDR account research brief
Before an AE joins a discovery call, turn the prospect's website into a 1-page executive brief.
Recipe:
-
Trigger via CRM webhook. When a lead books on Calendly/HubSpot, trigger
Fast Website Content Crawler(maxPagesPerSite: 5) + Website Email & Tech Stack Finder. -
Synthesize positioning & ICP. Feed the crawled
markdowninto an LLM to summarize what the company sells, who they sell to, and 3 tailored discovery questions.
Post the brief to Slack 15 minutes before the call.
Want to get other B2B, Social, Financial, or MCP Agent data?
Chain this tool with our other 128 MB / 256 MB zero-proxy Apify Scrapers and Live Streamable-HTTP MCP Servers:
🔌 Using Fast Website Content Crawler with the Apify API, SDKs & MCP
Automate Fast Website Content Crawler programmatically from Python, Node.js/TypeScript, cURL, workflow builders (n8n, Make, Zapier, Clay, Pipedream), or Model Context Protocol (MCP) AI clients.
Python (apify-client)
from apify_client import ApifyClientclient = ApifyClient("<YOUR_APIFY_API_TOKEN>")run = client.actor("church_of_loujess/fast-website-content-crawler").call(run_input={"startUrls": ["https://crawlee.dev","https://apify.com"],"maxPagesPerSite": 10,"useSitemap": True,"includeRagChunks": True,"chunkSizeWords": 250})for item in client.dataset(run["defaultDatasetId"]).iterate_items():print(item)
Node.js / TypeScript (apify-client)
import { ApifyClient } from 'apify-client';const client = new ApifyClient({ token: '<YOUR_APIFY_API_TOKEN>' });const run = await client.actor('church_of_loujess/fast-website-content-crawler').call({"startUrls": ["https://crawlee.dev","https://apify.com"],"maxPagesPerSite": 10,"useSitemap": true,"includeRagChunks": true,"chunkSizeWords": 250});const { items } = await client.dataset(run.defaultDatasetId).listItems();console.log(`Extracted ${items.length} items:`, items);
One-Line Synchronous REST API (cURL — ideal for n8n, Make & Clay HTTP Columns)
curl -X POST "https://api.apify.com/v2/acts/church_of_loujess~fast-website-content-crawler/run-sync-get-dataset-items?token=<YOUR_APIFY_API_TOKEN>" \-H "Content-Type: application/json" \-d '{"startUrls": ["https://crawlee.dev", "https://apify.com"], "maxPagesPerSite": 10, "useSitemap": true, "includeRagChunks": true, "chunkSizeWords": 250}'
Claude Desktop, Cursor, Windsurf & OpenAI MCP Configuration
{"mcpServers": {"apify-fast-website-content-crawler": {"url": "https://mcp.apify.com/?actors=church_of_loujess/fast-website-content-crawler","headers": {"Authorization": "Bearer <YOUR_APIFY_API_TOKEN>"}}}}
❓ Frequently Asked Questions
How do I scrape ONLY the exact URLs I pass without crawling subpages?
Set "maxPagesPerSite": 1. Each URL in startUrls will be fetched once and converted directly into clean Markdown, metadata, JSON-LD, and RAG chunks without queuing internal links.
Does it automatically parse sitemap.xml and nested sitemap indexes?
Yes. When maxPagesPerSite > 1 and useSitemap: true (default), the crawler checks /robots.txt for Sitemap: declarations as well as /sitemap.xml (including <sitemapindex> children) alongside live internal links.
How is estimatedTokens calculated?
It estimates BPE tokenization (cl100k_base / o200k_base / Claude tokenizer approximation) from character and word density so you know immediately whether markdown fits in a single prompt or should be retrieved via ragChunks.
Can I turn off ragChunks if I only want markdown?
Yes. Set "includeRagChunks": false for a leaner JSON payload.
Does it stay strictly on the starting domain?
Yes. Internal BFS and sitemap crawling never wander onto external third-party domains (outbound links are recorded in externalLinks without being crawled).
Why is it so much cheaper than browser-based crawlers?
Because it uses async HTTP/2 connection pooling + fast HTML-to-Markdown DOM pruning inside 256 MB RAM instead of spinning up heavy headless Chrome instances.
Is it legal to extract public data with Fast Website Content Crawler?
Fast Website Content Crawler only accesses publicly available web pages, feeds, and registries that require no private account login or authentication bypass. Keep in mind that extracted contact fields (such as public business emails, names, or usernames) can still qualify as personal data under frameworks like the GDPR or CCPA depending on jurisdiction. Always ensure you have a lawful basis (such as legitimate B2B interest) and consult Apify's guide on the legality of web scraping.
Can I integrate Fast Website Content Crawler with Clay, n8n, Make, Zapier, or Google Sheets?
Yes. Every run outputs clean, flat-friendly fields alongside nested detail objects and integrates natively with Clay, n8n, Make, Zapier, Slack, HubSpot, Airtable, Google Sheets, Airbyte, LangChain, and CrewAI via Apify Integrations. You can also attach webhooks to trigger downstream workflows automatically the instant a run succeeds.
Can I call Fast Website Content Crawler programmatically via the Apify API?
Yes. Use the official apify-client Python package or Node.js NPM package, or call the synchronous /run-sync-get-dataset-items REST endpoint directly. Full specifications are in the Apify API v2 reference.
Fast Website Content Crawler not working or need a custom feature?
We actively maintain and monitor all 26 Actors and MCP Servers in our portfolio. If you encounter an edge case, unexpected output, or want to request a new field, please open a ticket on the Actor's Issues tab with your Run ID so we can resolve it rapidly.