🌐 Website Content Crawler — Web Scraping & AI Content Extraction Actor

Website Content Crawler is a powerful Apify Actor for website crawling, web scraping, content extraction, and AI data preparation. It crawls any website starting from your seed URLs, follows internal links, extracts clean plain text, Markdown, and rich metadata (title, description, canonical URL, Open Graph, JSON-LD, language, headers), and writes structured page data to an Apify Dataset in real time — perfect for RAG pipelines, LLM knowledge bases, vector database ingestion, semantic search, and SEO content audits.

💡 Need a custom version, private integration, or a tailored crawler? Email hello.dataminds@gmail.com.

It is a focused alternative to the official apify/website-content-crawler: a tighter feature surface, predictable cost, and AI-first output (text + Markdown + metadata) ready for LangChain, LlamaIndex, OpenAI, Pinecone, Qdrant, and Weaviate pipelines.

---

📑 Table of contents

• What is Website Content Crawler?
• Main features
• Who is this Actor for?
• Built for generative AI, LLMs, and RAG
• What the crawler does
• Crawling configuration
• HTML cleaning & content extraction
• Output format (Dataset)
• Example output (JSON)
• Quick start
• Input parameters reference
• Integrations: LangChain, LlamaIndex, OpenAI, Pinecone
• Pricing & how to control cost
• Frequently asked questions (FAQ)
• Troubleshooting
• Help, support & custom builds
• Is web scraping legal?
• SEO keywords targeted

---

🌐 What is Website Content Crawler?

Website Content Crawler is an Apify-hosted web scraping API and website crawler that turns any public website into clean, structured, AI-ready data. Unlike generic HTTP scrapers, this Actor:

• Crawls an entire site from one or more start URLs, staying on the same registrable domain.
• Renders JavaScript automatically when a page needs it, so single-page apps still yield real content.
• Cleans HTML by stripping navigation, footers, scripts, cookie banners, modals, and other noise.
• Extracts content as readable plain text and optional Markdown for token-efficient LLM ingestion.
• Captures metadata (title, description, canonical, Open Graph, JSON-LD, headers, language).
• Saves results as structured rows in an Apify Dataset, exportable to JSON, CSV, Excel, or via the Apify API.

If you need a website-to-dataset pipeline for AI agents, chatbot grounding, GPT knowledge files, RAG retrieval, or content analytics, this is the Actor.

---

✨ Main features

• 🌱 Multi-URL crawl seeds — start from one URL or hundreds; the crawler stays on the same domain by default.
• 🎯 Glob-based URL filtering — includeUrlGlobs / excludeUrlGlobs for surgical scope (/blog/*, /docs/**, skip /admin/**). Path-only patterns work as written; * stops at /, ** crosses folders.
• 🗺️ Discovery sources — seed the queue from sitemap.xml (useSitemaps, follows sitemap indexes) and llms.txt (useLlmsTxt) to reach pages that aren't linked, and honour robots.txt (respectRobotsTxtFile).
• 🛡️ Apify Proxy support — datacenter or residential groups, country targeting, or your own proxy URLs, with automatic fallback.
• 🍪 Initial cookies — initialCookies sent on every request and loaded into the browser for session or light-login access.
• 🖱️ Rendered-page interaction — maxScrollHeightPixels auto-scroll for lazy/infinite-scroll content, clickElementsCssSelector to expand accordions / "load more", and blockMedia to skip images/fonts/video for faster, cheaper renders.
• 🚫 Ad/tracker blocking — blockAds aborts known ad and tracker domains on rendered pages, on top of blockMedia.
• 📱 Mobile emulation — mobileView renders with a real mobile device profile (viewport, UA, touch) instead of desktop.
• 🕵️ PII redaction (best-effort) — redactPii masks emails, phone numbers, card-like numbers, and SSNs in text/markdown.
• 📐 Screenshot mode — screenshotFullPage (default on) toggles full-page vs viewport-only screenshots.
• 📕 PDF text extraction — extractPdfText opens discovered PDF links and extracts real text into text, instead of only listing them as file links.
• 📂 Depth & page limits — maxCrawlDepth, maxCrawlPages, maxResults to keep cost and runtime predictable.
• ⚡ Async concurrency — tune maxConcurrency from 1 to 50 parallel requests.
• 🖥️ Three rendering modes — auto (render only when a page needs it), always, or never for the cheapest possible run.
• 🪄 Smart retry — pages that come back incomplete are automatically retried for a fuller result.
• 🧼 HTML noise removal — strip nav, footer, modals, ads, and cookie banners via CSS selectors before extraction.
• 📝 Markdown export — token-efficient Markdown alongside plain text, ideal for LLM context windows.
• 🏷️ Rich metadata extraction — title, description, canonical URL, Open Graph, Twitter Card, JSON-LD, author, keywords, publish/modified dates, language code.
• 🧲 Keep-only extraction — keepElementsCssSelector to keep just an article body (article, main) and discard everything else before extraction.
• 🧾 Skipped-URL audit — storeSkippedUrls writes every discovered-but-not-crawled URL (with a reason) to skipped-urls.json.
• 📸 HTML snapshots & full-page screenshots stored in the run's key-value store, surfaced as direct clickable URLs on every row (original HTML, noise-removed HTML, readable HTML, PNG).
• ⏱️ Politeness controls — rateLimitDelaySecs, retry counts, request timeouts, and dynamic-content waits.
• 🎯 Wait-for-selector — block extraction until a CSS selector renders (great for React/Vue/Angular SPAs).
• 🛡️ Custom HTTP headers — User-Agent, sec-ch-ua, DNT, plus arbitrary cookies and auth headers via JSON.
• 🌐 Canonical & fragment handling — dedupe by canonical link or treat #anchors as distinct pages.
• 📦 Real-time Dataset writes — every page lands in your Dataset the moment it is ready, not at the end of the run.
• 🔧 Verbose diagnostics — link counts, status codes, content lengths, and snapshot links on every row.

---

👥 Who is this Actor for?

This Apify Actor is built for:

• 🤖 AI engineers building RAG, retrieval-augmented generation, or LLM grounding pipelines.
• 🧠 Data scientists assembling training corpora, fine-tuning datasets, or embedding indexes.
• 🛠️ Backend developers integrating website-to-database ETL flows or knowledge sync jobs.
• 📈 SEO professionals auditing titles, meta descriptions, canonical URLs, Open Graph, schema.org markup, and internal linking at scale.
• 📚 Documentation teams building searchable docs portals or AI-powered help bots.
• 🛒 Content strategists & competitive analysts monitoring competitor blogs, landing pages, and changelogs.
• 🔬 Researchers collecting public web text data for NLP, classification, or sentiment models.

If you've ever needed to "turn this whole website into a clean JSON file my LLM can read," this Actor is the answer.

---

🤖 Built for generative AI, LLMs, and RAG

This Actor is designed from the ground up for AI workflows:

• RAG pipelines — crawl docs, knowledge bases, FAQs, and blogs into chunkable Markdown.
• LLM knowledge bases — feed text + metadata into vector stores for grounded chatbot answers.
• Semantic search engines — index page content with embeddings for relevance-ranked retrieval.
• Vector database ingestion — drop directly into Pinecone, Qdrant, Weaviate, Chroma, Milvus, or pgvector.
• Custom GPT knowledge files — export Dataset JSON and upload to ChatGPT custom GPTs, Claude Projects, or Anthropic Files API.
• AI agent training & grounding — build autonomous agents with up-to-date web context.
• Content operations — summarization, translation, rewrite-assist, content moderation pipelines.
• Competitive intelligence — periodic crawls feeding analytics dashboards.

Practical AI use cases

1. Customer support chatbot grounding — crawl product docs, help center, and FAQ pages; embed text + metadata for retrieval.
2. Documentation intelligence & semantic search — turn product docs and changelogs into a searchable, embedding-indexed corpus.
3. SEO and content audit automation — analyze titles, meta descriptions, canonical links, and structured data sitewide.
4. Custom GPT / Claude Project knowledge files — generate a dataset JSON and upload to your AI assistant of choice.
5. Internal LLM knowledge base — sync intranet, wiki, or Notion-published content into a private vector store.
6. Lead enrichment & company intelligence — crawl prospect websites for descriptions, services, and contact pages.
7. Compliance and content monitoring — track changes to legal pages, terms of service, or pricing pages.

---

⚙️ What the crawler does

• Crawls from your seed URLs, follows internal links within domain scope, applies your includeUrlGlobs / excludeUrlGlobs filters, and respects maxCrawlDepth, maxCrawlPages, and maxResults.
• Handles JavaScript sites according to your renderingMode — auto renders only the pages that need it, always renders everything, never is the fastest and cheapest.
• Cleans each page with your CSS selectors (nav, footer, cookie banners, modals) before extracting text and Markdown.
• Writes every row to the Apify Dataset the moment it is ready — url, flat metadata columns, crawl info, metadata, text, markdown, snapshot links, and screenshotUrl. Downstream systems can consume rows while the crawl is still running.
• Keeps going when a page fails — bad URLs, non-HTML files, blocked pages, and oversized pages are handled per page, never by aborting the run.

---

🧭 Crawling configuration

• Start URLs are the only required input. Add as many as you like.
• By default, the crawler stays on the same registrable domain as your seeds — no accidental crawling of facebook.com from your blog's share buttons.
• includeUrlGlobs — keep only matching paths (e.g., /blog/*, **/blog/**).
• excludeUrlGlobs — skip paths (e.g., /admin/**, **/login, **/*.pdf).
• Filters apply to discovered links only — your start URLs are always crawled, so a narrow include pattern can't accidentally produce an empty run.
• useSitemaps — also seed the queue from robots.txt-advertised sitemaps and /sitemap.xml (following sitemap indexes). Reaches orphan pages that aren't linked from your seeds. Your include/exclude filters still apply.
• useLlmsTxt — also seed the queue from URLs listed in each start domain's /llms.txt.
• respectRobotsTxtFile — skip discovered URLs disallowed by the site's robots.txt (your explicit start URLs are always crawled).
• keepUrlFragments — treat #anchors as distinct URLs (off by default).
• ignoreCanonicalUrl — disable canonical-based dedup if you want every URL crawled distinctly.
• maxCrawlDepth — 0 = only seeds; 1 = seeds + their direct links; etc.
• maxCrawlPages — total unique pages opened (0 = unlimited).
• maxResults — total successful records saved.

🧪 Test glob patterns with the DigitalOcean glob tester.

---

🧼 HTML cleaning & content extraction

The Actor produces clean, readable text suitable for AI consumption:

• keepElementsCssSelector — keep only the elements matching this selector (e.g. article, main) and discard everything else. Applied before the remove list below; empty keeps the whole page.
• removeElementsCssSelector — comma-separated CSS selectors to strip before extraction (defaults remove nav, footer, scripts, cookie banners, modals, alerts).
• readableTextCharThreshold — how much text a page must have before it counts as complete.
• waitForSelector — pause until a key CSS selector appears (perfect for SPAs that mount content lazily).
• dynamicContentWaitSecs — extra settle time for late-arriving dynamic content.
• saveMarkdown — include a Markdown version of the page alongside plain text.

The result: token-efficient, noise-free content ready for embedding, indexing, or LLM context.

Rendered-page interaction

These apply only to rendered pages (renderingMode: auto when a page is rendered, or always):

• maxScrollHeightPixels — auto-scroll down up to this many pixels to trigger lazy-loaded / infinite-scroll content (0 = off).
• clickElementsCssSelector — best-effort click on matching elements (accordions, "show more", expanders) before extracting; e.g. [aria-expanded="false"].
• blockMedia — skip downloading images, fonts, and video while rendering for faster, cheaper runs (text and links unaffected).

---

📤 Output format (Dataset)

Results land in the default Apify Dataset. You can:

• Browse rows in the Apify Console with multiple prebuilt views (Quick View, Crawl Details, Metadata, Content, Storage & Debug).
• Export to JSON, CSV, Excel, XML, HTML, or RSS.
• Pull data programmatically via the Apify API.
• Stream into AI, ETL, or analytics pipelines while the run is still active.

Each row includes:

Field	Description
orderNo	1-based order in which the row was saved during the run.
url	The page's own URL — its canonical URL when one exists (and ignoreCanonicalUrl is off), otherwise the loaded URL.
loadedUrl	The URL actually fetched, after redirects.
referrerUrl	The page that linked to this one (referrerUrl input for start URLs).
depth	Link distance from the nearest start URL (0 for seeds).
httpStatusCode	HTTP status of the response.
title, description, canonicalUrl, author, keywords, datePublished, dateModified, languageCode	Flat metadata aliases so Dataset table views and CSV exports work without nesting.
crawl	Nested loadedUrl, loadedTime, referrerUrl, depth, httpStatusCode.
metadata	canonicalUrl, title, description, author, keywords, datePublished, dateModified, languageCode, openGraph (Open Graph + article + Twitter Card tags), jsonLd, headers.
text	Clean, readable plain text with paragraph structure preserved.
markdown	Real Markdown — headings, links, bold/italic, nested lists, blockquotes, GFM tables, fenced code (when saveMarkdown is on).
html	Cleaned readable HTML inline (when saveHtml is on).
htmlUrl	Direct URL to the readable HTML snapshot in the key-value store.
screenshotUrl	Direct URL to the full-page PNG (when saveScreenshots is on).
files	Non-HTML links found on the page (PDF, DOCX, XLSX, ZIP, …), or null.
debug	Per-row diagnostics — page/file link counts, response content type, snapshot URLs, and content byte lengths.
debugTotalLinks, debugStatusCode	Flat debug aliases for table views.

ℹ️ Rows over Apify's 9 MB item limit are trimmed automatically (html → markdown → text) and flagged with debug.truncated: true, so a single huge page never fails the run.

🧾 With storeSkippedUrls: true, discovered-but-not-crawled URLs are written to skipped-urls.json in the run's key-value store, each with a reason (off-domain, excluded, not-included, robots-disallow, non-http).

---

🧪 Example output (JSON)

Real output from a run against https://docs.apify.com/ (text and Markdown abbreviated):

{
  "orderNo": 4,
  "url": "https://docs.apify.com/proxy",
  "loadedUrl": "https://docs.apify.com/proxy",
  "loadedTime": "2026-07-23T14:31:52.015Z",
  "referrerUrl": "https://docs.apify.com/academy/scraping-basics-javascript",
  "depth": 1,
  "httpStatusCode": 200,
  "title": "Proxy | Platform | Apify Documentation",
  "description": "Learn to anonymously access websites in scraping/automation jobs.",
  "canonicalUrl": "https://docs.apify.com/proxy",
  "author": null,
  "keywords": null,
  "datePublished": null,
  "dateModified": null,
  "languageCode": "en",
  "crawl": {
    "loadedUrl": "https://docs.apify.com/proxy",
    "loadedTime": "2026-07-23T14:31:52.015Z",
    "referrerUrl": "https://docs.apify.com/academy/scraping-basics-javascript",
    "depth": 1,
    "httpStatusCode": 200
  },
  "metadata": {
    "canonicalUrl": "https://docs.apify.com/proxy",
    "title": "Proxy | Platform | Apify Documentation",
    "description": "Learn to anonymously access websites in scraping/automation jobs.",
    "author": null,
    "keywords": null,
    "datePublished": null,
    "dateModified": null,
    "languageCode": "en",
    "openGraph": [
      { "property": "og:url", "content": "https://docs.apify.com/proxy" },
      { "property": "og:locale", "content": "en" }
    ],
    "jsonLd": null,
    "headers": { "content-type": "text/html; charset=utf-8", "…": "…" }
  },
  "text": "Proxy\n\nApify Proxy lets you rotate IP addresses when scraping to avoid geographic blocking. …",
  "markdown": "# Proxy\n\n[Apify Proxy](https://apify.com/proxy) lets you rotate IP addresses when scraping …",
  "htmlUrl": "https://api.apify.com/v2/key-value-stores/<storeId>/records/kv__bfc3652cf15f843d170b_readable.html",
  "screenshotUrl": null,
  "files": null,
  "debug": {
    "totalLinks": 59,
    "pageLinks": 59,
    "fileLinks": 0,
    "response": { "statusCode": 200, "contentType": "text/html; charset=utf-8" },
    "snapshots": {
      "originalHtmlUrl": "https://api.apify.com/v2/key-value-stores/<storeId>/records/kv__bfc3652cf15f843d170b_original.html",
      "removedElementsHtmlUrl": "https://api.apify.com/v2/key-value-stores/<storeId>/records/kv__bfc3652cf15f843d170b_removed.html",
      "readableTextHtmlUrl": "https://api.apify.com/v2/key-value-stores/<storeId>/records/kv__bfc3652cf15f843d170b_readable.html"
    },
    "lengths": {
      "originalHtml": 75279,
      "readableHtml": 43168,
      "readableText": 10763,
      "markdown": 19203
    }
  }
}

---

🚀 Quick start

Run in Apify Console

1. Open the Actor in Apify Console.
2. Paste one or more start URLs (e.g., https://docs.apify.com/).
3. (Optional) Set maxResults, maxCrawlDepth, maxCrawlPages, and URL filters.
4. Click Start and watch rows stream into your Dataset.
5. Export as JSON / CSV or pull via the Apify API.

Run via API

curl -X POST "https://api.apify.com/v2/acts/<your-actor-id>/runs?token=<APIFY_TOKEN>"   -H "Content-Type: application/json"   -d '{
        "startUrls": ["https://docs.apify.com/"],
        "maxResults": 25,
        "maxCrawlDepth": 2,
        "renderingMode": "auto",
        "saveMarkdown": true
      }'

Then read the rows from the run's dataset:

$curl "https://api.apify.com/v2/actor-runs/<runId>/dataset/items?format=json&token=<APIFY_TOKEN>"

---

🛠️ Input parameters reference

Parameter	Type	Default	Description
startUrls	array	—	Required. Seed URLs. Always crawled — URL filters apply to discovered links only.
referrerUrl	string	""	Value recorded as referrerUrl on rows produced from the start URLs.
blockAds	bool	false	Abort known ad/tracker domains on rendered pages.
mobileView	bool	false	Render with a mobile device profile (viewport, UA, touch) instead of desktop.
redactPii	bool	false	Mask emails, phone numbers, card-like numbers, and SSNs in text/markdown (best-effort).
screenshotFullPage	bool	true	Full-page vs viewport-only screenshots.
extractPdfText	bool	false	Open discovered PDF links and extract real text into text instead of only listing them as files.
proxyConfiguration	object	{}	Apify Proxy (datacenter/residential/country) or your own proxy URLs.
includeUrlGlobs	array	[]	Keep only discovered URLs matching these globs (/blog/*, /docs/**).
excludeUrlGlobs	array	[]	Skip discovered URLs matching these globs (/admin/**, **/*.pdf).
useSitemaps	bool	false	Seed the queue from robots.txt-advertised sitemaps and /sitemap.xml (follows sitemap indexes).
useLlmsTxt	bool	false	Seed the queue from URLs listed in each start domain's /llms.txt.
respectRobotsTxtFile	bool	false	Skip discovered URLs disallowed by robots.txt (seeds always crawled).
keepUrlFragments	bool	false	Treat #anchors as distinct URLs.
ignoreCanonicalUrl	bool	false	Use the loaded URL as url instead of the canonical one.
maxCrawlDepth	int	2	Link-following depth from seeds (0 = seeds only).
maxCrawlPages	int	0	Hard cap on unique pages opened (0 = unlimited).
maxResults	int	10	Cap on successful saved rows.
maxConcurrency	int	10	Parallel page workers (1–50).
rateLimitDelaySecs	string	0.1	Throttle before each fetch; accepts decimals.
maxRequestRetries	int	3	Total attempts per URL (backoff; honours Retry-After on 429).
requestTimeoutSecs	int	60	Request / navigation timeout.
dynamicContentWaitSecs	int	3	Real settle time in seconds after a rendered page loads.
waitForSelector	string	—	Wait for this CSS selector; a missing selector triggers a more thorough retry.
renderingMode	enum	auto	auto (render only when a page needs it) · always · never (cheapest).
retryFailedWithRendering	bool	true	Give a failed page one more, more thorough attempt.
blockMedia	bool	false	Skip images/fonts/video on rendered pages for faster, cheaper runs.
maxScrollHeightPixels	int	0	Auto-scroll rendered pages up to N pixels for lazy/infinite-scroll content (0 = off).
clickElementsCssSelector	string	—	Best-effort click on matching elements (expanders / "load more") before extracting rendered pages.
ignoreHttpsErrors	bool	false	Accept bad certificates (dev/staging only).
keepElementsCssSelector	string	—	Keep only elements matching this selector (e.g. article, main); empty keeps the whole page.
removeElementsCssSelector	string	(sensible default)	CSS selectors stripped before extraction.
readableTextCharThreshold	int	100	Below this many characters a page counts as "thin" and is retried.
kvDir	string	kv	Storage prefix for snapshots & screenshots (empty = skip storage writes).
saveScreenshots	bool	false	Full-page PNG per row. Requires rendering every page.
saveMarkdown	bool	true	Include Markdown in each row.
saveHtml	bool	true	Include cleaned HTML inline; turn off for much smaller rows.
headerAcceptLanguage	string	en-US,en;q=0.9	Accept-Language header.
customHttpHeaders	object	{}	Extra HTTP headers (cookies, auth, lang) — these override everything else.
initialCookies	array	[]	Cookies sent on every request and loaded into the browser: [{"name":"...","value":"...","domain":"...","path":"/"}].
debugLog	bool	true	Drops the run log to DEBUG level.
storeSkippedUrls	bool	false	Write discovered-but-not-crawled URLs (with reasons) to skipped-urls.json in the key-value store.

Plus browser-identity headers (headerUserAgent, headerSecChUa, headerSecChUaMobile, headerSecChUaPlatform, headerDnt, headerUpgradeInsecureRequests). The defaults give the best success rate on most sites; set a custom headerUserAgent or headerSecChUa and your values are used verbatim instead.

---

🔌 Integrations: LangChain, LlamaIndex, OpenAI, Pinecone

This Actor's Dataset output plugs directly into the modern AI stack:

• 🦜 LangChain — use ApifyDatasetLoader or load JSON into Document objects with text + metadata.
• 🦙 LlamaIndex — feed Markdown rows into MarkdownReader or build a VectorStoreIndex.
• 🧠 OpenAI Embeddings & Assistants — chunk text / markdown, embed with text-embedding-3, and upload as OpenAI File for Assistants API or Custom GPT knowledge.
• 🤖 Anthropic Claude — upload Dataset JSON to Claude Projects, the Files API, or use as RAG context.
• 📍 Pinecone, Qdrant, Weaviate, Chroma, Milvus, pgvector — embed and upsert with metadata filters.
• 📊 n8n, Make.com, Zapier — wire crawl results into automations and notifications.
• 📦 Snowflake, BigQuery, PostgreSQL, MongoDB, S3 — load Dataset JSON/CSV into your warehouse or lake.
• 🧰 Apify API & SDKs — orchestrate runs from your own services in any language.

See Apify Publishing & integration docs for platform-level guidance.

---

💸 Pricing & how to control cost

Billable event

This Actor charges one pay-per-event event:

Event	When it fires	What you get
row_result	Once per page record written to the Dataset	A complete page row — text, Markdown, metadata, snapshot links

Pages that fail, are skipped (non-HTML, 404, filtered by globs), or are dropped after the maxResults cap are not charged. If the run hits its charging limit (ACTOR_MAX_TOTAL_CHARGE_USD), the crawl stops cleanly and everything collected so far stays in the Dataset.

⚙️ Publishing note: the row_result event must exist in Apify Console → your Actor → Publication → Monetization with the same name. Until pay-per-event is configured, the Actor still runs and saves data — it just logs Ignored attempt to charge for an event.

Compute cost

Cost on Apify also scales with compute units consumed during a run. The biggest drivers are:

• 📄 Number of pages crawled (maxResults, maxCrawlPages).
• 🖥️ Rendering mode — always is far heavier than auto, and auto is heavier than never.
• 🧮 Concurrency (maxConcurrency) and timeout settings.
• 🌐 Page complexity and network conditions.

To control cost, tune:

• maxResults — hard cap on saved rows.
• maxCrawlPages — hard cap on URLs opened.
• maxCrawlDepth — keep crawls shallow when possible.
• maxConcurrency — balance speed vs. politeness.
• renderingMode: "never" — the cheapest possible run.
• blockMedia: true — skip images/fonts/video on rendered pages to cut render time and bandwidth.
• blockAds: true — skip known ad/tracker requests on rendered pages.
• screenshotFullPage: false — capture only the viewport instead of the whole scrolled page.
• retryFailedWithRendering: false — skip the second attempt on pages that fail once.
• rateLimitDelaySecs — throttle to avoid being rate-limited (which wastes retries).
• saveScreenshots — leave off unless you need images; enabling it renders every page.
• saveHtml: false — drops the inline html field for much smaller (and cheaper to store) rows.

💡 Tip: start with maxResults: 10 for testing, then scale up once your filters and selectors are dialed in.

---

❓ Frequently asked questions (FAQ)

What is the best web scraper for AI and LLM training data?

A web scraper for AI / LLM training data must produce clean text or Markdown, preserve metadata, handle JavaScript-rendered pages, and export to a structured format (JSON, JSONL, Parquet). Website Content Crawler does all four — it strips boilerplate HTML, outputs both plain text and Markdown, captures rich metadata, and writes directly to an Apify Dataset that exports as JSON/CSV.

How do I scrape a website for a RAG pipeline?

1. Run Website Content Crawler with your docs/help/blog URLs as startUrls.
2. Set saveMarkdown: true and tune removeElementsCssSelector to strip nav/footer.
3. Export the Dataset as JSON.
4. Chunk each markdown field, embed with OpenAI / Cohere / Voyage / local models, and upsert to Pinecone/Qdrant/Weaviate.
5. At query time, retrieve top-K chunks and pass them as LLM context.

Can this Actor scrape JavaScript-heavy single-page applications (SPAs)?

Yes. renderingMode: "auto" already handles pages that need rendering automatically; set renderingMode: "always" when every page must be rendered. Use waitForSelector for content that mounts after the initial load, and dynamicContentWaitSecs for late-arriving content — it is a genuine wait in seconds.

How do I export the crawl results?

Open the run in Apify Console → Dataset tab → Export → choose JSON, CSV, Excel, XML, HTML, or RSS. You can also pull data programmatically via the Apify Dataset API.

Does this Actor respect robots.txt?

Optionally, yes — set respectRobotsTxtFile: true and discovered URLs disallowed by the site's robots.txt are skipped (your explicit start URLs are always crawled). It is off by default. Either way, you remain responsible for compliance with robots.txt, website terms of service, and applicable laws; you can also add excludeUrlGlobs to skip specific paths.

How do I avoid being rate-limited or blocked?

• Lower maxConcurrency (start at 2–5).
• Increase rateLimitDelaySecs (e.g., 1.0).
• Set realistic headerUserAgent and customHttpHeaders (cookies, language).
• Enable proxyConfiguration, or set renderingMode: "always" for sites with aggressive bot detection.

Can I crawl multiple websites in one run?

Yes — add multiple URLs to startUrls. By default the crawler stays on each seed's own domain, so you'll get multi-domain coverage.

How is this different from the official apify/website-content-crawler?

This Actor keeps the input surface small and the cost predictable: three rendering modes instead of a matrix of engines, Markdown and metadata by default, screenshots and HTML snapshots as direct links, and one billable event you can reason about. The official Apify Actor has a much larger feature surface. Choose this one when you want simpler input, predictable cost, and AI-first defaults; choose the official one when you need its full enterprise feature set.

What output formats are supported?

JSON, JSONL, CSV, Excel, XML, HTML, RSS — all via standard Apify Dataset exports.

Does it support proxies?

Yes — use the proxyConfiguration input to pick Apify Proxy datacenter or residential groups, a country, or your own proxy URLs. Requests are spread across several proxy sessions automatically. If the chosen proxy is unavailable, the run continues instead of failing.

Can I schedule recurring crawls?

Yes — use Apify Schedules to run this Actor hourly, daily, or weekly, and pipe new rows into your warehouse, vector DB, or notification channel.

---

🛠️ Troubleshooting

Symptom	Likely cause	Fix
No rows saved	Bad startUrls, over-restrictive filters, target offline	Verify startUrls, relax includeUrlGlobs / excludeUrlGlobs, check site availability
Crawl too slow	Rendering every page + high retries	Lower maxConcurrency, switch renderingMode to auto or never
Missing content / empty text	JS-rendered SPA, content behind selector	Set waitForSelector, increase dynamicContentWaitSecs, set renderingMode: "always"
Boilerplate in output	Default removeElementsCssSelector doesn't cover site's nav	Add site-specific selectors (e.g., .sidebar, .related-posts, .ad-slot)
Duplicate rows	Canonical handling off, fragments treated as distinct	Enable canonical (default), set keepUrlFragments: false
Cert errors on staging	Self-signed / expired cert	Set ignoreHttpsErrors: true (dev only)
Blocked / 403 / 429	Bot detection, rate limit	Enable proxyConfiguration (residential helps most), increase rateLimitDelaySecs, lower maxConcurrency, try renderingMode: "always"
No screenshots in output	saveScreenshots off, kvDir empty, or renderingMode: "never"	Set saveScreenshots: true, keep a kvDir prefix, and use auto or always
Only the start URL is crawled	includeUrlGlobs doesn't match any discovered link, or maxCrawlDepth: 0	Widen the globs (/section/**) and raise maxCrawlDepth
Rows are huge / slow to export	html field included	Set saveHtml: false and use htmlUrl instead

---

💬 Help, support & custom builds

For custom solutions, private integrations, white-label crawlers, or implementation help:

📧 hello.dataminds@gmail.com

Useful Apify documentation:

• 📘 Apify documentation
• 🤖 Actors overview
• 🔌 Apify API reference
• 📚 Apify Academy — web scraping
• 🧰 Apify SDKs

---

⚖️ Is web scraping legal?

Use this Actor only on pages you are allowed to access. Respect:

• ✅ robots.txt directives,
• ✅ website terms of service,
• ✅ rate limits and politeness norms,
• ✅ data-protection laws (GDPR, CCPA, etc.) in your jurisdiction,
• ✅ copyright and licensing of extracted content.

You are responsible for compliant data collection, storage, and use. When in doubt, consult a qualified attorney.

---

🔍 SEO keywords targeted

This README is intentionally keyword-rich for discovery on Google Search, Apify Store search, and Bing. Primary and secondary terms covered include:

Primary: website content crawler · web scraping · website crawler · content extraction · web scraping API · Apify Actor · website scraper · site crawler

Content & format: Markdown extraction · plain text extraction · metadata extraction · structured data extraction · HTML cleaner · readable text extractor · canonical URL extraction · Open Graph scraper · JSON-LD extraction · schema.org scraper

AI & LLM: RAG pipeline · retrieval-augmented generation · LLM training data scraper · vector database ingestion · semantic search crawler · embedding pipeline · custom GPT knowledge file · Claude Projects data · LangChain data loader · LlamaIndex reader · AI agent grounding

Capabilities: headless rendering · JavaScript rendering scraper · SPA scraper · React scraper · dynamic content scraper · concurrent crawler · anti-blocking scraper · sitemap crawler · llms.txt crawler · robots.txt compliant crawler · infinite scroll scraper

Use cases: documentation scraper · blog crawler · SEO content audit · competitive intelligence crawler · website-to-JSON · website-to-database · website-to-Markdown · knowledge base scraper · help center crawler · changelog scraper

Integrations: Pinecone ingestion · Qdrant ingestion · Weaviate · Chroma · pgvector · Snowflake · BigQuery · n8n · Make.com · Zapier · OpenAI Embeddings · Anthropic Files API

---

Reference patterns for structure and section ordering:

• 📘 Public Actor: apify/website-content-crawler
• 📗 Apify publishing guidance: Publishing and monetization
• 🔐 Console (access-gated): Console README