Website to AI / RAG Data Extractor

Turns any website into a clean, embedding-ready corpus. Strips navigation, footers and cookie banners, converts the real content to markdown, and splits it into overlapping chunks that each carry their own URL, title and metadata — so a chunk can go straight into a vector store without being joined back to anything.

Three ways to decide what to fetch

discoveryMode	What it does
urls	Fetch exactly the URLs you list. Follows nothing.
sitemap	Read Sitemap: from robots.txt (falling back to /sitemap.xml), follow nested sitemap indexes, then filter with your URL patterns.
crawl	Walk internal links breadth-first up to maxDepth.

Verified on docs.apify.com: sitemap mode reads the robots.txt declaration → a <sitemapindex> → 6 nested sitemaps → 3,962 URLs, then narrows to the section you asked for.

Three row shapes

• CHUNK (default) — RAG-ready. Carries content, chunkIndex, chunkCount, a stable chunkId, estimatedTokens, plus the page's url, title, description, language, author and publishedAt.
• PAGE — one row per page: full cleaned markdown, all metadata, JSON-LD, OpenGraph, headings outline, internal/external link lists, contentHash for change detection.
• CRAWL_SUMMARY — one per run: what was asked, what came back, and a skippedByReason breakdown.

Plus an ERROR row for any URL that genuinely failed, so every input maps to at least one row. A robots.txt refusal is not an error — it is reported under skippedByReason, because that is the system working.

Example input

{
  "startUrls": [{ "url": "https://docs.apify.com/" }],
  "discoveryMode": "sitemap",
  "includeUrlPatterns": ["/academy/"],
  "outputGranularity": "chunk",
  "chunkSize": 1500,
  "chunkOverlap": 150,
  "maxPages": 200
}

Chunking

Sizes are in characters, not tokens — and that is deliberate. A token count depends on your tokenizer, and a hardcoded ~4-chars-per-token divisor produces chunks that are wrong for every non-English corpus. Each chunk also reports estimatedTokens, clearly named as an estimate.

Chunks break at a paragraph, then line, then sentence boundary near the target, and only hard-cut when none of those appears within 15% of the target. Breaking mid-sentence hurts retrieval far more than a slightly long chunk does. chunkOverlap repeats the tail of the previous chunk so a fact spanning a boundary stays retrievable from either side.

Extraction quality is tagged, not assumed

Every row carries _source:

• S1-trafilatura — main-content extraction with boilerplate stripped, emitted as markdown so heading structure survives into the chunks.
• S2-selectolax — fallback for pages trafilatura declines. It returns nothing rather than guessing on very short or unusual documents, and "no content" is the wrong answer for a page that plainly has some.
• *-short — the page really is that short (a stub, a redirect notice).

S2 output is measurably noisier. A pipeline that wants to weight or filter by extraction quality needs to know which one produced a chunk, so the tag is on every row rather than inferred.

Politeness and policy

respectRobotsTxt defaults to on, and the summary row records which way the run went. A general-purpose crawler pointed at hosts nobody profiled is exactly what robots.txt exists to govern.

Per RFC 9309, an unreachable robots.txt (timeout, 5xx, connection error) means "no policy published" and allows the fetch — a 5xx is not consent withdrawn. A robots.txt that answers with an HTML app shell is likewise treated as no policy; shop.tiktok.com serves a 5.5 KB captcha page there.

Wildcards are matched properly (*, $, longest-match-wins, Allow beats Disallow at equal length). Python's stdlib robotparser matches by prefix only and silently ignores *, which produces false ALLOWs — the dangerous direction for a gate to be wrong in.

Turn respectRobotsTxt off only for a site you own or have written authorisation to crawl. The actor logs a warning when you do.

Known limits

• PDFs are not parsed. They are skipped with an explicit skipped_pdf reason rather than being silently absent, so a documentation site that is mostly PDF reports that honestly instead of looking like an empty crawl.
• No JavaScript. HTTP-only by design. A site that renders its content client-side yields a thin or empty page, reported as a no_content error row rather than a blank success.
• Page bodies are capped at 4 MB before parsing, with bodyTruncated: true on the row when that bites.
• URL patterns filter discovered links, never the URLs you typed. A user who names a URL has already decided they want it. (Sitemap mode is the exception: there the "seeds" are discovered, and narrowing a 40,000-URL sitemap to one section is the whole point of the patterns.)

Transport

chrome124 by default, rotating through chrome136, firefox133, safari18_0 and chrome99_android on a block. Rotating the TLS profile is the highest-value retry on an unprofiled host: most 403s there are JA3 gates, not IP ones.

Datacenter Apify Proxy is the default rather than Residential — most documentation sites, blogs and company websites have no bot mitigation, so residential rates would be wasted. Switch to Residential for hosts that need it.