Website Intelligence Crawler — site to clean text, Markdown and chunks

Give it one or more URLs. It crawls the same site, honours robots.txt, and returns each page as one row: clean text, Markdown, page metadata, same-site links and character-bounded chunks ready to hand to an embedding model.

This is the boring part of a RAG pipeline: turning a public website into rows you can embed. It's also fine if you just want a site's readable text and would rather not write a parser. No API key, no LLM call anywhere, no browser.

What a row looks like

{
  "ok": true,
  "requestedUrl": "https://example.com/",
  "url": "https://example.com/",
  "seedUrl": "https://example.com/",
  "siteKey": "example.com",
  "depth": 0,
  "status": 200,
  "title": "Example Domain",
  "description": null,
  "language": "en",
  "canonicalUrl": "https://example.com/",
  "contentType": "text/html",
  "text": "Example DomainThis domain is for use in documentation examples...",
  "markdown": "# Example Domain\n\nThis domain is for use in documentation examples...",
  "links": [],
  "wordCount": 17,
  "chunks": [
    { "chunkIndex": 0, "text": "Example Domain...", "charCount": 125, "estimatedTokens": 32, "relevanceScore": null }
  ],
  "query": null,
  "outputFormat": "both",
  "fetchedAt": "2026-08-21T16:20:24.453Z"
}

text is the readable body with script, style, nav, header, footer and form stripped out. markdown keeps headings, paragraphs, list items, blockquotes and code blocks. links is deduplicated same-site links with their anchor text, capped at 250 per page. relevanceScore is null unless you pass a query.

How full the columns actually are

Measured on a 60-row run on 2026-08-10. Three seeds (stripe.com, docs.github.com, python.org) at depth 2, 20 pages each. All 60 fetched, none blocked, no diagnostic rows, 14.3 seconds end to end.

Field	Filled
url, status, title, canonicalUrl	100%
text, markdown, chunks, wordCount, fetchedAt	100%
language	98.3%
links	98.3%
description	96.7%

description and language only get emitted when the page actually publishes a meta description / og:description and an html lang. Nothing is invented, so those columns are null on pages that don't. links is empty on a page with no same-site links. Median page: 335 words.

Why it gets pages a plain fetch gets 403 on

Requests carry a real browser TLS and HTTP/2 fingerprint (impit), sweeping a Chrome profile and then a Firefox one. Plain Node fetch gets rejected at the TLS handshake by Cloudflare, Akamai and friends before a single byte of the page is read. That one change is what took this from "works on example.com" to fetching Stripe and GitHub Docs without a proxy or a browser.

There's still no headless browser, so a page that renders nothing server-side comes back thin. See the limits.

Input

Field	What it does
startUrls / urls	Public HTTP(S) URLs. Bare domains accepted, www. normalised, fragments dropped, duplicates removed. Up to 200 seeds.
maxPages	Page records per seed. Default 10, max 100.
maxDepth	Same-site link depth. Default 1, max 5. 0 fetches only the seed.
maxConcurrency	Pages fetched at once. Default 3, max 10.
useRobotsTxt	On by default. Honours Disallow, and Crawl-delay up to a 10-second cap.
query	Whitespace-separated terms. Chunks get a relevanceScore (term hits plus a title-match boost) and are sorted best-first. Plain counting, no embedding service, no key.
chunkSizeChars / chunkOverlapChars	Default 1200 and 120 characters.
outputFormat	json drops markdown, markdown drops text, both keeps them.
requestTimeoutSecs / maxResponseSizeKb	Default 15 seconds and 2048 KB.
fallbackToProxy + proxyConfiguration	Off by default. When on, a page that comes back blocked or errored on direct traffic is retried once through the proxy you configured.

Leave the input empty and you get exactly one labelled _sample: true row and zero charges.

Billing

$0.60 per 1,000 pages, event name page, plus a $0.001 start fee. One charge per successful HTML page row. On the 60-page run above the platform billed exactly {apify-actor-start: 1, page: 60} — one event per real row, nothing else.

Never charged: the empty-input sample row, ROBOTS_DISALLOWED rows, HTTP_ERROR, NETWORK, TIMEOUT, SSRF_BLOCKED, RESPONSE_TOO_LARGE and UNSUPPORTED_CONTENT diagnostics. A run that fetches nothing charges nothing but the start fee.

Worth knowing where that price comes from: a crawl of a site that enforces a Crawl-delay spends about $0.0002 a page in platform compute just waiting, and the price is set to stay above that on the slow sites rather than only on the fast ones.

What it doesn't do

• No headless browser. A page whose content only appears after client-side JavaScript returns whatever the server sent, which is often a thin shell. Use a browser-based crawler for those.
• No cross-site crawling. Links are followed only inside the seed's registrable host.
• No login, no cookies, no paywalled content.
• No AI summarisation, embeddings or entity extraction. relevanceScore is term counting.
• No PDF, DOCX or image parsing. A non-HTML response becomes an UNSUPPORTED_CONTENT diagnostic row.
• Sites behind an interactive challenge (hCaptcha, "press and hold") aren't solved.

Questions

How do I turn a website into Markdown for a RAG pipeline? Put the home page in startUrls, set maxDepth to 2–3 and maxPages to how many pages you want, then read the markdown and chunks fields off each row.

Does it need an OpenAI or Anthropic key? No. There's no LLM call in this actor at all. chunks and relevanceScore are deterministic.

Does it respect robots.txt? Yes, by default. Disallowed paths come back as uncharged ROBOTS_DISALLOWED rows rather than being silently skipped. Crawl-delay is honoured up to a 10-second cap — Hacker News asks for 30 and the crawler waits 10. Googlebot ignores Crawl-delay entirely; this is the middle ground. Set useRobotsTxt: false on sites you own.

How many pages will it return? maxPages per seed, capped at 100. Ten seeds at maxPages: 100 is 1,000 rows.

Can I crawl a site that blocks bots? Often, without a proxy — see the fingerprint note. If a specific site still blocks you, turn on fallbackToProxy and attach an Apify Proxy configuration; blocked pages get retried once through it.

What happens if I give it a bad URL? One uncharged diagnostic row with errorCode: BAD_INPUT and the reason. Private, local and link-local addresses are refused with SSRF_BLOCKED.