RAG Website Crawler — Markdown & AI Chunks

Turn websites into clean Markdown and deterministic, deduplicated chunks ready for RAG pipelines, vector databases, AI agents, search indexes, and knowledge bases.

HTTP-first for low cost. Browser rendering, proxies, and asset downloads remain explicit opt-ins.

Why use this Actor?

A generic crawler returns pages. This Actor returns a practical ingestion contract:

Crawl	Clean	Prepare for AI	Control cost
Same-site links and depth limits	Main-content Markdown	Stable chunk IDs and metadata	HTTP mode by default
Include/exclude URL patterns	Canonical URL and language	SHA-256 content hash	Proxy disabled by default
Per-item error rows	Author and publication metadata	Duplicate detection	Byte, page, depth, and concurrency caps

Best for

• RAG and AI teams building reliable ingestion pipelines.
• Developers syncing documentation or help centers into vector databases.
• Agencies exporting clean client-site content through Dataset/API/webhooks.
• Knowledge-base owners who need canonical metadata and duplicate suppression.

Output highlights

Every page produces a traceable Dataset row with:

• url and canonicalUrl;
• clean markdown;
• title, description, author, language, and publication date;
• deterministic contentHash;
• isDuplicate and duplicateOf;
• chunks with stable IDs and page metadata;
• optional downloaded assets;
• machine-readable errorType for failed pages.

Duplicate pages remain visible for auditing but do not create redundant chunks.

Quick start

{
  "startUrls": [{ "url": "https://docs.example.com" }],
  "maxCrawlDepth": 2,
  "maxPages": 100,
  "chunkForRag": true,
  "chunkSize": 500,
  "chunkOverlap": 50,
  "deduplicateContent": true,
  "useBrowser": false,
  "downloadAssets": false,
  "proxyConfiguration": { "useApifyProxy": false }
}

Recommended workflow:

Start URLs → safe HTTP crawl → main-content extraction → Markdown
          → canonical metadata → hash/dedupe → RAG chunks → Dataset/API

Input

Field	Default	Purpose
startUrls	required	Seed pages
maxCrawlDepth	3	Link depth from seeds
maxPages	100	Hard page budget
includeUrlPatterns	[]	Optional URL regex allowlist
excludeUrlPatterns	[]	Optional URL regex blocklist
chunkForRag	true	Create vector-ready chunks
chunkSize	500	Approximate target tokens
chunkOverlap	50	Approximate overlap tokens
deduplicateContent	true	Suppress duplicate chunks
useBrowser	false	Render JavaScript-heavy pages
downloadAssets	false	Download up to 20 images/PDFs per page
proxyConfiguration	disabled	Apify Proxy settings
maxConcurrency	10	Parallel HTTP requests; browser mode is capped at 5
requestDelay	500 ms	Delay between requests
maxContentBytes	1,000,000	Maximum page body size
allowPrivateUrls	false	Explicit internal-network override

Invalid or risky regexes are rejected before crawling. chunkOverlap must be smaller than chunkSize.

Example page result

{
  "url": "https://docs.example.com/getting-started",
  "canonicalUrl": "https://docs.example.com/getting-started",
  "title": "Getting started",
  "markdown": "# Getting started\n\nInstall the SDK...",
  "language": "en",
  "contentHash": "sha256...",
  "isDuplicate": false,
  "duplicateOf": null,
  "useful": true,
  "chunks": [
    {
      "id": "<contentHash>:0",
      "index": 0,
      "text": "# Getting started...",
      "tokenCount": 186,
      "metadata": {
        "url": "https://docs.example.com/getting-started",
        "title": "Getting started",
        "language": "en"
      }
    }
  ],
  "chunkCount": 1,
  "error": null
}

OUTPUT separately reports attempted, technically successful, useful, duplicate, and failed pages so empty/error rows cannot inflate success metrics.

Security and reliability

Public URL fetchers can become SSRF relays. This Actor blocks local/private/special IP ranges, metadata endpoints, credentialed URLs, unsafe redirects, and unsafe browser subrequests by default. It also redacts sensitive query values in logs and caps page/asset response sizes.

allowPrivateUrls: true is an expert override for controlled internal environments.

Cost guidance

• Keep useBrowser: false for static sites and documentation.
• Enable browser rendering only when the raw HTML lacks useful content.
• Leave proxy and assets disabled unless the target requires them.
• Use URL patterns and a small maxPages while calibrating a new site.

Actual cost depends on memory, browser use, proxy traffic, assets, and target response time. Run a small sample before scaling.

Limitations

• This Actor crawls public HTTP(S) pages; it does not log in or bypass access controls.
• JavaScript-heavy sites may require browser mode.
• Content quality depends on the target HTML structure.
• Sites can block automated traffic or impose their own crawl rules.
• DNS validation reduces SSRF risk, but absolute protection against DNS rebinding requires network-level egress controls.

Pricing

Currently kept on normal Apify platform usage to reduce adoption friction while real external demand is measured. No separate Pay-Per-Event charge is added by this Actor.

Support

For issues or feature requests, contact the author through the Apify platform and include the run ID plus a redacted sample URL.