Website to RAG Dataset

Convert public websites, documentation, blogs, and XML sitemaps into clean Markdown, structured page metadata, and stable chunks for RAG pipelines and vector databases.

The Actor uses bounded static HTTP fetching. It does not render JavaScript, follow page links recursively, create embeddings, or use proxies.

What it does

• Accepts direct page URLs, an XML sitemap, or a sitemap index.
• Enforces page, sitemap, concurrency, timeout, redirect, and response-size limits.
• Blocks private, loopback, link-local, reserved, and mixed public/private DNS destinations.
• Respects robots.txt by default.
• Removes common navigation and layout noise.
• Converts the selected page content to Markdown.
• Extracts canonical URLs, titles, descriptions, headings, language, robots metadata, Open Graph data, and Twitter metadata.
• Extracts JSON-LD types, breadcrumbs, article dates, HTTP Last-Modified, and sitemap lastmod values.
• Creates deterministic, query-sensitive chunk IDs and bounded overlapping chunks.
• Writes pages/errors and RAG chunks to separate datasets.
• Saves an aggregate run summary to the OUTPUT key-value-store record.

Input

Provide urls, sitemapUrl, or both:

{
  "sitemapUrl": "https://example.com/sitemap.xml",
  "maxPages": 100,
  "maxSitemaps": 100,
  "sameDomainOnly": true,
  "respectRobotsTxt": true,
  "outputMode": "pages_and_chunks",
  "includeMarkdown": true,
  "includeSchema": true,
  "includeFreshnessSignals": true,
  "chunkSize": 1200,
  "chunkOverlap": 150,
  "maxConcurrency": 5
}

At least one URL or sitemap is required. Only public HTTP and HTTPS destinations are allowed. The Actor revalidates every redirect and every DNS result before connecting.

Output modes

• pages: successful page rows and skipped/error rows in the default dataset.
• chunks: successful chunks in the chunks dataset; skipped/error page rows remain in the default dataset.
• pages_and_chunks: successful pages in the default dataset and successful chunks in the chunks dataset.

Chunk creation is derived from outputMode; there is no separate chunk toggle.

Important controls

maxPages limits eligible source pages. maxSitemaps independently limits XML sitemap files, including child sitemaps.

urlAllowlistPatterns and urlBlocklistPatterns are regular expressions applied before page fetching. Filtered sitemap URLs produce capped skipped rows so filtering remains inspectable without creating an unbounded dataset.

onlySelectors selects known content containers. removeSelectors removes unwanted elements after content selection.

chunkSize and chunkOverlap are approximate tokens using a characters/4 estimate. Oversized paragraphs are split so emitted chunks remain within chunkSize.

Outputs

Pages and errors

The default dataset contains page-level results and all skipped/error rows:

{
  "itemType": "page",
  "url": "https://example.com/docs/getting-started",
  "finalUrl": "https://example.com/docs/getting-started",
  "canonicalUrl": "https://example.com/docs/getting-started",
  "status": "ok",
  "statusCode": 200,
  "metadata": {
    "title": "Getting Started",
    "description": "Learn how to get started.",
    "language": "en",
    "h1": "Getting Started",
    "headings": [
      { "level": 1, "text": "Getting Started" },
      { "level": 2, "text": "Install" }
    ]
  },
  "freshness": {
    "lastModifiedHeader": "2026-06-21T15:42:00.000Z",
    "sitemapLastmod": "2026-06-20T00:00:00.000Z",
    "dateModified": "2026-06-21T00:00:00.000Z"
  },
  "content": {
    "markdown": "# Getting Started\n\nThis guide explains...",
    "contentChars": 4820,
    "estimatedTokens": 1205,
    "textHash": "sha256:...",
    "markdownHash": "sha256:...",
    "lowContent": false
  },
  "chunks": {
    "count": 2,
    "chunkIds": ["example-com-docs-getting-started-a1b2c3d4e5f6-000"]
  },
  "error": null
}

Unsafe, filtered, robots-disallowed, duplicate-canonical, budget-limited, empty chunks-only, non-HTML, and failed targets use status: "skipped" or status: "error" with a structured error object.

RAG chunks

The chunks dataset contains only chunk rows:

{
  "itemType": "chunk",
  "url": "https://example.com/docs/getting-started",
  "canonicalUrl": "https://example.com/docs/getting-started",
  "chunkId": "example-com-docs-getting-started-a1b2c3d4e5f6-000",
  "chunkIndex": 0,
  "chunkText": "# Getting Started\n\nThis guide explains...",
  "headingPath": ["Getting Started"],
  "charLength": 3900,
  "estimatedTokens": 975,
  "textHash": "sha256:...",
  "metadata": {
    "title": "Getting Started",
    "language": "en",
    "schemaTypes": ["TechArticle", "BreadcrumbList"]
  }
}

Chunk IDs combine a readable URL prefix, a hash of the complete normalized source URL, and the chunk index. Query-string variants therefore do not collide.

Run summary

The OUTPUT record reports discovery, selection, processing, filtering, billing, chunks, errors, domains, warnings, and applied cost guards:

{
  "pagesDiscovered": 120,
  "pagesSelected": 100,
  "pagesAttempted": 100,
  "pagesProcessed": 96,
  "pagesFiltered": 20,
  "pagesSkipped": 2,
  "errors": 2,
  "chunksCreated": 214,
  "chargedPages": 96,
  "billingEnabled": true,
  "sitemapsFetched": 3,
  "averageContentChars": 4210,
  "domains": ["example.com"],
  "warnings": [],
  "costGuardrails": {
    "maxPagesApplied": true,
    "maxSitemapsApplied": false,
    "filteredRowsTruncated": true,
    "browserRenderingUsed": false,
    "proxyUsed": false,
    "recursiveCrawlingUsed": false,
    "embeddingsUsed": false
  }
}

chargedPages is zero during local and non-PPE runs. It must equal the number of successfully delivered source pages in a configured PPE cloud run.

Content extraction

The Actor selects content in this order:

1. User-provided onlySelectors
2. <main>
3. <article>
4. [role="main"]
5. Common documentation, post, and content selectors
6. <body>

It removes navigation, headers, footers, scripts, styles, forms, iframes, and configured selectors before Markdown conversion.

Pricing

The publication model is pay per successfully processed source page using the custom page-processed event. Errors, skips, duplicate canonicals, empty chunks-only pages, and budget-limited targets are not charged. Chunks are never charged individually.

The Actor does not manually charge the synthetic apify-actor-start event and must not enable apify-default-dataset-item, which would incorrectly charge default-dataset error rows.

The pre-publish cloud profiles processed 100 pages each with no errors:

Profile	Platform cost	Cost per page
Apify Docs, pages and chunks	$0.005891	$0.0000589
Apify Blog, pages and chunks	$0.006108	$0.0000611
Apify Docs, chunks only	$0.004385	$0.0000438

The guarded cost is 1.5 x $0.0000611 = $0.0000916 per page. Twice that guarded cost rounds up to the minimum $0.0005 increment, so the final page-processed prices remain $0.004/$0.003/$0.002/$0.0015 for the FREE/BRONZE/SILVER/GOLD tiers. These prices exceed the required 30% post-share margin under the guarded cost.

Limitations

• Static server-rendered HTML only; no JavaScript rendering.
• No login-gated, private-network, paywalled, or authenticated pages.
• No recursive page-link crawling.
• No PDFs, browser screenshots, embeddings, vector-database writes, proxies, webhooks, or historical diffs.
• Sites may still reject the Actor's user agent or rate-limit requests.
• Main-content extraction is heuristic; use selectors when a site needs explicit targeting.

Legal and responsible use

Only process public pages you are allowed to use. Respect website terms, copyright, privacy, database rights, and applicable law. robots.txt checks are enabled by default but do not replace the user's legal responsibilities.

Troubleshooting

If output is empty, inspect the default page/error dataset first. It records URL safety blocks, filters, robots decisions, HTTP failures, non-HTML responses, canonical duplicates, and chunks-only pages without usable content.

If Markdown contains layout noise, add removeSelectors. If the main article is missing, add a precise onlySelectors value.

If a sitemap stops early, inspect warnings, maxPagesApplied, and maxSitemapsApplied in OUTPUT.

Local development

npm install
npm test
npm run build
apify validate-schema
apify run --purge