Website to RAG Dataset

What is Website to RAG Dataset?

Website to RAG Dataset fetches public, server-rendered HTML from HTTP(S) page URLs, an XML sitemap, or a sitemap index. It produces main-content Markdown, metadata, freshness signals, and stable chunks in separate page and chunk datasets. Use them to build a controlled corpus for retrieval, search, or later embedding.

The Actor uses bounded static HTTP requests. It does not render JavaScript, follow links recursively, authenticate to websites, use proxies, create embeddings, or write to a vector database.

What data can it extract or produce?

• Main-content Markdown, with CSS selectors for inclusion and removal.
• Canonical URL, title, description, language, robots meta, H1, headings, Open Graph, and Twitter metadata.
• Optional JSON-LD types, breadcrumbs, dates, block counts, and parse errors.
• Optional freshness signals from HTTP Last-Modified, sitemap lastmod, and JSON-LD dates.
• Optional internal and external links, capped at 250 per group.
• Overlapping chunks with stable IDs, heading paths, lengths, token estimates, and SHA-256 hashes.
• Page ok, skipped, and error records plus an OUTPUT summary.

Results depend on the source. Missing metadata is null or empty; the Actor does not infer absent values.

What can you use the data for?

• Build documentation corpora for RAG assistants or internal search.
• Convert help-center articles into support-agent knowledge chunks.
• Prepare blogs or knowledge bases for separate embedding and indexing.
• Preserve source URLs, headings, hashes, and freshness for traceable retrieval.
• Create sitemap snapshots for quality or regression checks.

How much does it cost?

The Actor uses pay-per-event pricing. A page-processed event is charged for each page that completes extraction and is accepted for output. Filters, errors, robots skips, duplicate canonicals, spending-limit skips, and empty pages skipped in chunks mode are not charged; chunks are not charged individually.

Apify plan	Per processed page	Per 1,000 processed pages
Free	$0.004	$4.00
Bronze	$0.003	$3.00
Silver	$0.002	$2.00
Gold, Platinum, or Diamond	$0.0015	$1.50

A $0.00005 Actor-start event also applies per run. Formula: Actor start + (processed pages x plan price). For example, 100 processed pages on Gold cost $0.00005 + (100 x $0.0015) = $0.15005.

Check the Pricing tab before large runs. Use maxPages and Apify's maximum total charge limit as guardrails.

How to use Website to RAG Dataset

1. Add page URLs, a sitemap URL, or both in Apify Console.
2. Choose pages, chunks, or pages_and_chunks, then set the page and sitemap limits.
3. Run the Actor and inspect Pages and errors, RAG chunks, and Run summary.
4. Export the appropriate dataset or pass its ID downstream.

What input does the Actor accept?

Provide urls, sitemapUrl, or both. This minimal input uses schema defaults:

{
  "urls": ["https://example.com/"]
}

See the Actor's Input tab for the full schema.

Field	Behavior and interaction
urls, sitemapUrl	Direct pages and/or XML sitemap. Fragments and duplicates are removed. Requires public HTTP(S) URLs without credentials.
maxPages, maxSitemaps	Select 1-10,000 eligible pages; fetch 1-1,000 sitemap files. Filtered diagnostic rows have a separate maxPages-sized cap; chunks are not page-capped.
sameDomainOnly	Require the sitemap hostname, or first direct hostname. Subdomains differ.
urlAllowlistPatterns, urlBlocklistPatterns	Regex filters: URLs must pass the allowlist and not match the blocklist.
respectRobotsTxt, userAgent	Check robots rules with this agent. Retrieval failure warns and allows the page request.
outputMode	Write pages, chunks, or both; diagnostics always use the default dataset.
includeMarkdown, includeHtml	Control page Markdown and content.cleanedHtml; chunks still use Markdown. HTML enlarges output.
includeSchema, includeFreshnessSignals, includeLinks	Enable JSON-LD, freshness, and links. Freshness can parse dates while schema output is off.
chunkSize, chunkOverlap	Characters/4 estimates. Size: 200-4,000; overlap: 0-1,000 and smaller than size.
minContentChars	Marks content.lowContent, without rejecting the page. Range: 0-10,000.
maxConcurrency, requestTimeoutSecs	1-25 parallel pages; 5-120 seconds per request. High concurrency risks rate limits.
onlySelectors, removeSelectors	First inclusion match defines content, then removals apply. No match can yield empty content.
dedupeByCanonical	Keep one result per canonical URL, or final URL as fallback.
saveRunSummaryToKeyValueStore	Save OUTPUT in the default key-value store.

HTML, links, JSON-LD, high overlap, and pages_and_chunks increase processing or output, but not the per-page event rate.

What does the output look like?

Location	Contents
Default dataset (pages)	Successful page rows in pages or pages_and_chunks mode, plus all skipped and error rows in every mode.
Named chunks dataset	Successful chunk rows in chunks or pages_and_chunks mode. It never contains page errors.
Default key-value store, OUTPUT	Run-level totals and warnings when saveRunSummaryToKeyValueStore is enabled.

For the minimal input above, a compact chunk row can look like this:

{
  "itemType": "chunk",
  "url": "https://example.com/",
  "normalizedUrl": "https://example.com/",
  "status": "ok",
  "processedAt": "2026-08-26T15:42:16.972Z",
  "chunkId": "example-com-0f115db062b7-000",
  "chunkIndex": 0,
  "chunkText": "# Example Domain\n\nThis domain is for use in documentation examples without needing permission. Avoid use in operations.\n\n[Learn more](https://iana.org/domains/example)",
  "headingPath": ["Example Domain"],
  "estimatedTokens": 42,
  "title": "Example Domain",
  "description": null,
  "schemaTypes": []
}

See the run's Output tab for full schemas. Page metadata, freshness, schema, content, links, and error are objects or null by status and flags. finalUrl, canonicalUrl, statusCode, contentType, and descriptive fields can be null. Excluded markdown and content.cleanedHtml are null. Chunk canonicalUrl, freshness, title, description, and language can be null; schemaTypes may be empty.

Export either dataset through Console or the Dataset API as JSON, JSONL, CSV, HTML, XLSX, XML, or RSS. JSON or JSONL best preserves nested fields.

How are errors and partial results handled?

Runs may mix outcomes. Compare default-dataset status and error.code with the counts and warnings in OUTPUT.

Symptom	What to check
Empty default dataset	In chunks mode, successes are in chunks. Otherwise inspect sitemap warnings and filters.
No chunks	Confirm the mode requests chunks; check JavaScript-only content, onlySelectors, or NO_CHUNKABLE_CONTENT.
Partial sitemap	Inspect maxPagesApplied, maxSitemapsApplied, filters, timeouts, HTTP errors, and FILTERED_ROWS_TRUNCATED.
Duplicate skipped	DUPLICATE_CANONICAL means another target resolved to the same canonical or final URL. Disable deduplication only intentionally.
Failed target	Use error.retryable and codes such as HTTP_NOT_OK, FETCH_TIMEOUT, RESPONSE_TOO_LARGE, NON_HTML_CONTENT, or URL-safety errors.
Run stopped for budget	CHARGE_LIMIT_REACHED rows are uncharged. Raise the run's total charge limit or reduce maxPages.

What are the limits, performance characteristics, and best practices?

• Responses are capped at 5 MiB and pages follow at most five redirects. Redirects and DNS results are safety-checked.
• Sitemap indexes traverse to depth two, bounded by maxSitemaps and the response cap.
• The 512 MB Actor supports static HTML, not PDFs, authenticated or private-network pages, or client-rendered content.
• Retryable network failures, HTTP 429, and 5xx responses retry up to twice. Time depends on source latency, sitemap size, and throttling.
• Start with low concurrency, a narrow allowlist, and a small maxPages; inspect quality before scaling.
• Use onlySelectors for a known article container and removeSelectors for repeated menus, consent UI, or related-content blocks.
• Token counts are estimates, not tokenizer guarantees. Validate chunk size with the tokenizer used by your embedding model.

How can you use the API, schedules, webhooks, and integrations?

The Actor's API tab provides REST, Node.js, and Python examples for sebastian-actors/website-to-rag-dataset. The official Node.js and Python clients can start runs and retrieve storage records; keep tokens server-side.

Use schedules for periodic refreshes and webhooks for run events. Export JSONL to your ingestion job or pass the named chunks dataset ID to a separate integration such as Pinecone. The Actor itself does not embed or index data, and the integration must read chunks, not the default page/error dataset.

Frequently asked questions

Does it crawl every link on a website?

No. Only supplied URLs and sitemap discoveries are processed.

Can it render JavaScript-heavy pages?

No. Content must be in server-returned HTML.

Does it create embeddings?

No. Embedding and indexing are separate steps.

Does minContentChars remove short pages?

No. It sets content.lowContent. Only chunks mode skips pages with no chunkable Markdown.

Are chunk IDs stable?

Yes, for the same effective URL and chunk order. Content changes can move boundaries; compare hashes too.

Is it legal and responsible to use this Actor?

Use the Actor only for public pages you are authorized to process. Review site terms, copyright and database rights, privacy obligations, and applicable laws. respectRobotsTxt is an operational safeguard, not permission or a substitute for legal review. This is not legal advice.

Avoid personal or sensitive data without a lawful purpose and suitable controls. Do not bypass access controls, authentication, paywalls, or network restrictions.

Which related Actors should you consider?

Actor	Use it when
Website Crawl Preflight Auditor	Check robots, sitemap scope, static quality, JavaScript risk, and crawl settings first.
RAG Dataset Linter	Check existing chunks for duplicates, overlap, ordering, provenance, and malformed content.

Where can you get support?

Open the Issues tab with the run ID, redacted input, relevant error or warning, and expected result. Never post tokens, cookies, private URLs, or sensitive output; include a reproducible public URL when possible.