Website to RAG Dataset
Pricing
from $1.50 / 1,000 processed pages
Convert public websites, docs, blogs, and XML sitemaps into clean Markdown, structured metadata, and stable chunks for RAG pipelines and vector databases.
List of public web pages to process. You can provide individual URLs, docs pages, blog posts, or landing pages.
[]Optional XML sitemap or sitemap index URL. If provided, URLs from the sitemap are processed up to Max pages.
Maximum number of pages to process.
Only process URLs on the same hostname as the first URL or sitemap URL.
Check robots.txt and skip URLs disallowed for the configured user agent.
Choose whether to output page rows, chunk rows, or both.
Include cleaned Markdown in page output rows.
Extract JSON-LD schema types and key schema fields where available.
Extract Last-Modified, sitemap lastmod, datePublished, and dateModified signals when available.
Include internal and external links found in the main content.
Include cleaned HTML in page rows. Disabled by default to keep output smaller.
Approximate chunk size in estimated tokens. Token estimate uses a simple characters/4 heuristic.
Approximate overlap between chunks in estimated tokens.
Pages with less cleaned content than this value are marked as low-content.
Number of pages fetched in parallel.
Timeout for each page request.
Maximum number of sitemap and sitemap-index files to fetch.
User agent used for requests and robots.txt checks.
Optional regex patterns. If provided, only matching URLs are processed.
[]Optional regex patterns. Matching URLs are skipped.
[]CSS selectors to remove before Markdown conversion.
[ "nav", "footer", "header", "aside", "script", "style", "noscript", "iframe", "form"]Optional CSS selectors for the content area. If provided, only matched elements are used.
[]Avoid processing duplicate pages with the same canonical URL.