Website to Markdown for RAG: Clean Text, Chunks, Sitemaps
Pricing
from $1.00 / 1,000 page converteds
Convert static web pages to clean Markdown, text and RAG chunks from URLs, sitemaps or a crawl. HTTP only, no JavaScript. $1.00 per 1,000 pages.
url
Required
string. The page URL as queued (normalised, no fragment), e.g. "https://quotes.toscrape.com/page/2/". For sitemap problems, the sitemap URL. Never null.
finalUrl
Optional
string. Where the page ended up after redirects. Only present when different from url.
source
Optional
string. How the page was found: url (Page URLs), sitemap, start (Start URLs), crawl (followed link) or sitemap-file (a problem with the sitemap file itself).
depth
Optional
integer. Links away from the start URL: 0 for listed pages, sitemap pages and start pages.
status
Required
string. ok (converted, charged $0.001), low-content (little or no text, often a JavaScript-only page; free), blocked-by-robots (robots.txt disallows it; free) or error (see errorCode and message; free). Only "ok" is charged.
errorCode
Optional
string. Present only when status is error. Machine-readable reason: http, not-html, too-large, timeout, network, dns, blocked, platform, invalid-url, redirects, host-refused, robots-unavailable, extract-failed, extract-timeout, too-deeply-nested, bot-challenge (a bot-check or block page: free, never worked around), slow-host (robots.txt asks for a Crawl-delay over the 1.75 s maximum this Actor honours: free), not-processed (the run's time limit was reached before the page could be fetched or converted: free), row-too-large (the converted page does not fit in one dataset row: free), push-failed and sitemap codes.
message
Optional
string. Human-readable explanation of any status other than ok. Absent when status is ok.
count
Optional
integer. On a sitemap problem row only: how many child sitemap files it stands for (one row per sitemap URL you gave, not one per file).
statusCode
Optional
integer. HTTP status code of the page response, e.g. 200. Absent when no response was received.
title
Optional
string or null. The page title. null when the page has none or was not converted.
description
Optional
string or null. The page's meta description. null when the page has none.
lang
Optional
string or null. Language from the html lang attribute, lower case, e.g. "en". null when not declared.
wordCount
Optional
integer. Whitespace-separated words in text. Absent on rows that were not converted.
tokenCountApprox
Optional
integer. Estimated token count of the text: characters divided by 4. Absent on rows that were not converted.
contentSha256
Optional
string. SHA-256 hex hash of the plain text (computed after removal of emails and phones if that is on). Use it to detect changes.
changedSinceLastRun
Optional
boolean or null. Only with a Change detection key: true if the page is new or changed since the last run with that key, false if unchanged. null otherwise.
headings
Optional
array of {level, text}. Headings of the main content in order, e.g. [{"level": 1, "text": "Quotes"}]. Empty or absent when there are none.
markdown
Optional
string. Main content as Markdown (GFM tables, fenced code, absolute links, no images). Absent when Include Markdown is off or the page was not converted.
text
Optional
string. Main content as plain text, blocks separated by blank lines. Absent when Include plain text is off or the page was not converted.
chunks
Optional
array of {index, text, tokens, headingPath}. RAG chunks, present when Chunk size is above 0.
links
Optional
array of {url, text}. Every http(s) link on the page (up to 2,000), present only with Include links.
extraction
Optional
string. readability (a main-content extractor found the article) or body-fallback (the page body was used instead).
warnings
Optional
array of strings. For example low-content, or chunks-dropped-row-too-large / text-dropped-row-too-large / markdown-truncated-row-too-large when a very large page had to be cut to fit one dataset row. Empty or absent when there are none.
bytes
Optional
integer. Bytes downloaded for this page.
fetchedAt
Required
string (ISO 8601 UTC date-time). When the page was fetched.
