Website & Page to Markdown for RAG and LLMs
Pricing
from $0.70 / 1,000 pages
Turn web pages or whole sites into clean Markdown for RAG and LLMs: main content only, with title, links and word count. Whole-site mode reads the sitemap first, can return only changed pages, and splits pages into heading-based chunks. Respects robots.txt and AI opt-outs. No browser.
Versions follow MAJOR.MINOR.PATCH (src/version.py); Apify shows MAJOR.MINOR from .actor/actor.json.
Every run logs its version and records it in the RUN_STATS key-value record.
About future changes in output: main-content detection is heuristic. When a site redesigns, what counts as its main content can shift. The run log and
RUN_STATSname every URL that failed and why; one broken site never affects the other URLs in a run.
RUN_STATS keeps the reason as before.GET /?url=<page> returns the page's Markdown; &format=json the full record; &includeImages=true keeps
images. Same reader and rules as a normal run (robots.txt, AI opt-outs, no private addresses), same price: one page
event per page returned, failed pages free and answered with a status code and the reason. Each page is also in the
Standby run's dataset. Endpoints are described in the actor's Endpoints tab.humble-echidna-<name>, and if that is
refused too it runs without memory for that run. Users' existing memory is untouched: the usual name is tried first.Retry-After answers: when a site or API asks it to wait before retrying, it now waits up
to 90 seconds (was 30), once per site, with at most 3 minutes of such long waits per run; a longer wait, or a
site that asks again after it was waited out, is reported with the reason instead of stalling the run.urls is no longer marked required, so AI agents (Apify's MCP server) see that a dataset alone (datasetId) is enough. Nothing else changes: a run with neither still stops with a clear message.Convert the pages in another actor's results, or chain it after any actor with an integration.
datasetId, a dataset picker with read access) and Field with the
page URL (datasetUrlField). With a dataset, each item's page URL becomes a line of urls and urls is
ignored (Apify fills in its default on API and integration runs; it isn't fetched or charged). In an integration,
"datasetId": "{{resource.defaultDatasetId}}" passes the finished run's results. The field is found automatically
when left empty: the first of url, pageUrl, link, website, loadedUrl with a web address in the first 100
items, then the same names one level down (metadata.url); a dotted path names a nested one. A Google Maps link
(a Maps scraper's url) is never used.utm_* dropped) is converted once.
Items without a URL and values that aren't a public web address are skipped, never fetched or charged, and
counted in the new RUN_STATS.dataset record. A dataset with no page URL at all, or one that can't be read,
fails the run with the reason.sourceTitle, sourcePlaceId and sourceIndex (the source item's title or name, its
placeId, its position in the dataset), on every page found from that item; null for URLs typed in.maxChunkTokens, optional): size chunks for your embedding model, e.g. 512 tokens,
instead of characters (about 4 characters per token). Every chunk now also has approxTokens. Nothing else
changes: without the new field, chunks are split exactly as before, and "only changed pages" keeps its memory
from earlier runs. Price unchanged; chunks are still free.chunkOutput chunk), the smallest max chunk size is now 1,000 characters; a smaller
maxChunkChars is raised to 1,000 and the run log says so. One row per page (the default) still allows 200.RUN_STATS.
The run now stops about 45 seconds before its timeout and succeeds with every page converted so far (delivered and
charged as before); the status message and RUN_STATS.stoppedAtTimeLimit say it stopped early. Raise the run's
timeout for the rest, or re-run with "Only changed pages" to carry on.ogImage and ogSiteName: the page's Open Graph image (absolute URL) and site name, null
when the page has none. Every existing field is unchanged. Price unchanged./platform/storage ->
/storage) now reads the sitemap for the section where it lives, instead of falling back to following links,
which could miss pages that no link reaches. RUN_STATS records the new section as scopeMovedTo.crawlWholeSite): each URL is a site, a section of one (only pages under its path) or a
sitemap URL. Pages are found sitemap first, through the shared mms_sitemap package (robots.txt Sitemap: lines,
else /sitemap.xml and /sitemap_index.xml, indexes followed); a site without a usable sitemap is crawled by
following its same-site links, with no depth limit (a start URL that redirects is scoped to where it lands). maxPagesPerSite caps the pages returned per site (default
100, up to 10,000), and at most 5 pages are fetched per page returned (plus 100).includeUrlPatterns, excludeUrlPatterns) for pages found by crawling, in both crawl modes;
excluded pages are never fetched. The URLs you list are always converted.onlyChangedPages): a sitemap page whose lastmod is unchanged since this search's last
run is skipped without being fetched; any other page is fetched and skipped, not charged, when its Markdown hashes
the same. The memory lives in the page-to-markdown-changes key-value store in the user's account, one record per
input URL and set of options, and only covers pages that were actually delivered.chunkMarkdown, maxChunkChars, chunkOutput): each page's Markdown split by headings,
with oversized sections split at paragraph, then line breaks; each chunk has its heading path, URL, title and
index. Either a chunks array on the page row, or one row per chunk in the run's second dataset, chunks (the
default dataset keeps one row per page).discoveredVia (input, link or sitemap), lastmod, contentHash, chunkCount, chunks;
depth is null for pages found in a sitemap.mms_extract package, shared with the new article-extractor actor. No change in behaviour or output.First release.
www. is the same site; each page is converted and charged once per run.HumbleEchidnaApify/1.0 (+https://apify.com/humble-echidna), the same as every
actor in this portfolio.HumbleEchidnaApify (never the full User-Agent string), with a byte-order mark
handled and paths percent-encoding-normalised (RFC 9309); unreachable robots.txt (5xx/429/network) = disallowed,
per RFC 9309. Opt-outs from AI use honoured too: a page that robots.txt disallows for any well-known AI crawler
(GPTBot, CCBot, ClaudeBot, Google-Extended, ...) is skipped and reported as "site opts out of AI use"
(optedOutOfAI in RUN_STATS). Crawl-delay honoured, retries included (a site asking for more than 30 s is
reported instead of fetched), Retry-After honoured (seconds or a date; more than 30 s is reported), at most 2
concurrent requests per host, and a total time limit per download.mms_common
implementation every actor in this portfolio uses.markdownTruncated).apify-default-dataset-item event.queriesSkipped in RUN_STATS).http://[bad, bad IDNA), as a start URL or a redirect target, is reported for that URL; a
malformed link on a page is skipped and the page is still converted.RUN_STATS has the same keys on every exit path (dropped_invalid, failure and error included).