Website to Markdown: LLM-Ready Page Content avatar

Website to Markdown: LLM-Ready Page Content

Pricing

from $2.00 / 1,000 page converted (http)s

Go to Apify Store
Website to Markdown: LLM-Ready Page Content

Website to Markdown: LLM-Ready Page Content

Article text extractor for public web pages: clean Markdown for RAG and AI agents, with boilerplate (nav, footer, site headers, sidebars, scripts) stripped and headings, links, lists and tables kept. Honours robots.txt. Failed and empty pages are free.

Pricing

from $2.00 / 1,000 page converted (http)s

Rating

0.0

(0)

Developer

kelp forest

kelp forest

Maintained by Community

Actor stats

0

Bookmarked

2

Total users

1

Monthly active users

25 days ago

Last modified

Features

Limited permissions

Share

Turn public web pages into clean Markdown for RAG and AI agents: headings, links, lists and tables kept; scripts, nav/footer/aside and role="navigation" blocks removed (menus and banners built from plain divs stay). Crawl from one or more start URLs on the same site, honour robots.txt, and get one row per page. Failed and empty pages are free.

Use cases

  • RAG and knowledge bases: feed LLM-ready Markdown instead of raw HTML.
  • Documentation mirrors: crawl a docs site to a depth you choose and export Markdown.
  • Content audits: keep titles, word counts and link counts beside the Markdown body.
  • Agents: a simple JSON API of {url, title, markdown} without running your own crawler.

What it does

  • Starts from each Start URL, crawls that site only, up to maxDepth and maxPages. "Same site" means the same host name and port (http and https on their default ports count as one site), plus the host a start URL redirects to. www. and the bare domain are different hosts unless the start URL redirects from one to the other.
  • Fetches HTML over HTTP (identified user agent) and converts it to Markdown: h1–h6, paragraphs, lists, tables (as Markdown tables), definition lists, links (also around bold text or images), emphasis, code/pre, blockquotes, horizontal rules, and images as Markdown image syntax with alt text and absolute src. Relative links and images are resolved against the page URL, or against the page's <base href> when it has one (the first base with an href, http/https only, not inside template or svg; an empty href means the page URL).
  • Dropped: scripts, styles, noscript, template, embedded content (svg including its text, MathML math formulas, iframe, object including its fallback text, canvas), dialog (also when open), nav, footer, aside, form controls (buttons, selects, text areas, and input fields with their values), every header that is not inside article or main, with its headings (site banners and hero blocks, and also section or card headers on pages without main; a header inside article or main is kept), and hidden elements (hidden, aria-hidden="true", inline display:none / visibility:hidden). Also dropped as page chrome: anything marked role="navigation" (the same as nav, for example Wikipedia navboxes and documentation sidebars; never on main or article), "Skip to content" / "Jump to content" links that point to the same page (an href starting with #; a skip link written with the full page URL is kept as a normal link, except MediaWiki mw-jump-link links, which are always dropped), other same-page skip links (such as "Skip to navigation" or "Skip to search"), language-switcher entries (a list item that holds only a link to another language version: the link has hreflang plus a lang attribute or a language-link class, or the list is made only of such links, each in a different language and each labelled with a language name or code such as "Deutsch" or "FR"; hreflang alone is not enough, a link inside a heading is never dropped, and an item with any other text keeps its link), and the MediaWiki interface blocks for category links, "Retrieved from" and "[edit]" section links. Cookie banners and other visible chrome built from plain divs are kept. Text that looks like Markdown syntax is escaped, and text that looks like HTML (for example <div>) is written as &lt;div&gt; so it stays text when the Markdown is rendered. What else is dropped, exactly: links with no visible text (icon-only links), and inside headings only the permalink mark itself (a link whose visible text is empty or just a mark glyph such as #, , § or 🔗; words such as "Link", "Anchor" or "Permalink" are text and are kept). The mark ↗ is dropped the same way, and a mark link is dropped also when it links to another site. Heading text is always kept, also when the heading text is itself a link to its own section (<a href="#id">Title</a>, named anchors, FAQ toggles); such text is kept without the link. A heading link to another page or site keeps its URL (## [Title](https://…)). aria-label / title text is never added to the Markdown. Media with no Markdown form are not output either: video, audio and embed (their fallback text is kept) and data: images (only their alt text is kept). An img without src also gives only its alt text. Content is never invented, and apart from these and the size limits below nothing else that is visible in the HTML is removed.
  • Hero heading (one exception to the dropped header):
    • When the page has no other h1, the first h1 of a dropped header (one outside article/main) is kept as the page heading, for example a docs home page whose title sits in a hero banner. It is only added where the header starts a new block, not in the middle of a line of text, and not when the header is inside a list, quote or table.
    • No heading is added, and later h1s are not tried, when that first h1's only text is image alt text such as a logo, or when it is longer than 2,000 characters, counted on the heading text including any menu text and image alt text (buttons, svg, select and hidden elements do not count).
    • If that first h1 has no heading text at all (for example it is empty or holds only a button or a hidden element), the next h1 with text in that header or a later such header is used instead. A first h1 whose only text is a bare # is skipped the same way for a later header, but blocks a second h1 in the same header.
    • Buttons, svg, select and hidden elements inside the heading h1 are left out of it, but other text inside it, such as a nav menu or a "Skip to content" link, is kept as part of the heading.
  • Lists keep their structure: paragraphs, code blocks and nested lists stay inside their item, and numbered lists keep their numbers (<ol start>). Code blocks are copied verbatim (no Markdown added inside, blank lines kept, language tag kept, e.g. python). A link around a heading (news cards) becomes a heading that links to the story, followed by the rest of the card.
  • Decodes each page with its charset: a byte-order mark first, then the HTTP header, then <meta charset>, falling back to UTF-8 and then windows-1252. Charset labels are read with the WHATWG Encoding Standard label table that browsers use (for example x-sjis, windows-31j → Shift_JIS; gb2312, x-gbk → GBK; windows-949, ks_c_5601-1987 → EUC-KR; x-x-big5 → Big5; iso-8859-1 → windows-1252; iso-8859-9 → windows-1254; tis-620 → windows-874), and legacy encodings decode with their superset codec (Shift_JIS as windows-31J, EUC-KR as windows-949, Big5 as Big5-HKSCS, EUC-JP including NEC symbols such as ①). The row's charset field is the WHATWG canonical name (e.g. UTF-8, Shift_JIS, EUC-KR, GBK, windows-1252); a label outside that table is reported with its Python codec name.
  • Writes one dataset row per page: status ok with markdown, title, wordCount; or a free error / skipped row with a reason.
  • Duplicates (same final URL from another start or a redirect) are converted and charged once per run. A failed or skipped URL reached from several start URLs can give one free error/skipped row per start URL.

robots.txt and our user agent

  • User agent: KelpForestLabs-WebsiteContentMarkdown/0.7 (+https://apify.com/kelp_forest_labs; identified bot, respects robots.txt and Crawl-delay).
  • robots.txt product tokens we obey, most specific first: KelpForestLabs-WebsiteContentMarkdown, then KelpForestLabs (all our crawlers), then *. Crawl-delay is honoured (up to 10 s).
  • robots.txt is checked for every redirect hop before it is requested, using the robots.txt of the host the redirect points to. A redirect into a disallowed path is not followed and not charged.
  • Following RFC 9309, if robots.txt answers 5xx or 429, or cannot be reached (network error, timeout), we assume the whole host is disallowed: its pages become free skipped rows. A 404 robots.txt means no restrictions.
  • Private and reserved addresses are refused on every request and redirect hop.

Limits on size

  • HTML is read up to 10 MB per page; bytes beyond that are not read. The row then has htmlTruncated: true (the page is still converted and charged).
  • Pages with extreme nesting are flattened past 50 levels, at most 3 million characters of visible page text are read (spaces and HTML indentation do not count; if this limit is reached the row has truncated: true, textCapped: true and originalTextChars, and the Markdown ends with …(truncated)), and a page that takes longer than 30 s to convert becomes a free error row ("too complex to convert"). The time is checked between parsed tags and text, so one pathological tag can run past 30 s before the row is ended. On a page with more than 20,000 unclosed elements, or more than 20,000 visible header elements outside article/main and outside another such header, the hero h1 exception above is not applied (every header outside article/main is dropped; a <base href> found before that point still applies, a later one is not used). About half a million or more attributes in one start tag can need 0.5-3 GB of memory, more than the 512 MB default; such a run can fail for lack of memory (no attribute limit yet).
  • wordCount counts space-separated runs of visible text (link URLs and image syntax are not words, and a link directly followed by punctuation stays one word), so Chinese or Japanese text without spaces counts as few "words" (informational only; billing does not depend on it).
  • Markdown is capped at 500,000 characters per page. Longer output is cut on a paragraph (or line) boundary and ends with …(truncated); the row has truncated: true and originalMarkdownChars. wordCount and markdownChars always describe the Markdown you receive. A truncated page is charged once.

Pricing (pay per event)

EventPrice
page (one HTML page fetched over HTTP and converted to Markdown, with at least one word of text)US$0.002

Failed and empty pages are free: HTTP errors, timeouts, DNS failures, robots.txt blocks (including an unreachable robots.txt), private targets, non-HTML responses, cross-site redirects and pages with no text in the HTML (skipped, "needs JavaScript") give a free row. A page whose only text is a hero h1 kept as the page heading (see above) is a converted page and is charged as one page. 1,000 converted HTTP pages cost US$2.00.

Each run also has Apify's run-start charge of US$0.00005, once per run for up to 1 GB of memory plus once per extra GB (US$0.00005 at the default 512 MB), charged when the run starts. You can set a maximum charge per run in the Apify Console; the actor stops before the first page it could not be paid for.

How to use

  1. Paste one or more Start URLs.
  2. Set Max pages and Crawl depth (0 = start pages only). For more than a few hundred pages, raise the run timeout (default 300 s).
  3. Leave Fetch mode on http.
  4. Run the actor and open the Pages or Markdown view, or export as JSON / CSV.
  5. Open the SUMMARY record for converted / error / skipped counts and stopReason.

Input example

{"startUrls": [{"url": "https://docs.apify.com/"}], "maxPages": 10, "maxDepth": 1, "fetchMode": "http"}

Output example (fixture)

{"url": "https://example.test/about", "status": "ok", "event": "page", "httpStatus": 200,
"title": "About", "markdown": "# About\n\nWe make kelp.\n", "wordCount": 4, "fetchMode": "http", "depth": 1}

FAQ

Does it run JavaScript? Not in this version. HTTP mode reads the HTML the server returns. A page whose HTML has no text (typical for JavaScript apps) gives a free skipped row saying it needs JavaScript. A browser mode is planned; it is not priced or available yet.

Will you invent missing text? No. Markdown only reformats what was in the HTML. Sparse pages stay sparse; pages with no text are free skipped rows.

Do you follow other sites? No. Only the start URL's own site (same host and port) is crawled. External links are counted, not fetched.

How do I block or slow it down on my site? Use the robots.txt product token KelpForestLabs-WebsiteContentMarkdown (or KelpForestLabs for all our crawlers) with Disallow or Crawl-delay lines.

Limits: what this actor does not do

  • It does not log in, use cookies, solve captchas, or use proxies to evade blocks.
  • It does not fetch private or reserved network addresses (SSRF guard on every hop).
  • It does not render JavaScript yet (browser mode planned).
  • It does not scrape sites whose robots.txt disallows our crawler, and it does not ignore Crawl-delay.
  • It does not return personal data as a product; page content is whatever the site publishes.
  • It does not charge for failures or for pages it skips.

Changelog

  • 0.7.3 (October 2026): documentation only, conversion is unchanged from 0.7.2. The README now states which hero h1 is used when the first one has no text, what is left out of the hero heading, that a full-URL skip link is kept, and that a page whose only text is the hero heading is charged once (as in 0.7.2).
  • 0.7.2 (October 2026): a page whose only h1 sits in a hero header outside main keeps that h1 (for example docs.apify.com); relative links and images follow <base href>; private-use characters (U+E000, U+E001) in page text are kept; control characters are removed from image alt text. role="navigation" blocks are handled exactly as in 0.7.1 (always removed).
  • 0.7 (October 2026): page chrome is stripped more cleanly: role="navigation" blocks (Wikipedia navboxes, documentation sidebars), "Skip to content" / "Jump to content" links, language-switcher lists (recognised by a real language signal, not by hreflang alone, because some site builders put hreflang on every link), and MediaWiki category links, "Retrieved from" and "[edit]" links. wordCount no longer counts a link next to punctuation as two words (for example Vec<T> in API docs), so counts on link-heavy pages are slightly lower.
  • 0.6.3 (October 2026): documentation only, no code change. The README now lists every element the converter drops (noscript, template, svg, math, iframe, object, canvas, dialog, input fields, media without a Markdown form, and every header outside article/main); the 0.6.2 list was incomplete.
  • 0.6.2 (October 2026): documentation only, no code change. The README and Store description now list every element the converter drops (noscript, template, svg, math, iframe, object, canvas, dialog), say plainly that menus and banners built from plain divs are kept, and mention the 300 s default run timeout.
  • 0.6 (October 2026): words such as "Link", "Anchor" or "Permalink" in a heading are never treated as a permalink mark (0.5 could drop them); links inside headings keep their URL unless they point to the same page; charset labels follow the WHATWG table (x-sjis, x-gbk, windows-949, x-euc-jp, iso-8859-9, tis-620 and more) and charset now reports the WHATWG canonical name (0.5 reported Python codec names such as cp932); EUC-JP pages with NEC symbols (①) decode cleanly; word counting is inside the per-page time limit.
  • 0.5 (October 2026): heading text that is itself a link to its section (self-links, named anchors, FAQ toggles) is kept again (0.4 dropped it); only permalink marks are removed; the page-text limit ignores whitespace and is always reported as truncated; Shift_JIS, EUC-KR and Big5 pages decode with their superset codecs; unclosed bold/italic tags end at the next block, so later headings, lists and links keep their structure.
  • 0.4 (October 2026): heading permalink icons and other text-less links no longer add their aria-label/title to the Markdown (and never make a page billable); a hidden table row with omitted end tags no longer hides the rest of the table; <ol reversed> counts down; a heading inside a table cell stays plain text; deep or hostile nesting is bounded with a per-page time limit (free error row); gb2312 pages decode as GBK; clearer messages when a host refuses connections or times out.
  • 0.3 (October 2026): list items with paragraphs, code or nested lists stay in the list (no empty bullets); <ol start>; code blocks verbatim; a hidden element with an omitted end tag no longer hides the rest of the page; wordCount counts text only; text like <div> stays text; a byte-order mark beats the HTTP charset.
  • 0.2 (October 2026): pages with form inputs and article headers keep their content; tables, definition lists and block separation; free "needs JavaScript" skip for empty pages; robots.txt on every redirect hop and RFC 9309 handling of unreachable robots.txt; charset detection; truncation flags; same site = same host and port.
  • 0.1 (October 2026): first version — HTTP crawl to Markdown, robots.txt, PPE page event; browser mode planned.