Website to Markdown: LLM-Ready Page Content

Turn public web pages into clean Markdown for RAG and AI agents: headings, links, lists and tables kept; scripts and nav, footer and aside elements removed (menus and banners built from plain divs stay). Crawl from one or more start URLs on the same site, honour robots.txt, and get one row per page. Failed and empty pages are free.

Use cases

• RAG and knowledge bases: feed LLM-ready Markdown instead of raw HTML.
• Documentation mirrors: crawl a docs site to a depth you choose and export Markdown.
• Content audits: keep titles, word counts and link counts beside the Markdown body.
• Agents: a simple JSON API of {url, title, markdown} without running your own crawler.

What it does

• Starts from each Start URL, crawls that site only, up to maxDepth and maxPages. "Same site" means the same host name and port (http and https on their default ports count as one site), plus the host a start URL redirects to. www. and the bare domain are different hosts unless the start URL redirects from one to the other.
• Fetches HTML over HTTP (identified user agent) and converts it to Markdown: h1–h6, paragraphs, lists, tables (as Markdown tables), definition lists, links (also around bold text or images), emphasis, code/pre, blockquotes, horizontal rules, and images as Markdown image syntax with alt text and absolute src.
• Dropped: scripts, styles, noscript, template, embedded content (svg including its text, MathML math formulas, iframe, object including its fallback text, canvas), dialog (also when open), nav, footer, aside, form controls (buttons, selects, text areas, and input fields with their values), every header that is not inside article or main, with its headings (site banners and hero blocks, and also section or card headers on pages without main; a header inside article or main is kept), and hidden elements (hidden, aria-hidden="true", inline display:none / visibility:hidden). Cookie banners and other visible chrome built from plain divs are kept. Text that looks like Markdown syntax is escaped, and text that looks like HTML (for example <div>) is written as <div> so it stays text when the Markdown is rendered. What else is dropped, exactly: links with no visible text (icon-only links), and inside headings only the permalink mark itself (a link whose visible text is empty or just a mark glyph such as #, ¶, § or 🔗; words such as "Link", "Anchor" or "Permalink" are text and are kept). Heading text is always kept, also when the heading text is itself a link to its own section (<a href="#id">Title</a>, named anchors, FAQ toggles); such text is kept without the link. A heading link to another page or site keeps its URL (## [Title](https://…)). aria-label / title text is never added to the Markdown. Media with no Markdown form are not output either: video, audio and embed (their fallback text is kept) and data: images (only their alt text is kept). Content is never invented, and apart from these and the size limits below nothing else that is visible in the HTML is removed.
• Lists keep their structure: paragraphs, code blocks and nested lists stay inside their item, and numbered lists keep their numbers (<ol start>). Code blocks are copied verbatim (no Markdown added inside, blank lines kept, language tag kept, e.g. python). A link around a heading (news cards) becomes a heading that links to the story, followed by the rest of the card.
• Decodes each page with its charset: a byte-order mark first, then the HTTP header, then <meta charset>, falling back to UTF-8 and then windows-1252. Charset labels are read with the WHATWG Encoding Standard label table that browsers use (for example x-sjis, windows-31j → Shift_JIS; gb2312, x-gbk → GBK; windows-949, ks_c_5601-1987 → EUC-KR; x-x-big5 → Big5; iso-8859-1 → windows-1252; iso-8859-9 → windows-1254; tis-620 → windows-874), and legacy encodings decode with their superset codec (Shift_JIS as windows-31J, EUC-KR as windows-949, Big5 as Big5-HKSCS, EUC-JP including NEC symbols such as ①). The row's charset field is the WHATWG canonical name (e.g. UTF-8, Shift_JIS, EUC-KR, GBK, windows-1252); a label outside that table is reported with its Python codec name.
• Writes one dataset row per page: status ok with markdown, title, wordCount; or a free error / skipped row with a reason.
• Duplicates (same final URL from another start or a redirect) are converted and charged once per run. A failed or skipped URL reached from several start URLs can give one free error/skipped row per start URL.

robots.txt and our user agent

• User agent: KelpForestLabs-WebsiteContentMarkdown/0.6 (+https://apify.com/kelp_forest_labs; identified bot, respects robots.txt and Crawl-delay).
• robots.txt product tokens we obey, most specific first: KelpForestLabs-WebsiteContentMarkdown, then KelpForestLabs (all our crawlers), then *. Crawl-delay is honoured (up to 10 s).
• robots.txt is checked for every redirect hop before it is requested, using the robots.txt of the host the redirect points to. A redirect into a disallowed path is not followed and not charged.
• Following RFC 9309, if robots.txt answers 5xx or 429, or cannot be reached (network error, timeout), we assume the whole host is disallowed: its pages become free skipped rows. A 404 robots.txt means no restrictions.
• Private and reserved addresses are refused on every request and redirect hop.

Limits on size

• HTML is read up to 10 MB per page; bytes beyond that are not read. The row then has htmlTruncated: true (the page is still converted and charged).
• Pages with extreme nesting are flattened past 50 levels, at most 3 million characters of visible page text are read (spaces and HTML indentation do not count; if this limit is reached the row has truncated: true, textCapped: true and originalTextChars, and the Markdown ends with …(truncated)), and a page that takes longer than 30 s to convert becomes a free error row ("too complex to convert").
• wordCount counts space-separated runs, so Chinese or Japanese text without spaces counts as few "words" (informational only; billing does not depend on it).
• Markdown is capped at 500,000 characters per page. Longer output is cut on a paragraph (or line) boundary and ends with …(truncated); the row has truncated: true and originalMarkdownChars. wordCount and markdownChars always describe the Markdown you receive. A truncated page is charged once.

Pricing (pay per event)

Event	Price
page (one HTML page fetched over HTTP and converted to Markdown, with at least one word of text)	US$0.002

Failed and empty pages are free: HTTP errors, timeouts, DNS failures, robots.txt blocks (including an unreachable robots.txt), private targets, non-HTML responses, cross-site redirects and pages with no text in the HTML (skipped, "needs JavaScript") give a free row. 1,000 converted HTTP pages cost US$2.00.

Each run also has Apify's run-start charge of US$0.00005, once per run for up to 1 GB of memory plus once per extra GB (US$0.00005 at the default 512 MB), charged when the run starts. You can set a maximum charge per run in the Apify Console; the actor stops before the first page it could not be paid for.

How to use

1. Paste one or more Start URLs.
2. Set Max pages and Crawl depth (0 = start pages only). For more than a few hundred pages, raise the run timeout (default 300 s).
3. Leave Fetch mode on http.
4. Run the actor and open the Pages or Markdown view, or export as JSON / CSV.
5. Open the SUMMARY record for converted / error / skipped counts and stopReason.

Input example

{"startUrls": [{"url": "https://docs.apify.com/"}], "maxPages": 10, "maxDepth": 1, "fetchMode": "http"}

Output example (fixture)

{"url": "https://example.test/about", "status": "ok", "event": "page", "httpStatus": 200,
 "title": "About", "markdown": "# About\n\nWe make kelp.\n", "wordCount": 4, "fetchMode": "http", "depth": 1}

FAQ

Does it run JavaScript? Not in this version. HTTP mode reads the HTML the server returns. A page whose HTML has no text (typical for JavaScript apps) gives a free skipped row saying it needs JavaScript. A browser mode is planned; it is not priced or available yet.

Will you invent missing text? No. Markdown only reformats what was in the HTML. Sparse pages stay sparse; pages with no text are free skipped rows.

Do you follow other sites? No. Only the start URL's own site (same host and port) is crawled. External links are counted, not fetched.

How do I block or slow it down on my site? Use the robots.txt product token KelpForestLabs-WebsiteContentMarkdown (or KelpForestLabs for all our crawlers) with Disallow or Crawl-delay lines.

Limits: what this actor does not do

• It does not log in, use cookies, solve captchas, or use proxies to evade blocks.
• It does not fetch private or reserved network addresses (SSRF guard on every hop).
• It does not render JavaScript yet (browser mode planned).
• It does not scrape sites whose robots.txt disallows our crawler, and it does not ignore Crawl-delay.
• It does not return personal data as a product; page content is whatever the site publishes.
• It does not charge for failures or for pages it skips.

Changelog

• 0.6.3 (October 2026): documentation only, no code change. The README now lists every element the converter drops (noscript, template, svg, math, iframe, object, canvas, dialog, input fields, media without a Markdown form, and every header outside article/main); the 0.6.2 list was incomplete.
• 0.6.2 (October 2026): documentation only, no code change. The README and Store description now list every element the converter drops (noscript, template, svg, math, iframe, object, canvas, dialog), say plainly that menus and banners built from plain divs are kept, and mention the 300 s default run timeout.
• 0.6 (October 2026): words such as "Link", "Anchor" or "Permalink" in a heading are never treated as a permalink mark (0.5 could drop them); links inside headings keep their URL unless they point to the same page; charset labels follow the WHATWG table (x-sjis, x-gbk, windows-949, x-euc-jp, iso-8859-9, tis-620 and more) and charset now reports the WHATWG canonical name (0.5 reported Python codec names such as cp932); EUC-JP pages with NEC symbols (①) decode cleanly; word counting is inside the per-page time limit.
• 0.5 (October 2026): heading text that is itself a link to its section (self-links, named anchors, FAQ toggles) is kept again (0.4 dropped it); only permalink marks are removed; the page-text limit ignores whitespace and is always reported as truncated; Shift_JIS, EUC-KR and Big5 pages decode with their superset codecs; unclosed bold/italic tags end at the next block, so later headings, lists and links keep their structure.
• 0.4 (October 2026): heading permalink icons and other text-less links no longer add their aria-label/title to the Markdown (and never make a page billable); a hidden table row with omitted end tags no longer hides the rest of the table; <ol reversed> counts down; a heading inside a table cell stays plain text; deep or hostile nesting is bounded with a per-page time limit (free error row); gb2312 pages decode as GBK; clearer messages when a host refuses connections or times out.
• 0.3 (October 2026): list items with paragraphs, code or nested lists stay in the list (no empty bullets); <ol start>; code blocks verbatim; a hidden element with an omitted end tag no longer hides the rest of the page; wordCount counts text only; text like <div> stays text; a byte-order mark beats the HTTP charset.
• 0.2 (October 2026): pages with form inputs and article headers keep their content; tables, definition lists and block separation; free "needs JavaScript" skip for empty pages; robots.txt on every redirect hop and RFC 9309 handling of unreachable robots.txt; charset detection; truncation flags; same site = same host and port.
• 0.1 (October 2026): first version — HTTP crawl to Markdown, robots.txt, PPE page event; browser mode planned.