Docs to Markdown Crawler

Give your agent documentation text with source URLs and optional section chunks for retrieval. Convert public HTML or Markdown pages, or follow documentation links within a bounded crawl. Each successful page costs $0.0006, plus $0.00005 per run start at default memory.

Try it

{"inputs":[{"url":"https://developers.cloudflare.com/workers/","maxPages":1}]}

An empty input {} converts the same single page, with chunk: false and concurrency: 2. No upstream API key is needed. HTML extraction reads the server response without executing JavaScript.

Use from an agent

Add https://mcp.apify.com?tools=miacx/docs-to-markdown to your MCP client and sign in to Apify. Call the dedicated Actor tool, then use get-dataset-items to read the completed run's dataset. Or set APIFY_TOKEN to your Apify API token and call the API for JSON rows:

curl --fail-with-body 'https://api.apify.com/v2/actors/miacx~docs-to-markdown/run-sync-get-dataset-items' \
  -H "Authorization: Bearer $APIFY_TOKEN" -H 'Content-Type: application/json' \
  -d '{"inputs":[{"url":"https://developers.cloudflare.com/workers/","maxPages":1}]}'

What you get

The dataset contains one row per extracted page or individual input/page failure. Read status first. On ok, use markdown; otherwise read error.code and error.message. Duplicate canonical pages share one result within the run.

Selected fields from replaying the synthetic test/fixtures/install.md fixture follow. Markdown is shortened.

{
  "status": "ok",
  "ok": true,
  "inputUrl": "https://docs.example.com/docs/install.md",
  "url": "https://docs.example.com/docs/install.md",
  "title": "Install",
  "markdown": "# Install\n\nRun `npm install example`.",
  "headings": [
    {"level": 1, "text": "Install"},
    {"level": 2, "text": "Configure"},
    {"level": 2, "text": "Configure"}
  ],
  "lastModified": null,
  "source": "markdown"
}

Successful rows include fetchedAt, the page-body fetch time, and headings in document order. url is the final canonical URL, or a corpus section's declared Source: URL. source is html, markdown or llms-full. lastModified uses HTTP Last-Modified, then sitemap lastmod, then HTML metadata; it is null when unavailable.

With chunk: true, successful rows include chunks with id, headingPath and markdown. Joining chunk Markdown in order reproduces the page. IDs depend on URL, heading path and ordinal, so body-only edits preserve them. Failed rows omit content fields and contain error; their url is null when no page was reached.

The run's key-value record OUTPUT also contains discovery reports and request usage. Read inputs[].truncated there to detect pages left behind by limits.

Input

Field	Default	Accepted input
inputs	Single-page example above	1–5 objects with url, optional maxPages, include and exclude. An empty array is invalid.
inputs[].url	Required in each object	Public HTTP(S) root, page, llms.txt or llms-full.txt URL, at most 2,048 characters.
inputs[].maxPages	10 when omitted in a supplied object	Integer 1–20. A value of 1 converts the exact URL without index discovery.
inputs[].include, inputs[].exclude	[]	Up to 20 pathname globs each, starting with /, at most 200 characters each. * matches within a segment; ** spans segments. Exclude wins.
chunk	false	Include heading-scoped section chunks.
concurrency	2	Integer 1–4, shared across inputs and constrained by per-origin request spacing.

Scope covers the supplied origin and path boundary. A root at /docs includes /docs/**, but excludes /docs-other. Index-file inputs scope to their containing directory. If the supplied URL redirects, scope comes from where it lands: https://docs.example.com that moved to https://example.com/docs is crawled under /docs. The 20-page allowance splits between valid, distinct inputs; equivalent inputs share one allowance. Malformed envelopes fail input validation. Invalid URLs become individual result rows.

Discovery prefers llms.txt links, then sitemap pages, then same-scope HTML links. An explicit llms-full.txt splits on # Title followed by Source: URL. A corpus without those labels stays one document.

Pricing

One Page event costs $0.0006 and includes metadata and optional chunks. Ten successful pages cost $0.00605, including $0.00005 per run start at default 256 MB memory. One thousand Page events cost $0.60, plus run starts.

Failed pages incur no Page events. Duplicate pages incur no additional Page events within a run. Reused page content still counts when returned as a successful page. The $0.00005 run start applies even with no billable pages at default memory. Apify counts one start event per GB of memory, with a minimum of one event.

Limits and data freshness

• Each run shares 20 page attempts, 40 outbound attempts including DNS, and a 25-second work budget.
• Each upstream body is capped at 512 KiB. Discovery follows at most one child sitemap.
• Fetching respects robots.txt, crawl-delay and a minimum 100 ms between same-origin request starts.
• Public documentation only. Credentials and private hosts are unsupported, and discovered pages that redirect out of scope are skipped.
• HTML extraction removes navigation and boilerplate; JavaScript-only content can return empty_content.

The Actor caches and deduplicates within each run, with no persistent cross-run page cache. Reused bodies retain their original fetchedAt; lastModified is source metadata, not a freshness guarantee.

If a row fails

Status	Next step
invalid_url, url_not_allowed	Supply an absolute public HTTP(S) URL without credentials.
robots_disallowed, out_of_scope	Check published access rules or supply the final URL as a separate input.
no_pages	Check path filters or request an exact page with maxPages: 1.
too_large, unsupported_content, empty_content	Use a smaller HTML or Markdown document with server-rendered text.
fetch_failed, http_status	Inspect the error and upstream HTTP status; retry transient failures later.
budget_exhausted	Request fewer pages or narrower roots; inspect OUTPUT for truncation.

A batch can contain successful and failed pages together. For workflows exceeding the synchronous API's 300-second wait, use asynchronous runs.