URL to Markdown for AI Agents: Clean LLM Context

Clean main-content Markdown from up to 200 URLs a run, for AI agents and RAG: title, token estimate, optional chunks. $2.00 per 1,000 pages converted, failures free. Plain HTTP, no browser: a JavaScript-rendered page returns needs_browser, a PDF not_html. No crawl: it reads only the URLs you give.

Price (pay per event): $0.002 per page converted ($2.00 per 1,000 pages), plus the platform's $0.00005 start event per run. Every status other than ok has no page charge.

What it does not do: No browser, no PDF text, no crawl: it reads only the URLs you give (sitemap-url-extractor lists a site's URLs).

Quickstart (with your Apify token):

• API: curl -X POST "https://api.apify.com/v2/acts/edgef-4~url-to-markdown-for-ai-agents/run-sync-get-dataset-items" -H "Authorization: Bearer $APIFY_TOKEN" -H "Content-Type: application/json" -d '{"urls":["https://example.com/article"]}'
• MCP call-actor: {"actor":"edgef-4/url-to-markdown-for-ai-agents","input":{"urls":["https://example.com/article"]}}
• Python: ApifyClient(os.environ["APIFY_TOKEN"]).actor("edgef-4/url-to-markdown-for-ai-agents").call(run_input={"urls":["https://example.com/article"]})

Turn web pages into clean, main-content-only Markdown sized for LLM context. Give up to 200 URLs per run and get one dataset item per URL with the title, canonical URL, language, publish date, word count, token estimate and the Markdown. You can also cap the length, drop links, or get 800 to 1,200 token chunks for RAG.

The per-page charge applies only to pages that convert. SPA shells, PDFs, 404s, robots.txt blocks and timeouts are reported with a clear status and have no page charge.

For AI agents

Call it synchronously and get the items back in one HTTP request:

curl -s -X POST "https://api.apify.com/v2/acts/edgef-4~url-to-markdown-for-ai-agents/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://example.com/article"], "maxChars": 20000, "includeLinks": false}'

• The response is a JSON array with one object per unique input URL, in completion order (not input order): match results to inputs by url. Spellings of one page (host case, default port, fragment, missing scheme) count once.
• run-sync-get-dataset-items waits at most 300 s. For long lists (many URLs on one site with a large Crawl-delay) start a normal run and read its dataset when it finishes; the run keeps converting (and charging) after a sync call gives up.
• Use markdown when status == "ok". Otherwise read message: needs_browser means use a browser-based scraper, not_html means the URL is a file (PDF, image, ...). Only ok items are charged.
• Field names are stable and every item has every field (unused ones are null, 0, false or []).

Input

Field	Type	Default	Description
urls	array of strings	required	Absolute http(s) URLs, max 200. A bare domain gets https://. {"url": "..."} objects also work. Duplicates are processed once, also when spelled differently (host case, default port, fragment, missing scheme).
includeLinks	boolean	true	false keeps link text but drops the URLs (fewer tokens). Images always stay as .
maxChars	integer ≥ 500	none	Cap the Markdown per page. Cuts at the last heading that fits in the second half of the limit, else at the last paragraph break. Sets truncated: true.
chunk	boolean	false	Add chunks: 800 to 1,200 token pieces along heading boundaries, each with its headingPath.
maxConcurrency	integer 1 to 20	5	Pages fetched in parallel across all domains.
minDelayPerDomainSecs	number 0.2 to 30	1	Minimum gap between requests to the same domain. A larger robots.txt Crawl-delay (up to 10 s) wins.

Example input:

{
  "urls": ["https://docs.python.org/3/library/asyncio-task.html", "https://example.com/blog/post"],
  "includeLinks": true,
  "maxChars": 40000,
  "chunk": true
}

Output

One dataset item per URL. Example (shortened):

{
  "url": "https://news.example.com/uk/2026/10/flood-barrier-approved?utm_source=x",
  "finalUrl": "https://news.example.com/uk/2026/10/flood-barrier-approved?utm_source=x",
  "status": "ok",
  "message": null,
  "httpStatus": 200,
  "contentType": "text/html",
  "redirectChain": [],
  "title": "City council approves £48m riverside flood barrier",
  "canonicalUrl": "https://news.example.com/uk/2026/10/flood-barrier-approved",
  "language": "en-GB",
  "publishedAt": "2026-10-27T08:15:00+00:00",
  "description": "The £48m scheme will protect 3,200 homes along the Wend.",
  "markdown": "# City council approves £48m riverside flood barrier\n\nCouncillors voted 41 to 9 ...\n\n## What happens next\n\n...",
  "wordCount": 270,
  "tokenEstimate": 442,
  "tokenizer": "chars/4",
  "truncated": false,
  "chunks": [
    {"index": 0, "headingPath": ["City council approves £48m riverside flood barrier"], "markdown": "# City council ...", "tokenEstimate": 441}
  ],
  "fetchedAt": "2026-10-29T13:20:00+00:00"
}

The Actor's dataset schema describes every field. truncated is also set when a page is cut to fit the platform's 9 MB limit per dataset item (possible with chunk mode on a very long page). tokenizer is normally cl100k_base; the example shows the chars/4 fallback, used only if that encoding cannot load.

Status values

status	Meaning	Charged
ok	Converted. markdown is set.	✅ yes
needs_browser	JavaScript-rendered page (SPA shell) with almost no server-side text. Use a browser-based scraper.	no
not_html	PDF, image, JSON or another non-HTML file, also one the server mislabelled as HTML. contentType says which.	no
http_error	The server answered 4xx/5xx, or a 3xx that is not a followable redirect such as 300 or 304 (see httpStatus).	no
blocked_by_robots	robots.txt disallows the URL (or robots.txt itself answered 5xx / was unreachable).	no
blocked_address	The URL, a redirect, or the DNS answer for its host points to a private, loopback, link local or cloud metadata address. Never fetched.	no
timeout	No complete response within 15 s for one request, no result within 60 s for the URL as a whole (redirects and retries included, waits for the per domain delay not counted), or the run reached its time limit first.	no
fetch_error	DNS/TLS/connection error, redirect loop, unsupported Content-Encoding, body over 10 MB, page with more than 400,000 HTML tags, or a result that could not be stored.	no
invalid_url	Not an http(s) URL, or it contains credentials (user:password@). message says which.	no
empty_content	HTML loaded but had no readable main content.	no
skipped	Not processed because the run reached the max charge you set.	no

What the Markdown looks like

• Main content only. Navigation, headers, footers, sidebars, cookie and consent banners, ads, share buttons, newsletter boxes, comments, hidden elements, forms and scripts are removed.
• Starts with # Title (added if the content has no H1). Heading levels are kept.
• Tables become Markdown tables (pipes escaped, ragged rows padded, colspans expanded, captions in bold).
• Links become [text](absolute URL); images become ![alt text](absolute URL) (lazy-loaded data-src resolved, tracking pixels and base64 placeholders dropped). Spaces, brackets and parentheses in URLs are percent-encoded so a page cannot break out of a link target; images that are not http(s) keep only their alt text.
• Code blocks are fenced and keep their language when the page declares it (language-python, Sphinx highlight-*, ...).

How it fetches

Plain HTTP (httpx), no browser. Redirects are followed manually (max 10) and recorded in redirectChain. robots.txt is checked for every URL and every redirect hop (RFC 9309 rules: 4xx = no rules, 5xx/unreachable = disallow). Requests to the same domain are spaced by minDelayPerDomainSecs. Each request has a 15 s timeout; 429/502/503/504 and network errors are retried twice (honouring Retry-After up to 10 s), and one URL gets at most 60 s of fetching (time spent waiting for its per domain slot is not counted), so a bad page never stalls the run. Non-HTML responses are detected from the headers (and file signatures) and their bodies are not downloaded; neither are error pages. Only gzip and deflate are accepted as Content-Encoding, inflated under the same caps: HTML bodies over 10 MB are refused and robots.txt is read up to 500 KiB. Pages with more than 400,000 HTML tags are not converted, and very long pages are converted one at a time in bounded slices, so one huge or hostile page cannot exhaust the run's memory. The run stops 30 s before its platform timeout and reports unfinished URLs as timeout, so every URL has an item.

Safety

Only http and https URLs are fetched, and never ones with credentials in them. Before every request (the input URL, each redirect hop and robots.txt) the host is resolved and the request is refused with blocked_address if any address is private, loopback, link local, carrier grade NAT, multicast or reserved, including cloud metadata (169.254.169.254, fd00:ec2::254), IPv4 mapped, compatible and translated IPv6 and numeric spellings such as http://2130706433/. IPv6 is fetched only from global unicast addresses (2000::/3). The connection layer resolves the name again, checks again and connects to the checked IP, so DNS rebinding cannot slip past the first check. Proxy settings from the environment are ignored.

Pricing (pay per event)

Event	Price	When
apify-actor-start	0.00005 USD	Apify's standard start event: once per run at up to 1 GB of memory, once more per extra GB.
page-converted	0.002 USD	Once per item with status: "ok".

Every other status has no page charge, and there is no per-item dataset charge. If you set a maximum cost per run, the Actor stops converting once it is reached and reports the remaining URLs as skipped.