Agent Fetch: URL to Markdown for LLMs and RAG

Agent Fetch is a URL to Markdown converter for LLMs, RAG pipelines and AI agents. Give it up to 25 web page URLs and it returns each page's main content as clean Markdown, without menus, cookie banners, sidebars or footers, plus the title, canonical URL, language, word count and the links inside the text.

It fetches over plain HTTP by default. Pages that only show content after JavaScript runs are loaded in a headless browser automatically. An optional JSON Schema pulls named fields such as author or publish date out of the page's own JSON-LD and meta tags, with no LLM and no API key. Blocked, failed and empty pages come back as labelled records and cost nothing.

Who it is for

• AI agents that need to read a web page mid-task. Raw HTML burns thousands of tokens on markup and menus. One call returns one flat record with Markdown that drops straight into a prompt.
• RAG ingestion. Turn a list of documentation pages or articles into Markdown with titles, canonical URLs and word counts already attached.
• Research pipelines. Pull ten sources into one dataset and summarise them in a single pass.
• Content and link monitoring. Track what a page says and which pages it links to over time.
• Structured fields without an LLM bill. Get author, publish date and description from the page's JSON-LD, deterministically.

How to convert a URL to Markdown

1. Paste one or more URLs into URLs (or send {"url": "..."} through the API).
2. Leave the defaults and start the run.
3. Read the Pages view of the dataset, or fetch the items through the API or an MCP client.

Minimum input:

{
  "url": "https://en.wikipedia.org/wiki/Markdown"
}

A fuller input:

{
  "urls": [
    "https://en.wikipedia.org/wiki/Markdown",
    "https://docs.python.org/3/tutorial/introduction.html"
  ],
  "autoRender": true,
  "includeLinks": true,
  "includeArticleJson": false,
  "maxMarkdownChars": 120000,
  "schema": {
    "properties": {
      "headline": { "type": "string" },
      "author": { "type": "string" },
      "datePublished": { "type": "string", "format": "date" }
    }
  }
}

Field	Default	What it does
urls		1 to 25 http or https URLs. Duplicates and #fragments are removed.
url		One page, for callers that send {"url": "..."}. Merged in front of urls.
render (Rendered page)	false	Load every page in headless Chromium. Charged as a render.
autoRender	true	Load only empty JavaScript app shells in the browser.
includeLinks	true	Add the links array.
includeArticleJson	false	Add the article metadata object.
maxMarkdownChars	120000	Cut longer Markdown at a paragraph or line break, with a note at the end, never past this length (500 to 500000).
schema		JSON Schema with a properties map, for field extraction.
respectRobotsTxt	true	Skip paths robots.txt disallows for AgentFetchBot.
timeoutSecs	30	Time limit for loading each page (5 to 120). Conversion is not counted.
userAgent	AgentFetchBot/1.0	Sent with every request. robots.txt is always matched as AgentFetchBot.

What you get for each URL

Field	Always present	Notes
url	yes	exactly the URL you asked for
finalUrl	yes	after redirects
status	yes	see the status table below
httpStatus	yes	0 when no response was received
method	yes	http or browser
fetchedAt	yes	ISO 8601
title	yes	from the readability pass, falling back to the page title
canonicalUrl	yes	rel=canonical, falling back to og:url
language	yes	from html lang, falling back to og:locale
markdown	yes	the main content as Markdown
wordCount, charCount, linkCount	yes	measured on the Markdown
truncated	yes	true when maxMarkdownChars cut the text; the Markdown then ends with [truncated by maxMarkdownChars]
durationMs	yes	time from starting this URL to its record, including any wait for the converter
reason	on failures	why this URL produced no content
autoRendered	when it happened	the page was an app shell and was loaded in the browser
links	with includeLinks	{ url, text, external }, absolute URLs, up to 500
article	with includeArticleJson	excerpt, byline, site name, dates, image, JSON-LD
extracted, extractedSources	when a schema found something	your fields, and where each value came from
extractedFieldCount	when a schema was given	how many fields were found

Optional keys are left out rather than set to null.

status	Meaning	Charged
ok	content extracted	yes
blocked_by_robots	robots.txt disallows this path	no
http_error	the server answered 4xx or 5xx (5xx is retried once)	no
timeout	the page did not finish inside timeoutSecs	no
fetch_error	DNS, TLS or connection failure (retried once)	no
invalid_url	not http or https, or the URL, a redirect or its resolved address is private	no
unsupported_content_type	a PDF, image, archive, feed, or a page too large to convert	no
render_unavailable	render was on and the browser would not start	no
empty_content	fetched fine, held no readable text	no
needs_render	an empty JavaScript app shell, and autoRender was off	no

A bad URL is a record, never a crashed run. One dead link out of 25 costs nothing and does not touch the other 24.

Output example

Real records from runs on 8 October 2026. In one of those runs, the prefill input converted 3 of 3 pages in 3.5 seconds of processing; the whole run took 7 seconds.

A documentation page. markdown is cut here to its first paragraph; the full field is 18,194 characters, and links is cut to its first 2 of 19:

{
  "url": "https://docs.python.org/3/tutorial/introduction.html",
  "finalUrl": "https://docs.python.org/3/tutorial/introduction.html",
  "status": "ok",
  "httpStatus": 200,
  "method": "http",
  "fetchedAt": "2026-10-08T21:59:21.347Z",
  "title": "3. An Informal Introduction to Python",
  "canonicalUrl": "https://docs.python.org/3/tutorial/introduction.html",
  "language": "en",
  "wordCount": 2834,
  "charCount": 18194,
  "linkCount": 19,
  "truncated": false,
  "durationMs": 3127,
  "markdown": "In the following examples, input and output are distinguished by the presence or absence of prompts ([\\>>>](https://docs.python.org/3/glossary.html#term-0) and […](https://docs.python.org/3/glossary.html#term-...)): to repeat the example, you must type everything after the prompt, when the prompt appears; lines that do not begin with a prompt are output from the interpreter. Note that a secondary prompt on a line by itself in an example means you must type a blank line; this is used to end a multi-line command.",
  "links": [
    {
      "url": "https://docs.python.org/3/glossary.html#term-0",
      "text": ">>>",
      "external": false
    },
    {
      "url": "https://docs.python.org/3/glossary.html#term-...",
      "text": "…",
      "external": false
    }
  ]
}

The Wikipedia article from the same run, metadata only. Its markdown holds 41,633 characters, with the infobox as a table:

{
  "url": "https://en.wikipedia.org/wiki/Markdown",
  "finalUrl": "https://en.wikipedia.org/wiki/Markdown",
  "status": "ok",
  "httpStatus": 200,
  "method": "http",
  "fetchedAt": "2026-10-08T21:59:21.313Z",
  "title": "Markdown",
  "canonicalUrl": "https://en.wikipedia.org/wiki/Markdown",
  "language": "en",
  "wordCount": 5575,
  "charCount": 41633,
  "linkCount": 209,
  "truncated": false,
  "durationMs": 2822
}

A JavaScript-only page. The plain fetch got an empty app shell, so autoRender loaded it in the browser and it was charged as a render:

{
  "url": "https://quotes.toscrape.com/js/",
  "status": "ok",
  "httpStatus": 200,
  "method": "browser",
  "autoRendered": true,
  "title": "Quotes to Scrape",
  "language": "en",
  "wordCount": 202,
  "charCount": 1064,
  "durationMs": 6725
}

With a schema of headline, author and datePublished, on a news article that publishes JSON-LD:

{
  "url": "https://www.propublica.org/article/3m-forever-chemicals-pfas-pfos-inside-story",
  "status": "ok",
  "title": "Toxic Gaslighting: How 3M Executives Convinced a Scientist the Forever Chemicals She Found in Human Blood Were Safe",
  "extractedFieldCount": 3,
  "extracted": {
    "headline": "Toxic Gaslighting: How 3M Executives Convinced a Scientist the Forever Chemicals She Found in Human Blood Were Safe",
    "author": "Sharon Lerner, Haruka Sakaguchi",
    "datePublished": "2024-05-20T10:00:00+00:00"
  },
  "extractedSources": {
    "headline": "json-ld",
    "author": "json-ld",
    "datePublished": "json-ld"
  }
}

Failed pages come back labelled and are not charged:

[
  {
    "url": "https://www.google.com/search?q=markdown",
    "status": "blocked_by_robots",
    "httpStatus": 0,
    "markdown": "",
    "wordCount": 0,
    "reason": "robots.txt at https://www.google.com/robots.txt disallows /search?q=markdown for this user agent."
  },
  {
    "url": "https://en.wikipedia.org/wiki/Agent_Fetch_page_that_does_not_exist",
    "status": "http_error",
    "httpStatus": 404,
    "markdown": "",
    "wordCount": 0,
    "reason": "Server answered HTTP 404 Not Found"
  }
]

Each run also writes a summary to the key-value store under OUTPUT, with the counts actually charged:

{
  "ok": true,
  "requested": 3,
  "returned": 3,
  "succeeded": 3,
  "byStatus": {
    "ok": 3
  },
  "charged": {
    "fetch": 3,
    "render": 0,
    "extract": 0
  },
  "totalWords": 16795,
  "tookMs": 3541,
  "parser": {
    "jobs": 3,
    "fastParserJobs": 0,
    "workersStarted": 1,
    "recycled": 0,
    "outOfMemory": 0,
    "timedOut": 0
  }
}

Use it from Claude, Cursor or any MCP client

Add this URL as a remote MCP server in Claude, Cursor, VS Code or any MCP client:

https://mcp.apify.com?tools=omargnagy/agent-fetch-markdown

For clients configured with JSON:

{
  "mcpServers": {
    "agent-fetch": {
      "url": "https://mcp.apify.com?tools=omargnagy/agent-fetch-markdown"
    }
  }
}

Connect your Apify account when the client asks. The agent then sees one tool, omargnagy--agent-fetch-markdown, that takes url or urls and returns the records above.

From any HTTP client, one call runs the Actor and returns the pages:

curl -X POST "https://api.apify.com/v2/acts/omargnagy~agent-fetch-markdown/run-sync-get-dataset-items" \
  -H "Authorization: Bearer YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://en.wikipedia.org/wiki/Markdown"}'

From Python:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("omargnagy/agent-fetch-markdown").call(
    run_input={"urls": ["https://en.wikipedia.org/wiki/Markdown"]}
)
for page in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(page["status"], page["title"], page["wordCount"])

Pricing

Pay per event. You pay only for pages that produced content:

Event (Store title)	Price on the Free plan	Charged when
fetch (Fetched page)	$1.00 per 1,000 pages	a page was fetched over plain HTTP and converted to Markdown
render (Rendered page)	$3.00 per 1,000 pages	a page was loaded in the headless browser (with render on, or an app shell under autoRender)
extract (Schema extraction)	$5.00 per 1,000 pages	your schema found at least one field the page itself states

Prices are lower on Bronze, Silver and Gold plans. There is no start fee. A page charges fetch or render, never both; extract is on top. Blocked, failed, timed-out and empty pages are free, and so are fields filled only by fallbacks.

If you set a maximum cost per run, each page is charged before it is delivered, no new URLs start once the budget is spent, and nothing is delivered that was not charged. The status message then reads "Stopped at your maximum cost", and OUTPUT reports the real charged counts. A run can end a page or two short of the budget; it is never charged past it.

How the field extraction works

The schema is matched against what the page publishes about itself, in this order: JSON-LD, then meta and microdata tags, then label and value pairs (definition lists, two-column tables), then labelled lines of text. Values are coerced to your declared type and checked against any enum. JSON-LD @id references are resolved, so an author comes back as a name rather than an internal id.

extractedSources tells you where each value came from. json-ld, meta, page:pairs and text:label are stated by the page. page:title, page:canonical, page:lang, page:wordCount, html:time (the first <time> element) and text:pattern (the first email or URL in the text) are fallbacks: returned so you can judge them, never charged. Dates are never guessed from free text.

Limits

• 25 URLs per run. Split longer lists across runs, or schedule them.
• Very large pages take seconds each. A typical article converts in about a second; a 3MB page such as a long Wikipedia country article takes several, and a run of 25 of those took about 3.6 minutes in our measurements. The synchronous API call (run-sync-get-dataset-items) waits at most 300 seconds, so send a long list of very large pages as a normal run and read the dataset when it finishes.
• Each call is a normal Actor run, not an always-on endpoint. A warm start takes 2 to 3 seconds. When a server first has to download the Actor, start-up took 11 to 24 seconds in our measurements.
• No proxy. Requests leave from datacenter IPs. Sites behind bot protection usually answer 403, which comes back as a free http_error record. Rendering does not change that.
• Public pages only. No login, cookies or sessions.
• HTML and plain text only. PDFs, images, feeds and archives return unsupported_content_type, free.
• 12MB per page. Larger responses, and the rare page too large to convert in the run's memory, return unsupported_content_type, free.
• robots.txt is respected by default, fetched once per host and cached for the run. A redirect to another host is checked against that host's robots.txt before anything is charged.
• Private and loopback addresses are refused. The URL, every redirect hop and every address a host name resolves to are checked, and the connection is made to the address that was checked, so a host name cannot be switched to a private address between the check and the connection. This covers plain HTTP and the headless browser, whose every connection goes through the same address check. It covers the addresses this Actor connects to; it does not cover what a public site chooses to show.
• Markdown is not a pixel-perfect copy. Headings, lists, tables, code blocks and links survive. Complex layouts and canvas content do not.

FAQ

Why is my Markdown nearly empty? Most JavaScript-only pages are loaded in the browser automatically (autoRendered: true). If a page still comes back thin, set render: true to load every page in the browser.

Can it get past Cloudflare or other bot protection? No. It uses no proxy or unblocker. Those pages come back as http_error and are not charged.

Can it crawl a whole site? No. It reads exactly the URLs you give it. The links array lets you pick the next URLs yourself, or use a site crawler such as Website Content Crawler.

Does the schema use an LLM? No. It reads what the page publishes about itself and parses its structure. It is deterministic, needs no API key, and leaves a field out when the page does not state it. For fields a page never states, add an LLM step after this one.

Why did I get blocked_by_robots on a page I can open in a browser? robots.txt speaks to automated clients. If you own the site or are licensed to crawl it, set respectRobotsTxt: false.

Can I get the raw HTML? No. It returns Markdown and metadata only.

Can I run it on a schedule? Yes. Use Apify schedules, or call it from your own job through the API or an MCP client.