Agent Fetch: URL to Clean Markdown

Give it a URL and get back the page as clean markdown an LLM can read straight away. No navigation, no cookie bar, no sidebar, no footer, no script tags. Just the main content, plus the title, the canonical URL, the language, a word count and the links that live inside the text.

Up to 25 URLs per run. Plain HTTP by default, a headless browser when a page needs one, and an optional JSON Schema that pulls named fields out of the page without calling any LLM.

Who this is for

AI agents that need to read a web page mid-task. An agent holds a URL and needs the text, not the HTML. Feeding it raw HTML burns thousands of tokens on markup and menus before it reaches a sentence worth reading. This Actor is the step in between: one call, one flat record, markdown that drops straight into a prompt.

It is also for the plumbing around that:

• RAG ingestion. Turn a list of documentation pages into markdown chunks with titles, canonical URLs and word counts already attached.
• Research pipelines. Pull ten sources into one dataset and summarise them in a single pass.
• Link and content monitoring. Track what a page says and which pages it points at over time.
• Structured scraping without an LLM bill. Hand it a JSON Schema and get author, published date and description back from the page's own JSON-LD, deterministically.

What you get per URL

Field	Type	Always present	Notes
url	string	yes	exactly the URL you asked for
finalUrl	string	yes	after redirects
status	string	yes	see the status table below
httpStatus	integer	yes	0 when no response was received
method	string	yes	http or browser
fetchedAt	string	yes	ISO 8601
title	string	yes	from the readability pass, falling back to the page title
canonicalUrl	string	yes	rel=canonical, falling back to og:url
language	string	yes	from html lang, falling back to og:locale
markdown	string	yes	the main content
wordCount	integer	yes	words in the markdown
charCount	integer	yes	characters in the markdown, after any truncation
linkCount	integer	yes	links inside the extracted content
truncated	boolean	yes	true when maxMarkdownChars cut the text
durationMs	integer	yes	time spent on this URL
reason	string	on failures only	why this URL did not produce content
links	array	when includeLinks	{ url, text, external }
article	object	when includeArticleJson	excerpt, byline, site name, dates, image, JSON-LD
extracted	object	when a schema found something	your fields, coerced to your declared types
extractedSources	object	with extracted	where each field came from
extractedFieldCount	integer	when a schema was given	how many fields were found

Optional keys are left out rather than set to null, so the dataset never grows a column that is empty in every row.

Status values

status	Meaning	Charged
ok	content extracted	yes
blocked_by_robots	robots.txt disallows this path	no
http_error	server answered 4xx or 5xx	no
timeout	no response inside timeoutSecs	no
fetch_error	DNS, TLS or connection failure	no
invalid_url	not http or https, or a private address	no
unsupported_content_type	a PDF, image, archive or other binary	no
render_unavailable	render was on and the browser would not start	no
empty_content	fetched fine, held no readable text	no

A bad URL is a record, never a crashed run. One dead link out of twenty-five costs you nothing and does not touch the other twenty-four.

Input

Minimum:

{
  "urls": ["https://en.wikipedia.org/wiki/Markdown"]
}

Everything:

{
  "urls": [
    "https://en.wikipedia.org/wiki/Markdown",
    "https://docs.python.org/3/tutorial/introduction.html"
  ],
  "render": false,
  "respectRobotsTxt": true,
  "timeoutSecs": 30,
  "includeLinks": true,
  "includeArticleJson": true,
  "maxMarkdownChars": 120000,
  "schema": {
    "properties": {
      "author": { "type": "string" },
      "datePublished": { "type": "string", "format": "date" },
      "siteName": { "type": "string" }
    }
  }
}

Field	Type	Default	Notes
urls	array of strings		required, 1 to 25, http or https only
render	boolean	false	load every URL in a headless Chromium instead of a plain fetch
respectRobotsTxt	boolean	true	a disallowed path returns blocked_by_robots and is not charged
timeoutSecs	integer 5 to 120	30	per URL
userAgent	string	AgentFetchBot/1.0	also used to match robots.txt groups
includeLinks	boolean	true	adds the links array
includeArticleJson	boolean	false	adds the article object
maxMarkdownChars	integer 500 to 500000	120000	cut at a paragraph boundary
schema	object		JSON Schema with a properties map

Conditional behaviour, all of it stated on the input form too:

• render: true charges a render event per page instead of a fetch event, and runs pages one at a time instead of four at a time.
• schema is only acted on when it is an object with a properties map. Give it one and a page that yields at least one field also charges an extract event.
• includeLinks and includeArticleJson only add keys to records whose status is ok.
• urls over 25, a non-http scheme, or a private or loopback address is rejected as a caller mistake before any request goes out.

Output

Real records from a run on 5 September 2026.

A documentation page. The markdown value below is the verbatim first 559 characters of an 18183 character field:

{
  "url": "https://docs.python.org/3/tutorial/introduction.html",
  "finalUrl": "https://docs.python.org/3/tutorial/introduction.html",
  "status": "ok",
  "httpStatus": 200,
  "method": "http",
  "fetchedAt": "2026-09-04T22:41:35.655Z",
  "title": "3. An Informal Introduction to Python",
  "canonicalUrl": "https://docs.python.org/3/tutorial/introduction.html",
  "language": "en",
  "wordCount": 2834,
  "charCount": 18183,
  "linkCount": 19,
  "truncated": false,
  "durationMs": 1121,
  "markdown": "In the following examples, input and output are distinguished by the presence or absence of prompts ([\\>>>](https://docs.python.org/3/glossary.html#term-0) and [\u2026](https://docs.python.org/3/glossary.html#term-...)): to repeat the example, you must type everything after the prompt, when the prompt appears; lines that do not begin with a prompt are output from the interpreter. Note that a secondary prompt on a line by itself in an example means you must type a blank line; this is used to end a multi-line command.\n\nYou can use the \u201cCopy\u201d button (it appears",
  "links": [
    {
      "url": "https://docs.python.org/3/glossary.html#term-0",
      "text": ">>>",
      "external": false
    },
    {
      "url": "https://docs.python.org/3/library/functions.html#int",
      "text": "int",
      "external": false
    }
  ]
}

A Wikipedia article from the same run, metadata only. Its markdown field holds 40740 characters of converted article text, including the infobox rendered as a table:

{
  "url": "https://en.wikipedia.org/wiki/Markdown",
  "finalUrl": "https://en.wikipedia.org/wiki/Markdown",
  "status": "ok",
  "httpStatus": 200,
  "method": "http",
  "fetchedAt": "2026-09-04T22:41:35.638Z",
  "title": "Markdown",
  "canonicalUrl": "https://en.wikipedia.org/wiki/Markdown",
  "language": "en",
  "wordCount": 5334,
  "charCount": 40740,
  "linkCount": 204,
  "truncated": false,
  "durationMs": 1793
}

A URL that robots.txt disallows:

{
  "url": "https://docs.python.org/2.7/index.html",
  "status": "blocked_by_robots",
  "httpStatus": 0,
  "method": "http",
  "markdown": "",
  "wordCount": 0,
  "reason": "robots.txt at https://docs.python.org/robots.txt disallows /2.7/index.html for this user agent."
}

With a schema, on a news article that publishes JSON-LD:

{
  "extractedFieldCount": 6,
  "extracted": {
    "headline": "Toxic Gaslighting: How 3M Executives Convinced a Scientist the Forever Chemicals She Found in Human Blood Were Safe",
    "author": "Sharon Lerner, Haruka Sakaguchi",
    "datePublished": "2024-05-20T10:00:00+00:00",
    "siteName": "ProPublica",
    "wordCount": 8169
  },
  "extractedSources": {
    "headline": "json-ld",
    "author": "json-ld",
    "datePublished": "json-ld",
    "siteName": "json-ld",
    "wordCount": "json-ld"
  }
}

extractedSources is there so you can tell a value the page published from a value a heuristic guessed at. json-ld and meta are the page's own machine-readable claims. page:pairs, text:label and text:pattern come from parsing what a human would read.

The run also writes a summary to the key-value store under OUTPUT:

{
  "ok": true,
  "requested": 3,
  "returned": 3,
  "succeeded": 3,
  "byStatus": { "ok": 3 },
  "charged": { "fetch": 3, "render": 0, "extract": 0 },
  "totalWords": 16535,
  "tookMs": 1799
}

The examples/ folder of the source holds two full sample datasets: every failure mode in one run, and the same JavaScript-rendered page with render off and on side by side. Full markdown output is on the dataset of any run you start.

How the extraction works

1. Fetch the page, over plain HTTP or in Chromium.
2. Harvest the metadata the page publishes: JSON-LD, meta and microdata tags, definition lists, two-column table rows, and the visible text. This happens before step 3, because the readability pass rewrites the document.
3. Run a readability pass to find the main content and drop the furniture. If it declines the page, fall back to the body with navigation, headers, footers, sidebars and forms removed.
4. Convert that HTML to markdown, with GitHub-flavoured tables, fenced code blocks and inline links.
5. If you supplied a schema, match each property against the harvest in this order: JSON-LD, then meta and microdata, then structural label and value pairs, then labelled lines of text, then a format-driven pattern. Coerce to the declared type, check any enum, and drop anything that does not fit.

JSON-LD @id references are resolved against the rest of the page's graph, so "author": {"@id": "..."} comes back as the person's name rather than an internal URL.

Pay per event

Pricing is not set yet. This Actor is published with pay-per-event billing wired up and no rates configured, so nothing is being charged today. When rates are published they will appear on the Pricing tab, and these are the events they will apply to:

Event	Fires when
fetch	one page was retrieved over plain HTTP and converted to markdown
render	one page was loaded in the headless browser and converted to markdown
extract	one page produced at least one field from your schema

A page charges fetch or render, never both. extract is on top, and only when the schema actually found something.

Nothing is charged for a URL that was blocked by robots.txt, timed out, answered with an error status, was refused by the allowlist, or held no readable text. You pay for content, not for attempts.

Limits and things worth knowing

• 25 URLs per run. Above that the run stops with a message telling you to split the list. Chain runs, or schedule them.
• Datacenter egress only, no proxy. This Actor never asks for a residential IP. A site that blocks datacenter traffic will answer 403 and you get an http_error record. If you need residential IPs, this is not the tool.
• robots.txt is fetched once per host and cached for the run. Missing, empty or unreachable means allowed, which is what the standard says. The user agent token matched against robots.txt groups is AgentFetchBot, matched exactly, not by substring.
• Redirects are re-checked. If a URL redirects to a different origin, that origin's robots.txt is consulted before anything is extracted or charged.
• Private and loopback addresses are refused. localhost, 127.x, 10.x, 192.168.x, 172.16-31.x, 169.254.x, IPv6 loopback and unique-local, .local and .internal all return invalid_url without a request being made.
• 12MB page ceiling. Bigger responses return unsupported_content_type.
• links covers the extracted content, not the whole page. Site navigation and footer link farms are deliberately excluded, and anchors pointing back into the same page are not counted. The list is capped at 500 per page.
• render is slower and heavier. Rendered pages run one at a time and load with images, fonts and media blocked. Leave it off unless a plain fetch comes back nearly empty.
• Markdown is not a pixel-perfect copy. Tables, code blocks, headings, lists and links survive. Complex multi-column layouts, canvas, and content behind a login do not.
• Public pages only. No login, no cookies, no session. What you get is what an anonymous visitor gets.

What it does not do

It does not use an LLM, does not need any API key, does not log in to anything, does not run in Standby mode, does not use residential proxies, does not crawl links it finds, does not read PDFs or images, and does not extract fields the page never stated.

FAQ

Why is my markdown nearly empty? The page probably builds its content with JavaScript. Set render: true and run it again.

Why did I get blocked_by_robots on a page I can open in a browser? robots.txt speaks to automated clients, not to browsers. If you own the site or are licensed to crawl it, set respectRobotsTxt: false.

Does the schema use an LLM? No. It reads what the page publishes about itself and parses its structure. It is deterministic, it is free of API keys, and it returns nothing when the page does not say it. For fields a page never states, use an LLM step after this one.

Can I get the raw HTML back? No. This Actor returns markdown and metadata. If you need the HTML, that is a different tool.

Can I run it on a schedule? Yes. Use Apify's scheduler, or call it from your own job through the API or the MCP server.

Why is it not a Standby HTTP API? Deliberate. Batch runs with pay-per-event billing and limited permissions are the shape agentic-payment rails expect, and Standby would break that shape.

---

Built by Omar Nagy. Part of an agent-native data-tool series on Apify.