Agent Fetch: URL to Clean Markdown
Pricing
from $1.00 / 1,000 page fetcheds
Turn any URL into clean, LLM-ready markdown: main content only, plus title, canonical URL, language, word count and links. Optional headless rendering for JavaScript pages, and deterministic JSON Schema field extraction with no LLM and no API key. robots.txt respected, failures come back labelled.
The pages to convert, up to 25 per run. Only http:// and https:// are accepted; anything else is rejected before a single request is made. Duplicates are removed and the fragment (#section) is dropped, so the same page is never fetched or charged twice.
[ "https://en.wikipedia.org/wiki/Markdown"]Off by default. Leave it off for normal pages: a plain HTTP fetch is faster and cheaper. Turn it on only when a page builds its content with JavaScript and the plain fetch comes back nearly empty. When on, every URL in the run is loaded in the Chromium that ships inside this Actor's image, and each page is charged as a render event instead of a fetch event. No proxy is used in either path.
On by default. The robots.txt of each host is fetched once per run and cached. A path that is disallowed for this Actor's user agent comes back as a record with status blocked_by_robots and is not charged. If robots.txt is missing, empty or unreachable, the URL is treated as allowed, which is what the standard says. Switch this off only for hosts you own or are licensed to crawl.
How long a single page may take before it is given up on. A URL that runs out of time becomes a record with status timeout and is not charged; the rest of the run continues.
Sent with every request and used when matching robots.txt groups. Leave it as it is unless a site has asked you to identify differently.
On by default. Adds a links array to each successful record: the absolute URL, the anchor text and whether the link points off the page's own host. linkCount is always returned even when this is off.
Off by default. Adds an article object holding the page metadata that the readability pass and the page head expose: excerpt, byline, site name, published and modified time, description, lead image, and any JSON-LD blocks found in the page. Only the keys that actually have a value are included.
Markdown longer than this is cut at a paragraph boundary and a truncation marker is appended. charCount always reports the length after truncation. Raise it for long reference pages, lower it to keep records inside a small context window.
Optional. A JSON Schema object with a properties map, for example {"properties":{"author":{"type":"string"},"datePublished":{"type":"string","format":"date"}}}. Each property is looked for in the page's JSON-LD, its meta and microdata tags, its definition lists and tables, and finally in labelled lines of the page text, then coerced to the declared type. This is deterministic parsing, not an LLM: no model is called, no API key is needed, and a field that cannot be found honestly is left out rather than guessed. When a schema is given and at least one field is found, the record gains an extracted object and an extract event is charged. Leave it empty to skip extraction entirely.
{ "properties": { "author": { "type": "string" }, "datePublished": { "type": "string", "format": "date" }, "description": { "type": "string" } }}