Web Fetch - URL to Markdown for LLM, RAG & AI Agents

Feeding web pages to ChatGPT, Claude or a RAG pipeline is expensive when you paste raw HTML: menus, cookie banners, ads and scripts burn tokens and confuse the model. This actor takes a list of URLs and returns the main content as clean Markdown, plus title, meta description, language, author, date, headings, plain text and word count. Cheap, fast, and one call per page.

It uses plain HTTP and a content extractor (trafilatura, with a BeautifulSoup fallback). No browser, no JavaScript rendering, no proxies. It sends an honest RadarFetch/1.0 User-Agent and respects robots.txt.

What you get

• markdown: main article content with headings, lists, tables, code blocks and links. Boilerplate removed.
• text: the same content as plain text.
• title, description, language, author, publishedDate, siteName when the page provides them.
• headings: list of {level, text} for chunking and outlines.
• wordCount, truncated, statusCode, finalUrl (after redirects), fetchedAt.
• Optional internalLinks / externalLinks and images (URL + alt text).
• Optional same-site crawl and only-new mode (pay only for new or changed pages).

Use cases

• Give an AI agent or chatbot the content of any public page ("read this URL").
• Build a RAG knowledge base from documentation, blogs or help centers.
• Summarise articles, competitor pages and news for newsletters.
• Monitor pages and get only the ones whose content changed.
• Cut LLM cost: use maxCharsPerPage to fit a context window.

How to use it

1. Open the actor and add your URLs (one per line).
2. Optional: turn on Follow links on the same site and set Max pages and Max crawl depth.
3. Optional: set Max characters per page to truncate long pages.
4. Click Start and download the dataset as JSON, CSV or Excel, or call it from code.

API example (curl)

Replace YOUR_APIFY_TOKEN with your token (Apify Console, Settings, Integrations). This runs the actor and returns the pages in one request:

curl -X POST "https://api.apify.com/v2/acts/martin10291~url-to-markdown-llm/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": ["https://en.wikipedia.org/wiki/Web_scraping"], "maxCharsPerPage": 8000}'

MCP example (Claude, Cursor, any MCP client)

Apify exposes every actor as an MCP tool. Add the actor to the Apify MCP server:

https://mcp.apify.com?tools=martin10291/url-to-markdown-llm

In Claude Desktop or Cursor add it as a remote MCP server (OAuth login to Apify, or Authorization: Bearer YOUR_APIFY_TOKEN). Then ask: "Read https://example.com/docs and summarise it", and the agent calls this actor with startUrls.

Input

Field	Type	Description
startUrls	array	Pages to convert (http or https, public pages).
maxPages	integer	Hard limit on pages fetched per run (default 20).
crawl	boolean	Follow same-domain links from each start URL (default false).
maxDepth	integer	Crawl depth, 1 = start pages plus pages they link to.
includeLinks	boolean	Add internalLinks and externalLinks.
includeImages	boolean	Add images with URL and alt text.
maxCharsPerPage	integer	Cut markdown and text at this length (0 = no limit).
onlyNew	boolean	Return only pages that are new or changed since the last run.
monitorId	string	Separate memory for different watch lists.
includeFailed	boolean	Also save failed pages with an error (charged like normal results).
timeoutSecs	integer	Per-page timeout (default 20).

Example input

{
  "startUrls": ["https://en.wikipedia.org/wiki/Web_scraping", "https://blog.apify.com/introducing-x402-agentic-payments/"],
  "maxCharsPerPage": 20000,
  "includeLinks": false
}

Output example

{
  "url": "https://blog.apify.com/introducing-x402-agentic-payments/",
  "finalUrl": "https://blog.apify.com/introducing-x402-agentic-payments/",
  "statusCode": 200,
  "fetchedAt": "2026-11-01T22:09:29Z",
  "error": null,
  "title": "Your AI agent just got 20,000+ new tools on x402",
  "description": "...",
  "language": "en",
  "author": "...",
  "publishedDate": "2026-06-30",
  "siteName": "Apify Blog",
  "markdown": "The value of AI agents is proportional to the size of tasks ...",
  "text": "The value of AI agents is proportional to the size of tasks ...",
  "wordCount": 1659,
  "headings": [{"level": 2, "text": "..."}],
  "truncated": false,
  "extractor": "trafilatura"
}

A RUN_SUMMARY record in the key-value store lists counts (fetched, saved, unchanged, failed, blocked by robots.txt), the failed URLs and lastError.

Pricing

Pay per result: $1.00 per 1,000 pages (one result = one page saved to the dataset). Pages that fail are not charged unless you turn on includeFailed. In only-new mode unchanged pages are not charged. maxPages caps the cost of a run.

FAQ

Does it respect robots.txt? Yes. Pages disallowed for our User-Agent are skipped with the error blocked by robots.txt.

Does it run JavaScript? No. It reads the HTML the server returns. Pages that build their content only in the browser (some single-page apps) can return little or no text; the result then has an error like "no main content found". This is not a headless browser.

Can it read pages behind a login, paywall or CAPTCHA? No. Public pages only. It does not use proxies or try to bypass bot protection, so sites that block automated traffic will return an HTTP error.

What about PDFs and images? Only HTML and plain-text pages are converted.

How polite is it? At most 5 pages in parallel and at least 1 second between requests to the same domain.

How does only-new mode work? The actor stores a hash of each page content under your Monitor ID and returns a page only when the hash changes. Combine it with an Apify Schedule for change alerts.

Is the Markdown always perfect? Extraction is automatic and works best on articles, docs and blogs. Listing pages, dashboards and unusual layouts may be incomplete.

Use it as an integration (chain it after another actor)

This actor can run automatically after any actor whose results contain a URL.

1. Open the source actor in Apify Console and go to the Integrations tab.
2. Click Add integration -> Actor, pick this actor and set Dataset to {{resource.defaultDatasetId}} (leave startUrls empty). Set your other options and save.
3. When the source run finishes, this actor reads its dataset and processes every row. The field is detected automatically; set Dataset field if your column has another name.

You can also start it by hand with datasetId (pick a dataset or paste its ID) to process an existing dataset.

Good sources to chain from:

• apify/website-content-crawler
• martin10291/medium-articles-scraper
• martin10291/github-trending-scraper
• martin10291/reddit-posts-comments-extractor (link)
• any actor with a url column (news, blogs, docs)