Website Content to Markdown — LLM-ready text extractor

Convert any list of web page URLs into clean Markdown and structured JSON that is ready to feed into an LLM, a RAG pipeline, or an AI agent. One record per URL: title, main-content Markdown, plain text, links, metadata, word count, and a per-page status. Navigation, ads, cookie banners and footers are stripped using Mozilla Readability (the engine behind Firefox Reader View).

What you get

Each URL produces exactly one dataset record:

Field	Type	Description
url	string	The URL you passed in
finalUrl	string | null	URL after redirects, when different
title	string | null	Article / page title
markdown	string | null	Main content as Markdown (ATX headings, fenced code blocks, pipe tables). null when outputFormat is json
text	string | null	Main content as plain text with paragraph breaks. null when outputFormat is markdown
links	{ text, href }[]	Absolute http(s) links found in the main content (empty when includeLinks is false)
metadata	object	description, author, publishedAt (ISO 8601), lang, canonical, siteName, image — from meta tags, Open Graph and JSON-LD
wordCount	integer	Whitespace-separated words in text (for CJK text this is closer to a sentence count)
fetchedAt	string	ISO 8601 timestamp of the fetch
status	ok | failed | skipped	failed = fetch/extraction error, skipped = disallowed by robots.txt or spending limit reached
error	string	Present only when status is not ok

The run always finishes as SUCCEEDED, even if some URLs fail. Check status per record instead of the run status, so one dead link never breaks your pipeline.

Input

Field	Type	Default	Description
urls	string[]	required	Absolute http(s) URLs. Pass many at once; each page is a separate billable event
outputFormat	markdown | json | both	both	markdown returns only the markdown field; json returns text + structured fields without Markdown; both returns everything
renderJs	boolean	false	Load the page in headless Chromium before extraction. Use for single-page apps or content injected by JavaScript. Slower and billed at the higher rate
includeLinks	boolean	true	Collect links from the main content
includeMetadata	boolean	true	Extract page metadata
respectRobotsTxt	boolean	true	Skip URLs disallowed by the site's robots.txt
maxConcurrency	integer	5	Upper bound on parallel fetches (1–20). Same-host requests are additionally spaced out
requestTimeoutSecs	integer	30	Per-page load timeout (5–180)

Example input:

{
  "urls": [
    "https://example.com",
    "https://blog.apify.com/fixing-actors-invisible-to-agents/"
  ],
  "outputFormat": "both",
  "renderJs": false
}

Output example

{
  "url": "https://blog.apify.com/fixing-actors-invisible-to-agents/",
  "finalUrl": null,
  "title": "What Apify MCP does to your Actor before an AI agent sees it",
  "markdown": "This article was written by [Olivier Reynaud](https://apify.com/silentflow) as part of [Write for Apify](https://apify.com/resources/write-for-apify) ...\n\n## The MCP server rewrites your Actor\n\n...",
  "text": "This article was written by Olivier Reynaud as part of Write for Apify ...",
  "links": [
    { "text": "Olivier Reynaud", "href": "https://apify.com/silentflow" },
    { "text": "Apify Store", "href": "https://apify.com/store" }
  ],
  "metadata": {
    "description": "Olivier Reynaud audited his Apify Actors through the Apify MCP server to see how AI agents find and read them, and what developers should fix.",
    "author": "Olivier Reynaud",
    "publishedAt": "2026-09-24T13:25:20.000Z",
    "lang": "en",
    "canonical": "https://blog.apify.com/fixing-actors-invisible-to-agents/",
    "siteName": "Apify Blog",
    "image": "https://storage.ghost.io/.../i-audited-151-actors-for-ai-agents.png"
  },
  "wordCount": 3641,
  "fetchedAt": "2026-10-15T01:58:12.345Z",
  "status": "ok"
}

A failed URL looks like this and is not charged:

{
  "url": "https://this-domain-does-not-exist.example/",
  "status": "failed",
  "error": "getaddrinfo ENOTFOUND this-domain-does-not-exist.example",
  "markdown": null,
  "text": null,
  "links": [],
  "wordCount": 0,
  "fetchedAt": "2026-10-15T01:58:12.345Z"
}

Pricing

Pay per event. You are charged only for pages that return status: "ok". Failed and skipped pages are free.

Event	Price	When
page-processed	$0.001 per page	renderJs: false (default, plain HTTP fetch)
page-processed-js	$0.003 per page	renderJs: true (headless Chromium rendering)

Example: converting 1,000 static pages costs $1.00; 1,000 JavaScript-rendered pages cost $3.00. You can cap spending per run with the Maximum total charge setting in Apify Console or the maxTotalChargeUsd API parameter; when the cap is reached, remaining URLs are returned with status: "skipped".

Usage

Apify Console

Open the Actor, paste your URLs into Page URLs, click Start, and download the dataset as JSON, CSV or Excel.

JavaScript (apify-client)

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });

const run = await client.actor('fre.d/url-to-markdown').call({
  urls: ['https://example.com', 'https://blog.apify.com/fixing-actors-invisible-to-agents/'],
  outputFormat: 'markdown',
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const page of items) {
  if (page.status === 'ok') console.log(page.title, page.markdown.length);
  else console.warn(page.url, page.error);
}

Python (apify-client)

from apify_client import ApifyClient

client = ApifyClient(token=os.environ["APIFY_TOKEN"])

run = client.actor("fre.d/url-to-markdown").call(run_input={
    "urls": ["https://example.com", "https://blog.apify.com/fixing-actors-invisible-to-agents/"],
    "outputFormat": "both",
})

for page in client.dataset(run["defaultDatasetId"]).iterate_items():
    if page["status"] == "ok":
        print(page["title"], page["wordCount"])
    else:
        print("failed:", page["url"], page["error"])

AI agents via MCP

Add the Apify MCP server to your agent (Claude Desktop, Cursor, LangGraph, etc.) and allow this Actor:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com/?tools=fre.d/url-to-markdown",
      "headers": { "Authorization": "Bearer <APIFY_TOKEN>" }
    }
  }
}

The agent can then call the tool with {"urls": ["https://..."]} and read markdown or text directly from the result. Because urls is an array, an agent can fetch a whole reading list in one tool call.

HTTP API

curl -X POST "https://api.apify.com/v2/acts/fredman~url-to-markdown/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://example.com"], "outputFormat": "markdown"}'

How it works

1. URLs are validated, de-duplicated, and checked against the site's robots.txt (fetched once per host).
2. Pages are fetched over HTTP with Crawlee's CheerioCrawler, or rendered in headless Chromium when renderJs is on. Requests to the same host are spaced out and total concurrency is capped.
3. The main content is identified with Mozilla Readability. If Readability finds nothing, the Actor falls back to <main>, <article> or <body> with navigation and scripts removed.
4. The content is converted to Markdown with Turndown (fenced code blocks keep their language, simple tables become pipe tables) and to plain text with paragraph breaks.
5. Metadata is merged from <meta> tags, Open Graph, <link rel="canonical">, <time datetime>, and JSON-LD (Article, BlogPosting, ...).

The Actor identifies itself with the User-Agent Mozilla/5.0 (compatible; WebsiteContentToMarkdown/0.1; +https://apify.com/fre.d/url-to-markdown). Site owners can block it with a User-agent: WebsiteContentToMarkdown rule in robots.txt.

Limitations

• Only HTML pages are supported. PDFs, images and other binary resources return status: "failed".
• Pages behind a login, paywall or CAPTCHA cannot be extracted.
• Readability is tuned for article-like pages. For product listings, search results or dashboards, consider a dedicated scraper.
• wordCount counts whitespace-separated tokens; for Japanese, Chinese or Korean text it will be much lower than the character count.

robots.txt and responsible use

By default the Actor respects robots.txt and skips disallowed URLs. Turning respectRobotsTxt off is intended for sites you own or have permission to crawl.

The extracted text remains the property of the original publisher. You are responsible for how you store, redistribute or otherwise use the content you retrieve with this Actor, including compliance with the target site's terms of service and applicable copyright and data-protection law. Fetching a page for personal analysis or feeding it to an LLM as context is a different matter from republishing it; when in doubt, link to the source instead of redistributing the text.

Support

Open an issue on the Actor's Issues tab in Apify Store, or email the maintainer listed on the Actor page. Feature requests (custom selectors, PDF support, chunking for embeddings) are welcome.