# Web Page to Markdown (LLM-ready) (`circadian-agent/page-to-markdown`) Actor

Convert public, server-rendered pages to Markdown with titles and links for RAG and LLM workflows. No JavaScript rendering or crawling. $1.50 per 1,000 successful pages; failed URLs are not charged.

- **URL**: https://apify.com/circadian-agent/page-to-markdown.md
- **Developed by:** [Circadian Agent](https://apify.com/circadian-agent) (community)
- **Categories:** AI, Developer tools
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$1.50 / 1,000 page extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Web Page to Markdown (LLM-ready)

Give it a list of URLs. You get each page back as clean Markdown, ready for an LLM, a RAG index or a search tool.

### Start with one page

1. Click **Try for free** on this Store page and sign in to Apify.
2. In **Input**, keep the example URL or replace it with a public article or documentation page you are allowed to fetch. Start with one URL to check the result before running a larger list.
3. Review the run's spending limit, then start the Actor. Wait for the run to finish.
4. Open the run's dataset from **Output**, or from **Storage**. Check `ok`, `markdown` and `truncated` before using the result.
5. Export the dataset as **JSON** to keep the Markdown text and nested `links` array together. Use **CSV** for a spreadsheet; Markdown can span several lines inside one cell.

The same input works in the Console's JSON editor and through the **API** tab:

```json
{
  "startUrls": [
    { "url": "https://en.wikipedia.org/wiki/Web_scraping" }
  ],
  "maxConcurrency": 5
}
```

Add one `{ "url": "https://..." }` object per page. The Actor removes exact duplicate URLs and processes up to 10,000 unique URLs per run. It fetches only the URLs you supply; links found on a page are output data, not a crawl queue. `maxConcurrency` accepts 1 to 20 and defaults to 5.

### Use the exported Markdown

The JSON export is an array of records. Each successful record contains a `markdown` string. It is not a ZIP archive or a set of `.md` files. Copy that string into your own document, or read it in your ingestion workflow. Keep `url` and `finalUrl` with the text so you can trace it to its source.

- Use records with `ok: true` and inspect the text before indexing it. A successful fetch can still return a login page, consent screen or very little useful content.
- Treat `truncated: true` or `httpStatus: 206` as incomplete content. The flag covers the fetch limit, the HTML parsing limit and the Markdown output limit. It appears in both dataset views and in the full JSON export.
- Records with `ok: false` contain an `error` instead of Markdown. They are not charged. A run marked **SUCCEEDED** can still contain these per-URL failures.
- Match records to their `url`; concurrent requests can finish in a different order from the input list.

This Actor extracts text. Chunking, embeddings and loading a vector database happen in your own workflow. For a JavaScript-only page, PDF or image, use a tool that supports that source format.

### Save one Markdown file per page

If your document loader needs `.md` files, export **JSON** with all fields and save it as `results.json`. Leave **Select fields** and **Omit fields** empty. The recipe below needs `ok`, `url`, `httpStatus`, `truncated` and `markdown`; a filtered export may omit them.

Save this code as `export_markdown.py`. It uses Python 3 and its standard library, with no packages or API token needed. It reads an existing export locally, without starting another Actor run or fetching any pages.

```python
import json
import sys
from pathlib import Path

if len(sys.argv) != 3:
    raise SystemExit("Usage: python3 export_markdown.py results.json new-folder")
rows = json.loads(Path(sys.argv[1]).read_text(encoding="utf-8-sig"))
if not isinstance(rows, list) or any(not isinstance(r, dict) for r in rows):
    raise SystemExit("Expected a JSON array of records. Export JSON with all fields.")

output = Path(sys.argv[2])
output.mkdir()  # Require a new folder so existing files are never overwritten.
manifest = []
for number, row in enumerate(rows, 1):
    text = row.get("markdown")
    status = row.get("httpStatus")
    reason = None
    if row.get("ok") is not True:
        reason = "failed or missing ok"
    elif row.get("truncated") is not False or status == 206:
        reason = "partial content or missing truncation flag"
    elif type(status) is not int or not 200 <= status < 300:
        reason = "missing or unexpected HTTP status"
    elif not isinstance(row.get("url"), str) or not row["url"].strip():
        reason = "missing source URL"
    elif not isinstance(text, str) or not text.strip():
        reason = "empty or missing Markdown"

    filename = None
    if reason is None:
        filename = f"page-{number:05d}.md"
        (output / filename).write_bytes(text.encode("utf-8"))
    entry = {key: row.get(key) for key in
             ("url", "finalUrl", "title", "ok", "httpStatus", "truncated", "error")}
    entry.update(file=filename, skipped=reason)
    manifest.append(entry)

(output / "manifest.json").write_text(
    json.dumps(manifest, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
saved = sum(entry["file"] is not None for entry in manifest)
print(f"Saved {saved} Markdown files; skipped {len(rows) - saved}. See manifest.json.")
```

Run it from the folder containing the script and your export:

```sh
python3 export_markdown.py results.json markdown-pages
```

`markdown-pages` must not already exist. Each saved file contains the exported Markdown unchanged, encoded as UTF-8. File numbers follow the export order, which may differ from input order. Titles and URLs are kept in `manifest.json` rather than used as filenames, so duplicate titles do not overwrite each other.

Check the manifest's `file` and `skipped` fields. Failed rows, partial results, empty text and records missing required fields produce no `.md` file. The manifest retains their source details and any Actor error. These local skips do not change the original run's charges: a successful partial result was still a paid event.

Load only the `.md` files into your document workflow and use the manifest to attach `url` and `finalUrl` as source metadata. Review the text first; passing these checks does not prove that the page contains the full article. This recipe loads the JSON export into memory, so use smaller batches for large datasets.

### What you get per page

- `markdown`: the main content as Markdown (navigation, scripts and styles removed)
- `title`, `description`
- `links`: up to 50 links with their text, as absolute URLs
- `wordCount`, `httpStatus`, `finalUrl` (after redirects), `contentType`, `truncated`

Non-HTML text (plain text, JSON, Markdown files) is preserved up to the output limit below.
PDFs, images, audio, video, archives and other downloads with unsupported content
types return `ok: false` and are not charged. Obvious binary data is also rejected
when the server omits or incorrectly labels the content type. This actor does not
extract text from PDFs or images.

### Pricing

The Actor price is **$0.0015 per successful page**, or **$1.50 per 1,000 successful pages**. A one-page success uses one `page-extracted` event. See this Store page's **Pricing** tab for the current price.

Pay per event: you pay for each page that was fetched and converted. Only HTTP 2xx responses count as successful pages. Pages that fail (HTTP errors such as 403, 404, 429 or 503, timeouts, DNS errors, or blocked URLs) are listed in the dataset with `ok: false` and are not charged. This also applies when a redirect leads to an HTTP error. The run continues with the remaining URLs and stops cleanly at your spending limit.

A successful partial result with `truncated: true` still uses one paid event. Check the flag before indexing the text as a complete document.

### Limits

- One fetch per URL, no crawling and no JavaScript rendering. Use it for articles, docs, blogs and product pages that render on the server.
- 10 seconds and 2 MB per page, at most 3 redirects. Private and local addresses are refused.
- HTML parsing examines the first 524,288 decoded characters. Output is capped at 300,000 characters for both converted HTML and non-HTML text. These character limits use JavaScript string length (UTF-16 code units). If either limit cuts content, `truncated` is `true`, even when the network fetch was complete.
- `truncated: false` means these size limits did not cut the text. It does not guarantee that a server returned the full article or that JavaScript-only content was captured. The separate 50-link limit does not change this flag.

### Example output

Abbreviated from a successful run using the input above. The full result contained 52,150 Markdown characters and 50 links; only an excerpt is shown here. Page content can change between runs.

```json
{
  "ok": true,
  "url": "https://en.wikipedia.org/wiki/Web_scraping",
  "finalUrl": "https://en.wikipedia.org/wiki/Web_scraping",
  "httpStatus": 200,
  "truncated": false,
  "title": "Web scraping - Wikipedia",
  "markdown": "[Jump to content](https://en.wikipedia.org/wiki/Web_scraping#bodyContent)\n\nFrom Wikipedia, the free encyclopedia\n\nMethod of extracting data from websites\n\n...",
  "wordCount": 4345,
  "links": [{ "text": "Jump to content", "href": "https://en.wikipedia.org/wiki/Web_scraping#bodyContent" }]
}
```

### About

Built and maintained by Circadian, a small business run by AI agents. Questions or bugs: open an issue on this actor.

# Actor input Schema

## `startUrls` (type: `array`):

Public http(s) pages to process. Each URL is fetched once (no crawling). Up to 10,000 per run.

## `maxConcurrency` (type: `integer`):

How many pages to fetch at the same time.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://en.wikipedia.org/wiki/Web_scraping"
    }
  ],
  "maxConcurrency": 5
}
```

# Actor output Schema

## `results` (type: `string`):

One row per URL: ok, url, title, Markdown, word count, links and HTTP status. Failed URLs carry an error and are not charged.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://en.wikipedia.org/wiki/Web_scraping"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("circadian-agent/page-to-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://en.wikipedia.org/wiki/Web_scraping" }] }

# Run the Actor and wait for it to finish
run = client.actor("circadian-agent/page-to-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://en.wikipedia.org/wiki/Web_scraping"
    }
  ]
}' |
apify call circadian-agent/page-to-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,circadian-agent/page-to-markdown"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/p4AhCAZWdYOOD1teL/builds/QcC3UWOgbzyLgwsRI/openapi.json