# Website Pages to Clean Markdown for LLM and RAG (`usta/url-to-markdown`) Actor

Website to markdown for rag: turn public pages or a sitemap into llm ready text, one row per page. Headings, links and tables stay; menus and scripts go. robots.txt is always obeyed.

- **URL**: https://apify.com/usta/url-to-markdown.md
- **Developed by:** [US Tech Automations](https://apify.com/usta) (community)
- **Categories:** Other
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$3.00 / 1,000 result rows

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website Pages to Clean Markdown for LLM and RAG

This tool does website to markdown that is llm ready for rag: paste public page URLs or a sitemap and get one clean row per page.

Each row is the page title, the Markdown, a word count, and the fetch status. Headings, paragraphs, lists, links, code blocks and simple tables stay. Menus, headers, footers, sidebars, forms, scripts and cookie banners are dropped. `robots.txt` is always checked and cannot be turned off.

### Input

| Field | What it does |
|---|---|
| **Page URLs** | One or more public `https` page URLs. You can leave this empty if you set a sitemap. |
| **Sitemap URL** | Optional public `https` sitemap. A urlset is read directly. A sitemap index is read one level down, up to 10 child sitemaps. |
| **Same site only** | On by default. Sitemap links and redirects that leave the site of the URL you gave are not fetched. URLs you typed yourself are still fetched. |
| **Maximum pages** | How many pages to attempt. Default 50. Hard cap 5000. |
| **Maximum rows** | Hard ceiling on rows returned, and therefore on the cost of the run. Default 50. Hard cap 5000. The run stops at the smaller of this and Maximum pages. |
| **Proxy configuration** | Optional Apify proxy. Off by default, including on a local run. |

There is no switch for `robots.txt`. It is always on.

### Pricing

**Pay per result. $0.003 per row. No start fee.**

Every dataset row is one result, including a row that says the page was missing, blocked, disallowed, or not HTML. A run that returns no rows produces nothing to charge. **Maximum rows** is the spend cap. On Apify this package calls the charge event named `result` once per row. It does not call that charge on a local run.

### Output (one row per page)

One row from a real local run on 2026-09-22 against `https://docs.python.org/3/tutorial/index.html`. The `markdown` field here is only the start of that page. `word_count` is the full count from that run: 2593.

```json
{
  "url": "https://docs.python.org/3/tutorial/index.html",
  "final_url": "https://docs.python.org/3/tutorial/index.html",
  "http_status": 200,
  "status": "ok",
  "title": "The Python Tutorial — Python 3.14.7 documentation",
  "markdown": "# The Python Tutorial[¶](https://docs.python.org/3/tutorial/index.html#the-python-tutorial \"Link to this heading\")\n\nTip\n\nThis tutorial is designed for\n*programmers* that are new to the Python language,\n**not** *beginners* who are new to programming.\n\nPython is an easy to learn, powerful programming language. ...",
  "word_count": 2593,
  "fetched_at": "2026-09-22T19:31:23Z",
  "error": null
}
```

| Field | What it holds |
|---|---|
| `url` | The URL requested |
| `final_url` | The URL after redirects, or null if the page was not fetched |
| `http_status` | HTTP status, or null if the page was not fetched |
| `status` | `ok`, `robots_disallowed`, `not_html`, `not_found`, `blocked`, `off_domain`, `empty`, or `error` |
| `title` | The HTML title, or null |
| `markdown` | The page as Markdown, or null when there is no content |
| `word_count` | Words in `markdown`, or null |
| `fetched_at` | When this run handled the URL, in UTC |
| `error` | null on success. A short reason otherwise |

A disallowed URL has `status` `robots_disallowed` and no content. A PDF or image has `status` `not_html` and no content.

### What this does not do

- It does not run a browser, log in, send cookies, or solve a CAPTCHA.
- It does not read text out of PDFs or images. Those come back as `not_html`.
- It does not turn `robots.txt` off. A disallowed path is not fetched.
- It does not follow a redirect onto another site when **Same site only** is on.
- It does not keep the footer. On the Python docs the copyright line sits in that footer, so the Markdown may not include it. If you republish the text, keeping the notice is your job.
- It does not give you a licence to republish anyone's site. You supply the URLs. Each site's terms still apply.
- It does not invent a title or a paragraph when the page could not be read. Unreadable fields are null.
- It does not fetch `http://` URLs, local hosts, or URLs that contain a password.

### Limits

- `maxPages` and `maxItems` are each 1–5000 (default 50). The smaller one wins.
- At least one second between requests to the same host. A long list is slow on purpose. One local page plus its `robots.txt` took 1.11 seconds on 2026-09-22. That is one measurement, not a promise.
- A page or sitemap over 5 MB is refused, not half-converted.
- If `robots.txt` asks for a crawl delay over 60 seconds, that host is not fetched.
- Plain HTTPS GET only.

### Refunds and support

Refunds: if a run returns zero rows for a valid input, email operations@ustechautomations.com within 7 days and we refund that run. Support: same address, replies within 2 business days.

# Actor input Schema

## `startUrls` (type: `array`):

Public https page URLs. One row per page. You can leave this empty if you set a sitemap URL.

## `sitemapUrl` (type: `string`):

Optional public https sitemap (a urlset or one level of sitemap index). Page links are read from it. Leave blank to use only the page URLs above.

## `sameDomainOnly` (type: `boolean`):

On by default. Sitemap links and redirects that leave the site of the URL you gave are not fetched. Page URLs you typed are still fetched even if they are on different sites.

## `maxPages` (type: `integer`):

How many pages to attempt. Default 50. Hard cap 5000. The run stops at the smaller of this and Maximum rows.

## `maxItems` (type: `integer`):

Hard ceiling on rows returned, and therefore on the cost of the run. Default 50. Hard cap 5000. Every row is one result, including a row that says the page was blocked, disallowed, or not HTML.

## `proxyConfiguration` (type: `object`):

Optional Apify proxy. Leave off for a local run. Pages are ordinary HTTPS GET requests. There is no browser.

## Actor input object example

```json
{
  "startUrls": [
    "https://docs.python.org/3/tutorial/index.html"
  ],
  "sitemapUrl": "",
  "sameDomainOnly": true,
  "maxPages": 50,
  "maxItems": 50,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `pages` (type: `string`):

No description

## `csv` (type: `string`):

No description

## `report` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://docs.python.org/3/tutorial/index.html"
    ],
    "sitemapUrl": "",
    "maxPages": 50,
    "maxItems": 50
};

// Run the Actor and wait for it to finish
const run = await client.actor("usta/url-to-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://docs.python.org/3/tutorial/index.html"],
    "sitemapUrl": "",
    "maxPages": 50,
    "maxItems": 50,
}

# Run the Actor and wait for it to finish
run = client.actor("usta/url-to-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://docs.python.org/3/tutorial/index.html"
  ],
  "sitemapUrl": "",
  "maxPages": 50,
  "maxItems": 50
}' |
apify call usta/url-to-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,usta/url-to-markdown"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/wYaWJfAgwwOtsF812/builds/RL6uSihw4gwnqPhQT/openapi.json