# Website to Markdown & Text Crawler with Spend Cap (`hydrocore/website-to-markdown-text`) Actor

Crawl any website and turn its pages into clean Markdown and plain text for AI, RAG and LLMs. Hard limits on spend, pages and pages per domain. You are never charged for failed, empty or duplicate pages. Fast plain-HTTP mode, optional browser for JavaScript sites.

- **URL**: https://apify.com/hydrocore/website-to-markdown-text.md
- **Developed by:** [Hydrocore Data](https://apify.com/hydrocore) (community)
- **Categories:** AI, Developer tools, Automation
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.85 / 1,000 pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website to Markdown & Text Crawler with Spend Cap

Crawl a website and get every page as **clean Markdown and plain text**, ready for AI, RAG, LLM fine-tuning, search indexes or content audits.
You set hard limits on **money, pages, pages per domain and time**, and the crawler stops there. You are **never charged for pages that failed, came back empty or were duplicates**.

- Paste a URL, press Start, get Markdown + text + title, description and language for each page
- Plain HTTP by default (fast and cheap), optional headless browser for JavaScript sites
- One combined `.md` or `.txt` file for the whole site if you want it
- Respects robots.txt and Crawl-delay, crawls politely

### Why this one is better

Built around common complaints about website crawlers: runaway bills, runs that never stop, excludes that don't work, and paying for junk pages.

| | **This Actor** | Typical website crawler |
|---|---|---|
| **Fixed price per page** | $1.275 per 1,000 pages (plain HTTP), shown up front | Pay-per-usage crawlers bill compute time, so the cost depends on the site and settings |
| **Hard spend limit** | "Max spend (USD)" input + the run's "Maximum cost per run" are both enforced in code; the crawler stops *before* a page that would go over | Often only the page limit |
| **Hard page limit** | "Max pages" defaults to **50**, so a forgotten setting can't crawl 100,000 pages | Some crawlers default to millions of pages |
| **Per-domain limit** | "Max pages per domain" when you crawl many sites at once | Rarely available |
| **Failed pages** | Not charged: 404/500 errors, timeouts, DNS errors, redirect loops | Varies |
| **Empty pages** | Not charged (fewer than 10 words by default) | Often charged |
| **Duplicates** | Not charged and don't count toward Max pages: same canonical URL, same final URL after redirects, or **identical text** (content hash), plus tracking parameters like `utm_source` removed | Often deduplicated by canonical URL only, and duplicates may still count toward the limit |
| **Auto-stop** | Stops if no new page could be saved for 2 minutes ("Stop if stuck for"), plus a total "Max run time" (30 min default), and finishes cleanly before the platform timeout | Runs can keep going until the platform timeout |
| **Include / exclude patterns** | Checked on every link **and on redirect targets**, so excluded sections stay excluded. Simple rules: `/blog/**`, `example.com/docs/**`, or plain text like `/tag/` | Usually glob syntax, checked on links found on pages |
| **Browser pages** | Opt-in and priced separately, so you know which pages cost more | Often on by default |
| **robots.txt** | Respected by default, with its own user agent (`WebsiteToTextBot`) and Crawl-delay support | Varies |

**Compared with specific Apify Actors** (from their public Store pages, November 2026):

- **apify/web-fetch** costs $1.50 per 1,000 successful fetches on the Free and Bronze plans; this Actor costs $1.275 per 1,000 pages (**15% less**) on those plans and less on higher plans. Web Fetch fetches one URL per request and doesn't crawl a site; this Actor follows links for you. Web Fetch's README says it charges for a site's 404 page when it converts it to Markdown; this Actor never charges for 4xx/5xx pages. Web Fetch can get past bot protection; this Actor does not try to (see FAQ).
- **apify/website-content-crawler** is priced by platform usage (its README estimates about $0.20 per 1,000 pages with raw HTTP and $0.50-$5 per 1,000 with a browser). Its "Max pages" default is 9,999,999 and its README says duplicate pages (same canonical URL) still count toward Max pages, and that Crawl-delay isn't supported yet. This Actor has a fixed per-page price, a hard dollar cap, a per-domain cap, content-hash deduplication and an idle auto-stop. Website Content Crawler has more advanced browser features (adaptive Firefox, scrolling, file downloads) that this Actor doesn't.

### How to use it

1. Enter one or more **Start URLs**, for example `https://docs.example.com/`.
2. Choose the **Crawl scope**: pages under that path (default), the whole website, the whole domain with subdomains, or only the URLs you entered.
3. Set **Max pages** and, if you want, **Max spend (USD)**.
4. Press **Start**. Results appear in the **Output** tab as they are saved.

Tip: to convert a list of single pages (no crawling), paste them all and set Crawl scope to **Only the start URLs**.

### What you get

One dataset row per saved page. Every row has the same columns in the same order; empty values are `null`, counts are numbers and dates are ISO 8601 in UTC. A real row from a test run on 2026-11-04 (text shortened):

```json
{
  "url": "https://docs.apify.com/academy/scraping-basics-javascript/devtools-inspecting.md",
  "title": "Inspecting web pages with browser DevTools",
  "status": "ok",
  "wordCount": 1248,
  "language": null,
  "description": null,
  "httpStatus": 200,
  "fetchedVia": "http",
  "depth": 1,
  "linksCount": 10,
  "truncated": false,
  "canonicalUrl": null,
  "requestedUrl": "https://docs.apify.com/academy/scraping-basics-javascript/devtools-inspecting.md",
  "referrerUrl": "https://docs.apify.com/academy/scraping-basics-javascript?legacy-js-course=%2F",
  "skipReason": null,
  "error": null,
  "contentHash": "8cec9146fa007a347dfa26d6ae3209c6",
  "markdown": "> ## Documentation index\n> Fetch the complete documentation index at: https://docs.apify.com/llms.txt\n> Use this file to\n\n...",
  "text": "> ## Documentation index\n> Fetch the complete documentation index at: https://docs.apify.com/llms.tx\n\n...",
  "scrapedAt": "2026-11-05T04:08:21.940Z"
}
```

With **Include skipped pages** on, pages that were not saved get a row too, with `status: "skipped"`, a `skipReason` and the other columns `null` (real row, shortened to the filled columns):

```json
{
  "url": "https://example.com/does-not-exist-xyz",
  "status": "skipped",
  "httpStatus": 404,
  "depth": 0,
  "skipReason": "http-error",
  "error": "HTTP 404",
  "scrapedAt": "2026-11-05T04:08:21.383Z"
}
```

**Views.** The Output tab opens in **Overview** (URL, title, words, language, status, HTTP status, fetch method, depth). There are also **Markdown**, **Plain text**, **Skipped pages** and **Detailed** (every field) views. Pages with the same text, or the same final URL, are saved only once.

> Changed in v1.1: `crawledAt` is now `scrapedAt`; saved rows now have `status: "ok"`; saved and skipped rows have the same columns in a fixed order.

Also saved in the run's key-value store:

- `combined.md` or `combined.txt`: all pages in one file, if you turned on **Also save one combined file**. Each page starts with a small header (`source:` and `title:`), which RAG tools can use as metadata.
- `RUN_SUMMARY`: pages saved and charged, total charged, why the crawl stopped, and the number of skipped pages per reason.
- `SKIPPED_PAGES`: every skipped (free) page with the reason, such as `http-error`, `timeout`, `duplicate`, `empty`, `not-html`, `robots-txt`, `too-many-redirects`.

Download the data as JSON, CSV, Excel or HTML, or use the API, Make, Zapier, n8n, LangChain, LlamaIndex or the Apify MCP server.

### Pricing

Pay per event: you pay only for pages saved to your dataset, plus a tiny Actor start fee. No platform usage charges on top.

| Event | Free / Bronze | Silver | Gold and up |
|---|---|---|---|
| **Page** (plain HTTP) | $0.001275 ($1.275 / 1,000) | $0.0010625 | $0.00085 |
| **Browser page** (only with Browser or Auto crawler type) | $0.0025 ($2.50 / 1,000) | $0.0022 | $0.0019 |
| **Actor start** (at the start of each run, one per GB of memory, minimum one) | $0.0000425 | $0.0000425 | $0.0000425 |

A page is charged as either Page or Browser page, never both.
Examples: 1,000 pages over plain HTTP cost about $1.28 on the Free plan. A single-page run costs about $0.0013.

**Never charged:** pages that return an error (4xx/5xx), time out or can't be reached, redirect loops, pages with fewer words than your minimum, duplicates, files that aren't web pages (PDF, images, ZIP...), pages blocked by robots.txt, and pages outside your scope or excluded by your patterns.

**Apify free plan:** runs are limited to 100 pages each. Any paid Apify plan removes this limit.

### Input options

| Option | Default | What it does |
|---|---|---|
| Start URLs | (required) | Where the crawl starts. |
| Crawl scope | Pages under the start URL path | Which links to follow. |
| Max pages | 50 | Stop after this many saved (charged) pages. |
| Max spend (USD) | none | Stop before charging more than this. |
| Max pages per domain | 0 (no limit) | Cap per domain when crawling many sites. |
| Max link depth | 20 | How many clicks from a start URL. |
| Max run time | 30 min | Stop and save after this long. |
| Stop if stuck for | 120 s | Stop if nothing new could be saved for this long. |
| Only crawl URLs matching / Never crawl URLs matching | none | Include / exclude patterns (see FAQ). |
| Crawler type | Plain HTTP | Plain HTTP, Auto, or Browser. |
| Respect robots.txt | on | Follow robots.txt rules and Crawl-delay. |
| Max parallel requests | 5 | Total parallel requests (each site still gets at most one request every 0.25 s). |
| Page timeout | 30 s | Skip pages that load slower than this. |
| Text format | Markdown and plain text | Which text fields to return. |
| Also save one combined file | No | One `combined.md` / `combined.txt` for the whole crawl. |
| Skip pages with fewer words than | 10 | Treat short pages as empty (not saved, not charged). |
| Content area / Remove elements | automatic | CSS selectors to fine-tune what text is kept. |
| Keep menus, header and footer | off | Keep navigation text. |
| Keep image links in Markdown | off | Keep `![alt](url)` image links. |
| List skipped pages in the dataset | off | Add free rows that explain each skipped page. |
| Max page size | 10 MB | Skip bigger pages. |

### FAQ

**How do I make sure I never spend more than a set amount?**
Set **Max spend (USD)**, or set **Maximum cost per run** in Run options (or both; the lower one wins). The crawler checks the limit *before* fetching each page and stops when the next page would go over. **Max pages** is a second hard stop.

**What exactly am I charged for?**
Only for pages that were downloaded successfully, had real text, were not duplicates, and were saved to your dataset. Everything else is free and is listed in `SKIPPED_PAGES` with the reason. A tiny "Actor start" fee ($0.0000425 per GB of memory, so one fee at the default 512 MB) is charged when the run starts.

**How do include and exclude patterns work?**

- `https://example.com/blog/**` matches everything under /blog/ (`**` = anything, `*` = anything except `/`).
- `example.com/blog/**` is the same but also matches `http://` and `www.`.
- `/blog/**` is matched against the path only, on any of your start sites.
- `/tag/` (no `*`) means "the URL contains this text".
- `{a,b}` means "a or b", e.g. `/{blog,news}/**`.
  Excluded and non-matching links are never fetched, and a page that **redirects** into an excluded area is skipped too. Start URLs are always fetched.

**Why was a page skipped?**
Open `SKIPPED_PAGES` (or turn on **List skipped pages in the dataset**). Each entry has a reason: `http-error` (for example 404), `timeout`, `network-error`, `too-many-redirects`, `empty`, `duplicate`, `not-html`, `robots-txt`, `out-of-scope`, `excluded`, `too-large`.

**The page needs JavaScript and comes back empty.**
Set **Crawler type** to **Auto** (plain HTTP first; the browser only for pages that come back empty, look JavaScript-only or return 403) or **Browser** (every page). Browser pages cost more and need at least 1 GB memory (2 GB+ recommended for speed).

**Does it get past logins, paywalls, CAPTCHAs or bot protection?**
No, on purpose. It only reads pages that are publicly available without logging in, it does not solve CAPTCHAs, and it does not use tricks to get around blocking. If a site blocks it, those pages are skipped and not charged.

**Does it collect emails or phone numbers?**
No. It doesn't extract or list contact details or other personal data. It returns the page text as published. You are responsible for how you use the content and for respecting the site's terms and copyright.

**Is it legal to crawl websites?**
Crawling publicly available pages is generally legal in many countries, but copyright, database rights, terms of service and privacy laws (such as GDPR) still apply to what you do with the data. This Actor respects robots.txt by default. If in doubt, ask a lawyer or the site owner.

**Will it stop on its own if something goes wrong?**
Yes. It stops when it hits Max pages, Max spend, Max run time, or when nothing new could be saved for "Stop if stuck for" seconds. It also finishes cleanly about 25 seconds before the run timeout in Run options, so you keep everything saved so far.

**Can I use the output for RAG or a chatbot?**
Yes. Use the `markdown` field per page, or turn on **Also save one combined file** to get one `combined.md` for the whole site. Each row has `url`, `title`, `language` and `contentHash` for metadata and change detection.

**How fast is it?**
With plain HTTP, about 2-4 pages per second per site (it waits at least 0.25 s between requests to the same site, longer if robots.txt asks for it). Crawling several sites at once is faster. Our 20-page test crawl finished in about 7 seconds.

**Something doesn't work.**
Open an issue on the Issues tab with the run link. Runs always finish with a clear status message and a `RUN_SUMMARY`.

# Actor input Schema

## `startUrls` (type: `array`):

Pages where the crawl starts, for example https://docs.example.com/. Links on these pages are followed according to "Crawl scope". To fetch only these exact pages, set "Crawl scope" to "Only the start URLs".

## `crawlScope` (type: `string`):

Which links to follow. "Under the start URL path": starting at https://example.com/docs/ only crawls https://example.com/docs/... pages. "Whole website": every page on the same host (www. is ignored). "Whole domain": also subdomains such as blog.example.com. "Only the start URLs": just the pages you listed.

## `maxPages` (type: `integer`):

The crawl stops after this many pages have been saved (and charged). Skipped pages (failed, empty, duplicate) don't count. On Apify's free plan a run is limited to 100 pages.

## `maxSpendUsd` (type: `number`):

Optional. The crawl stops before it would charge more than this amount in total for this run. Leave empty for no extra limit. The "Maximum cost per run" you set in Run options is always respected too.

## `maxPagesPerDomain` (type: `integer`):

Optional. At most this many pages are saved from any one domain (www. is ignored). Useful when you crawl many sites at once. 0 means no per-domain limit.

## `maxDepth` (type: `integer`):

How many clicks away from a start URL the crawler may go. 0 means only the start URLs, 1 means start URLs plus the pages they link to, and so on.

## `maxRunMinutes` (type: `integer`):

The crawl stops and saves its results after this many minutes. 0 means no limit (the run timeout in Run options still applies; the crawler finishes cleanly just before it).

## `stallTimeoutSecs` (type: `integer`):

If no new page could be saved for this long (for example every page fails or is a duplicate), the crawl stops on its own instead of running forever.

## `includeUrlGlobs` (type: `array`):

Optional. If set, only links matching at least one pattern are crawled (start URLs are always crawled). Examples: "https://example.com/blog/**" (whole blog section), "/docs/**" (any URL whose path starts with /docs/), "/2025/" (no \* = URL must contain this text). "\*\*" matches anything, "\*" matches anything except "/".

## `excludeUrlGlobs` (type: `array`):

Optional. Links matching any pattern are never fetched, and pages that redirect to such a URL are skipped too. Same pattern rules as above. Examples: "/tag/**", "**/login\*", "?page=" (contains text).

## `crawlerType` (type: `string`):

"Plain HTTP" downloads the HTML directly and is charged as "Page". "Browser" opens every page in a headless Chrome so JavaScript content appears; charged as "Browser page" and needs at least 1 GB memory. "Auto" tries plain HTTP first and uses the browser only for pages that come back empty or need JavaScript; you pay the browser price only for those.

## `respectRobotsTxt` (type: `boolean`):

Follow each site's robots.txt rules and Crawl-delay. Pages that robots.txt disallows are skipped and not charged. Turn off only for sites you own or have permission to crawl.

## `maxConcurrency` (type: `integer`):

How many pages are fetched at the same time across all sites. Requests to one site are always spaced out (at least 0.25 s apart, or the site's Crawl-delay) to stay polite.

## `requestTimeoutSecs` (type: `integer`):

A page that doesn't load within this time is retried once, then skipped and not charged. Network errors are retried up to 2 times.

## `outputFormat` (type: `string`):

Which text fields each result has. Markdown keeps headings, lists, links and tables; plain text is just the words.

## `combinedOutput` (type: `string`):

Also save all pages in a single file in the key-value store, ready to upload to a RAG tool or an LLM. Very large crawls are split into parts of about 8 MB.

## `minWordCount` (type: `integer`):

Pages whose main text has fewer words are treated as empty: not saved and not charged.

## `contentSelector` (type: `string`):

Optional. Keep only this part of each page, for example "article" or "#content". By default the main content is found automatically (main, article or body).

## `removeSelectors` (type: `string`):

Optional. Extra elements to drop before conversion, for example ".newsletter, .related-posts". Menus, headers, footers, cookie banners, scripts and forms are removed already.

## `keepNavigation` (type: `boolean`):

Keep navigation menus, sidebars, page header and footer in the text. Off by default so you get only the main content.

## `keepImages` (type: `boolean`):

Keep images as Markdown image links (![alt](url)). Off by default to keep the text short for AI use.

## `includeSkippedPages` (type: `boolean`):

Also add a row for every skipped page (failed, empty, duplicate, blocked) with the reason. These rows are free. Either way, the full list is saved in the SKIPPED_PAGES record.

## `maxPageSizeMb` (type: `integer`):

Pages larger than this are skipped and not charged.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy/web-scraping-for-beginners"
    }
  ],
  "crawlScope": "start-url-path",
  "maxPages": 10,
  "maxPagesPerDomain": 0,
  "maxDepth": 20,
  "maxRunMinutes": 30,
  "stallTimeoutSecs": 120,
  "crawlerType": "http",
  "respectRobotsTxt": true,
  "maxConcurrency": 5,
  "requestTimeoutSecs": 30,
  "outputFormat": "both",
  "combinedOutput": "none",
  "minWordCount": 10,
  "keepNavigation": false,
  "keepImages": false,
  "includeSkippedPages": false,
  "maxPageSizeMb": 10
}
```

# Actor output Schema

## `overview` (type: `string`):

A compact table of saved pages without the long text fields.

## `markdown` (type: `string`):

URL, title and Markdown of each saved page.

## `text` (type: `string`):

URL, title, plain text and word count of each saved page.

## `results` (type: `string`):

One row per saved page with url, canonicalUrl, title, description, language, markdown, text, wordCount, linksCount, httpStatus, fetchedVia and scrapedAt.

## `combinedMarkdown` (type: `string`):

All pages in one Markdown file for RAG tools, when "Also save one combined file" is set to Markdown.

## `combinedText` (type: `string`):

All pages in one plain-text file, when "Also save one combined file" is set to text.

## `summary` (type: `string`):

Pages saved and charged, total charged, why the crawl stopped, and how many pages were skipped (not charged) for each reason.

## `skipped` (type: `string`):

Every page that was skipped and not charged, with the reason: failed, empty, duplicate, not HTML, blocked by robots.txt, timeout.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com/academy/web-scraping-for-beginners"
        }
    ],
    "maxPages": 10
};

// Run the Actor and wait for it to finish
const run = await client.actor("hydrocore/website-to-markdown-text").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://docs.apify.com/academy/web-scraping-for-beginners" }],
    "maxPages": 10,
}

# Run the Actor and wait for it to finish
run = client.actor("hydrocore/website-to-markdown-text").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy/web-scraping-for-beginners"
    }
  ],
  "maxPages": 10
}' |
apify call hydrocore/website-to-markdown-text --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,hydrocore/website-to-markdown-text"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/iALNYRux0FeG320JP/builds/aG3kQ1ExM3e7oSrax/openapi.json