# Website to Markdown — Content Extractor for LLMs & RAG (`insight.solutions/website-content-extractor`) Actor

Turn any website into clean Markdown and plain text for RAG pipelines, vector stores and LLM agents. Reads the sitemap or crawls links, strips navigation, cookie bars and footers, and returns one row per page with metadata, headings, links and optional chunks. Static HTML, no browser, no API key.

- **URL**: https://apify.com/insight.solutions/website-content-extractor.md
- **Developed by:** [Insight Solutions](https://apify.com/insight.solutions) (community)
- **Categories:** AI, Developer tools, Marketing
- **Stats:** 1 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.36 / 1,000 page extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website to Markdown — Content Extractor for LLMs & RAG

**Point it at a domain and get every page back as clean Markdown.** This Actor reads a site's `robots.txt`, walks its sitemap — nested indexes, gzip children and relative URLs included — or crawls the links when there is no sitemap, and turns each page into **boilerplate-free Markdown and plain text**: navigation, cookie bars, sidebars, share widgets, related-post strips and footers removed, headings and lists and tables and fenced code kept. Every row also carries the page's title, description, author, dates, language, canonical, headings, links and images, and — when you ask for it — an array of **chunks ready to embed**.

No API key. No browser. No JavaScript execution. **$0.60 per 1,000 pages** ($2.00 from October 20, 2026), proxy and chunking included. `robots.txt` and sitemaps are free, pages that had no content are free, and a run that extracts nothing costs nothing at all.

### Try it in 30 seconds

This is the Store prefill; paste it and run:

```json
{
  "startUrls": ["https://docs.apify.com/platform/actors"],
  "crawlLinks": true,
  "maxPages": 20,
  "maxDepth": 2,
  "output": "both",
  "chunkSize": 0
}
```

A URL with a path means *that page* plus the pages it links to. A bare domain — `basecamp.com` — means *the whole site*: robots.txt, sitemap, and up to `maxPages` of it. Want exactly the URLs you listed and nothing else? Set `crawlLinks: false`.

**Output** — one row per page; the fields you will use most are `url`, `title`, `markdown`, `text`, `headings`, `links` and `chunks` (full list under *Output reference*), plus one free `site` summary row per website. Anything that produced no content comes back as a free diagnostic row (`ok: false`, `errorType`, `error`) instead of a charge.

**Price** — $0.60 per 1,000 pages (from October 20, 2026: $2.00 per 1,000) on the FREE tier ($0.0006 a page, + $0.001 per site summary and $0.001 per run); robots.txt, sitemaps, disallowed URLs, pages with no content, chunking and the metadata are all free, and a run that extracted nothing costs nothing. Pay-per-event, no API key, no browser, limited permissions — works over the Apify MCP server (`mcp.apify.com`) and with agentic (x402) payments.

**From code** — `client.actor("insight.solutions/website-content-extractor").call(run_input={"startUrls": ["https://docs.apify.com/platform/actors"], "maxPages": 20})` with `apify-client`, or `POST https://api.apify.com/v2/acts/insight.solutions~website-content-extractor/run-sync-get-dataset-items`.

### What comes back

One `page` row per URL that had content. This one is real — read from `docs.apify.com/actors` on 2026-09-11, trimmed only where the text runs long:

```jsonc
{
  "ok": true,
  "rowType": "page",                       // "page" | "site" | "diagnostic"

  "input": "https://docs.apify.com/platform/actors",
  "website": "apify.com",
  "url": "https://docs.apify.com/actors",
  "finalUrl": "https://docs.apify.com/actors",
  "statusCode": 200,
  "contentType": "text/html; charset=utf-8",
  "responseTimeMs": 817,

  "title": "Actors | Platform | Apify Documentation",
  "description": "Learn how to develop, run and share serverless cloud programs. Create your own web scraping and automation tools and publish them on the Apify platform.",
  "siteName": null,
  "author": null,
  "publishedAt": null,
  "modifiedAt": null,
  "language": "en",
  "canonical": "https://docs.apify.com/actors",
  "image": "https://apify.com/og-image/docs-article?title=Actors",
  "keywords": [],
  "jsonLdTypes": [],

  "markdown": "# Actors\n\nActors are serverless cloud programs that take a structured JSON input, perform a task (web scraping, browser automation, data processing, and more), and optionally produce a structured output. Run them manually in [Apify Console](https://console.apify.com/actors), through the [API](https://docs.apify.com/api) or [CLI](https://docs.apify.com/cli)…\n\n## Actor lifecycle\n\n…",
  "text":     "Actors\n\nActors are serverless cloud programs that take a structured JSON input, perform a task (web scraping, browser automation, data processing, and more), and optionally produce a structured output…",
  "html": null,                            // filled only when includeHtml is on

  "wordCount": 295,
  "charCount": 1865,
  "tokenEstimate": 619,                    // characters ÷ 4
  "readingTimeMin": 1,

  "headings": [
    { "level": 1, "text": "Actors" },
    { "level": 2, "text": "Actor lifecycle" },
    { "level": 2, "text": "Actor components" },
    { "level": 2, "text": "Public and private Actors" }
  ],
  "links": [
    { "text": "Apify Console", "url": "https://console.apify.com/actors", "internal": true },
    { "text": "API", "url": "https://docs.apify.com/api", "internal": true }
  ],
  "internalLinkCount": 10,
  "externalLinkCount": 1,
  "images": [
    { "src": "https://docs.apify.com/assets/images/apify-actor-drawing-9e5b2c6bbe7a85acac72e5c7a13290a4.png",
      "alt": "Apify Actor diagram" }
  ],

  "chunks": null,                          // filled when chunkSize > 0
  "chunkCount": null,

  "extractionMethod": "article",           // how the content region was found
  "boilerplateRatio": 0.294,               // 29% of the page's text was chrome

  "depth": 0,
  "discoveredVia": "input",                // "input" | "sitemap" | "link"
  "sitemapLastmod": null,
  "scrapedAt": "2026-09-11T21:11:57.000Z"
}
```

Then one `site` row per website, written after its pages:

```jsonc
{
  "ok": true,
  "rowType": "site",
  "website": "apify.com",
  "url": "https://docs.apify.com",
  "pagesExtracted": 20,
  "discovery": "sitemap",
  "sitemapUrlCount": 20,
  "robotsBlocked": 0,
  "totalWords": 18450,
  "totalTokens": 26400,                    // budget an embedding run from this
  "avgBoilerplateRatio": 0.31,
  "failures": 3
}
```

And a free `diagnostic` row for anything that produced no content, saying which of the eight things happened.

### Static HTML only — read this before you run it

This Actor fetches the HTML a server sends and reads that. **It does not run JavaScript**, and there is no browser anywhere in it. That is the whole reason it is fast and costs $0.0006 a page.

For the large majority of the web — documentation, blogs, news, government sites, e-commerce product pages, marketing sites, wikis, forums — the server sends the content and this works perfectly. For a single-page app that renders its content in the browser, the server sends a shell, and a shell is what you get. The Actor does not pretend otherwise: a 200 OK page with fewer than 20 words after extraction produces a **free `empty` diagnostic row** that says so in plain language, and it is not billed.

If a site you need is JavaScript-rendered, use a browser-based Actor for that site. This one is the cheap, fast tool for everything else.

### How the extraction works

Four stages, and every one of them is visible in the output so a surprising row can be explained rather than guessed at.

**1. Remove what is never content.** `<script>`, `<style>`, `<noscript>`, `<svg>`, `<iframe>`, forms and form controls, comments; `<nav>`, `<header>`, `<footer>` and `<aside>`; anything with `role="navigation" | "banner" | "contentinfo" | "complementary" | "search"`; anything marked `aria-hidden` or `display:none`; and anything whose `id` or `class` contains a whole word from `nav, menu, sidebar, footer, header, cookie, consent, banner, breadcrumb, share, social, comment(s), related, promo, advert, ad(s), popup, modal, newsletter, subscribe, toc`.

That last rule has two guards, because it is the one that can go wrong. It is never applied to the element that was chosen as the content root — a site whose article lives in `<div class="content-header">` keeps its article. And a matched element survives anyway when it is *a link-light majority of its own container*, because an element that is most of what contains it is the page rather than an ornament on it. That is what keeps a Hacker News thread's `<div class="comment">` bodies while still deleting a blog's comment widget, a share bar and a table of contents.

**2. Find the content root**, in this order, each falling through when what it finds holds less than 150 characters:

| `extractionMethod` | What was used |
|---|---|
| `article` | The page's `<article>` element. With several of them it must hold half the page's text, or they are cards on a listing page and this strategy is skipped |
| `main` | `<main>` or `[role=main]` |
| `selector` | A well-known content container: `#content`, `#main`, `.post-content`, `.entry-content`, `.article-body`, `.article__body`, `.post-body`, `.markdown-body`, `.mw-parser-output`, `.prose`, `.content`, `.documentation`, `.doc-content` |
| `density` | Every block scored by `text length × (1 − link density)`, then up to the smallest ancestor holding 60% of the page's non-boilerplate text. This is what reads a table-laid-out forum thread |
| `body` | Last resort: the whole body, boilerplate still stripped |

**3. Convert to Markdown.** `h1`–`h6` → `#`…`######`; paragraphs; `<br>` → a line break; nested `ul`/`ol` with correct indentation and `start` honoured; `pre` → a fenced block with the language read from `class="language-x"`, `lang-x`, `highlight-x` or `brush: x`; inline `code` → backticks; `blockquote` → `>`; `table` → a GitHub pipe table with `|` escaped in cells and `colspan` filled with empty cells so the columns line up; `a` → `[text](absolute url)`; `img` → `![alt](absolute src)`; `strong`/`b` → `**`, `em`/`i` → `*`; `hr` → `---`; `dl`/`dt`/`dd`, `figure`/`figcaption` and `details`/`summary` all kept. Entities are decoded, whitespace runs collapse, and there is never more than one blank line between blocks.

A `<table>` used for page layout — no `<th>` anywhere, cells holding `<div>`s and paragraphs — is rendered as the container it is being used as, not as a pipe table. A five-column row containing a whole comment is not a table to anybody except the HTML.

**4. Read the metadata**: `og:title` → `<title>` → first `<h1>`; meta description → `og:description`; author from `meta[name=author]`, `article:author`, `rel=author` or JSON-LD `author.name`; dates from `article:published_time` / `article:modified_time`, JSON-LD `datePublished` / `dateModified`, or `<time datetime>`; plus `<html lang>`, `rel=canonical`, `og:site_name`, `og:image`, meta keywords and every JSON-LD `@type`.

### Chunking for RAG

Set `chunkSize` between 500 and 10,000 and every row gains a `chunks` array:

```jsonc
"chunks": [
  {
    "index": 0,
    "text": "## 3.1. Using Python as a Calculator\n\nLet's try some simple Python commands…",
    "headingPath": "3. An Informal Introduction to Python > 3.1. Using Python as a Calculator",
    "charCount": 1492,
    "tokenEstimate": 373
  }
]
```

Four guarantees, all tested against real captured pages:

- **Chunks break at block boundaries first.** A paragraph is only cut open when it is longer than `chunkSize` by itself, and then at a sentence end.
- **A fenced code block is never split**, and neither is a table. They go into one chunk whole.
- **`headingPath` is exact** — it comes from the `<h1>`–`<h6>` elements themselves, not from counting `#` characters, so a chunk from halfway down a page knows which section it is in.
- **`chunkOverlap` repeats the tail of the previous chunk** on a word boundary, and never opens a code fence it does not close.

The one documented exception to the size bound: a single code block or table longer than `chunkSize` becomes a chunk of its own rather than being cut in half. Everything else stays within `chunkSize + chunkOverlap`.

### Use cases

- **Build a RAG index over a documentation site.** One run, `chunkSize: 1500`, and every chunk arrives with its heading path already attached.
- **Feed an LLM agent the current state of a site** without a browser in the loop and without paying per token for navigation menus.
- **Archive a blog or a handbook as Markdown** — one file per row, `finalUrl` as the filename.
- **Competitive and content research** at a level a search snippet cannot reach: the actual words on 500 pages, with word counts and reading times.
- **Migrate a CMS.** Extract the old site to Markdown, import it into the new one.
- **Train or evaluate on a corpus you control**, with `sitemapLastmod` telling you what changed since the last crawl.

### How it works, and why it keeps working

- **Discovery ladder.** Every domain is tried on `https://`, then `https://www.`, then plain `http://`, using `robots.txt` as the probe — it is needed anyway, it is tiny, and any HTTP answer to it proves the host is there.
- **Sitemaps as sites really publish them.** `Sitemap:` lines from `robots.txt`, then `/sitemap.xml`, `/sitemap_index.xml`, `/sitemap-index.xml`, `/sitemap/sitemap.xml`. Indexes are followed three levels deep, `*.xml.gz` children are inflated whether the server pre-inflated them or not, and relative `<loc>` values are resolved against the sitemap's own URL. Collection stops as soon as there are enough URLs to fill `maxPages`.
- **Link crawl as the fallback, and as an option.** Links are read from the whole page — including the navigation this Actor strips out of the *text*, because that is where a site keeps the links to the rest of itself. Assets, feeds, API paths and PDFs are not followed.
- **One polite crawler.** Requests to one host are spaced 250–600 ms apart no matter how high `maxConcurrency` goes; each parallel worker holds its own proxy session; every request has a 20-second ceiling and the whole run has `maxRunSecs`.
- **Bot walls are retried once, from a different exit IP, and then reported.** Cloudflare's "Just a moment…" interstitial arrives as HTTP 200, so it is recognised from the page itself rather than from the status. A second refusal is about the URL, not the address, and hammering it would earn the run a blocklist entry.
- **Redirects are deduplicated on the final URL**, so two queued URLs that land on one page produce one row and one charge.

### Input reference

| Field | Type | Default | What it does |
|---|---|---|---|
| `startUrls` | array | — | Domains (the whole site) and/or URLs with a path (that page). Entries on the same host are one website: one crawl, one set of robots rules, one summary row. An email address or a bare IP is rejected with a free `invalid-input` diagnostic row |
| `crawlLinks` | boolean | `true` | Follow same-host links found on extracted pages |
| `maxPages` | integer | `50` | Pages per website. Max 5,000. This is the price of a run |
| `maxDepth` | integer | `3` | How far from the start the link crawl may travel. 0 = follow nothing |
| `includeUrlPatterns` | array | `[]` | Globs (`*/docs/*`) or `/regex/i`. A discovered URL must match one if any are given |
| `excludeUrlPatterns` | array | `[]` | Same spelling, and it wins. Never applied to a URL you typed |
| `output` | string | `both` | `both`, `markdown` or `text` |
| `includeHtml` | boolean | `false` | Also return the extracted region's HTML |
| `keepLinks` | boolean | `true` | `[text](url)` in the Markdown, or just the text |
| `keepImages` | boolean | `true` | `![alt](src)` in the Markdown, or just the alt |
| `chunkSize` | integer | `0` | 0 = off. 500–10,000 turns on the `chunks` array |
| `chunkOverlap` | integer | `0` | 0–200 characters repeated between chunks |
| `respectRobots` | boolean | `true` | Obey the `User-agent: *` group |
| `maxSitemapUrls` | integer | `5000` | Ceiling on URLs collected from sitemaps |
| `maxConcurrency` | integer | `8` | Pages of one site in flight. 1–16 |
| `maxRunSecs` | integer | `240` | Wall-clock budget. 30–3,600 |
| `proxyConfiguration` | object | `{"useApifyProxy": true}` | Datacenter by default; switch to RESIDENTIAL for hostile sites |

### Output reference

`rowType` tells you what a row is. Every row carries every column, so a CSV export is one rectangle.

**`page` rows** — the response (`url`, `finalUrl`, `statusCode`, `contentType`, `responseTimeMs`), what the page says about itself (`title`, `description`, `siteName`, `author`, `publishedAt`, `modifiedAt`, `language`, `canonical`, `image`, `keywords`, `jsonLdTypes`), the content (`markdown`, `text`, `html`, `wordCount`, `charCount`, `tokenEstimate`, `readingTimeMin`, `headings`, `links`, `internalLinkCount`, `externalLinkCount`, `images`, `chunks`, `chunkCount`, `extractionMethod`, `boilerplateRatio`) and where the URL came from (`depth`, `discoveredVia`, `sitemapLastmod`).

**`site` rows** — `pagesExtracted`, `discovery`, `sitemapUrlCount`, `robotsBlocked`, `totalWords`, `totalTokens`, `avgBoilerplateRatio`, `failures`.

**`diagnostic` rows** — `ok: false`, never charged, one per URL that produced nothing:

| `errorType` | What it means |
|---|---|
| `unreachable` | Nothing answered on https or http |
| `blocked` | Refused, or an anti-bot interstitial, from two different exit IPs |
| `timeout` | `maxRunSecs` ran out before this could be read |
| `invalid-input` | Not a website or page URL — an email address, an IP, a bad URL pattern |
| `not-html` | A PDF, an image, a JSON endpoint. Nothing to extract |
| `http-error` | A 4xx or 5xx. An error page is not content |
| `too-large` | Over 5 MB of markup |
| `empty` | 200 OK HTML, but under 20 words after extraction — almost always a JavaScript-rendered shell |

Five dataset views — **Content**, **Metadata**, **Chunks**, **Site summaries** and **What could not be read** — are set up in the Console so each shape exports as its own clean table.

### Pricing

| Event | When | FREE | BRONZE | SILVER | GOLD |
|---|---|---|---|---|---|
| **Page extracted** *(primary)* | One page of content in your dataset | $0.0006 | $0.0006 | $0.00048 | $0.00036 |
| Site summary | One website that produced at least one page | $0.001 | $0.001 | $0.001 | $0.001 |
| Run started | Once per run, after the first page is extracted | $0.001 | $0.001 | $0.001 | $0.001 |

**$0.60 per 1,000 pages**, proxy included, chunking included.

**Price change notice.** From October 20, 2026 (UTC) the `page` event is $0.002 / $0.002 / $0.0016 / $0.0012 (FREE / BRONZE / SILVER / GOLD), with `site` and `actor-start` unchanged; the table above applies until then.

#### What you are never charged for

- **`robots.txt` and every sitemap file**, including a fifty-file gzip index tree.
- **Any URL that produced no content** — a dead host, a PDF, a 404, a bot wall, a document over 5 MB, or a JavaScript-only shell. Each gets a free diagnostic row saying which.
- **URLs `robots.txt` disallows.** They are counted on the site row and never fetched.
- **Chunking, the `text` field, the `html` field and the metadata.** They are part of the page, not extras.
- **A run that extracted nothing.** A run whose URLs were all usable but produced nothing — a domain that does not resolve, a site that refused us, a JavaScript-only shell — finishes **SUCCEEDED with zero results**, a status message that says so, and a free diagnostic row per URL telling you why; nothing is charged, start fee included. A run finishes **FAILED** only when there was nothing usable to attempt (no start URL, or none this Actor can fetch), or the Actor itself hit an error.

Bound a run's spend with `ACTOR_MAX_TOTAL_CHARGE_USD`. The Actor stops *fetching* while it can still bill for what it holds, so a budget is never exceeded and never leaves a paid summary standing over unpaid detail.

### Limits, and the ones that might bite

- **No JavaScript rendering.** Said once above and repeated here because it is the limit that matters. Server-rendered HTML only.
- **5,000 pages per website per run**, and 5,000 sitemap URLs collected. Raise `maxRunSecs` to match: at roughly two to three pages a second per host, 1,000 pages is around seven minutes.
- **One `<article>` wins; a dozen do not.** On a listing page where each card is an `<article>`, the Actor falls through to `<main>` and returns the whole list. That is usually what you want from a listing page, and `extractionMethod` tells you it happened.
- **Boilerplate removal is a heuristic, not a parser.** It is tested against eleven real captured pages, and every one of them has its result written down in `TEST_RESULTS.md`. When a row looks wrong, `extractionMethod` and `boilerplateRatio` are the two columns that explain it: `body` plus a ratio near zero means no strategy found a content region.
- **`tokenEstimate` is characters ÷ 4.** A rule of thumb for budgeting, not a tokenizer. Your model's count will differ, usually by under 15% on English prose and by more on code.
- **A table that is genuinely tabular but has no `<th>` and holds block content** is rendered as prose rather than as a pipe table. That is the safe direction to be wrong in.
- **A page behind a login is not fetched.** There is no authentication in this Actor and there is not going to be one.
- **The upstream format may change.** Sites are rebuilt, sitemaps move, and a site that published a clean `sitemap.xml` last month can serve an index of gzip children this month. When that happens the discovery path changes rather than breaking — the `discovery` and `sitemapUrlCount` columns on the site row show which path was taken.

### Use it from an AI agent, or from code

One JSON object in, one flat array out — the shape agent runtimes want. The Actor runs with **limited permissions**, uses **pay-per-event** pricing and never enters Standby, so it works over the Apify MCP server and with x402 agentic payments. The **Integrations** tab pushes results to Slack, a webhook, Zapier, Make, Google Sheets, Snowflake or BigQuery.

```bash
curl -X POST "https://api.apify.com/v2/acts/insight.solutions~website-content-extractor/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":["https://docs.apify.com/platform/actors"],"maxPages":20,"chunkSize":1500}'
```

```python
## pip install apify-client
from apify_client import ApifyClient

client = ApifyClient("<APIFY_TOKEN>")
run = client.actor("insight.solutions/website-content-extractor").call(run_input={
    "startUrls": ["docs.apify.com"],
    "maxPages": 200,
    "chunkSize": 1500,
    "chunkOverlap": 150,
    "excludeUrlPatterns": ["*/changelog/*"],
})

for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    if row["rowType"] == "page":
        for chunk in row["chunks"] or []:
            embed(chunk["text"], metadata={
                "url": row["finalUrl"],
                "title": row["title"],
                "section": chunk["headingPath"],
            })
    elif row["rowType"] == "site":
        print("crawled", row["pagesExtracted"], "pages,", row["totalTokens"], "tokens")
```

### FAQ

**How does it find the pages?**
`robots.txt` first, for the `Sitemap:` lines and the crawl rules. Then those sitemaps, or `/sitemap.xml` and three other usual locations. Sitemap indexes are followed to their children, gzip children are inflated, and relative `<loc>` values are resolved. If none of that yields anything, it crawls from the home page. When you give a URL with a path, that page is read first and its links are followed.

**Does it render JavaScript?**
No. It reads the HTML the server sends. A page that renders its content in the browser comes back as whatever the shell contains, and when that is under 20 words the Actor files a free `empty` diagnostic row saying so rather than billing you for an empty row.

**How good is the boilerplate removal, really?**
Good enough that the Apify docs page comes back as its H1 and four sections with no sidebar and no footer, the Python tutorial keeps all 42 code samples with their `>>>` prompts, and MDN's `<table>` reference keeps both of its data tables and all 24 code blocks with the right languages. Every one of those is an assertion in the test suite, run against the captured bytes. The full list, including what is *not* perfect, is in `TEST_RESULTS.md`.

**What is `extractionMethod` for?**
Telling you which of the five strategies found the content, so a surprising row can be explained. `article` and `main` are the confident ones; `density` means the page had no semantic structure to work with; `body` means nothing was found and you are looking at the whole page minus its chrome.

**Can I get just the text, without Markdown syntax?**
Yes — `output: "text"`. Both formats come from one walk of the page, so they can never disagree about what it said; choosing one only leaves the other column null.

**Do the chunks overlap correctly across a code block?**
Yes. An overlap that would start inside a fenced block is trimmed past the fence, so a chunk never opens a fence it does not close. That is tested against the Python, MDN and GitHub captures at four chunk sizes.

**Why did a page I know exists not come back?**
Five possibilities, in order of likelihood: it is not in the sitemap and nothing links to it within `maxDepth`; `maxPages` was reached first; an `excludeUrlPatterns` entry matched it; `robots.txt` disallows it; or it produced no content. The site row's `sitemapUrlCount`, `pagesExtracted` and `robotsBlocked` distinguish the first four, and there is a free diagnostic row for the fifth.

**Does it follow links to other domains?**
No. A crawl stays on the host it started from, with `www.` folded away. Links to other domains are still reported in the `links` column with `internal: false`.

**Can I extract 5,000 pages?**
Yes, that is the per-website ceiling. Raise `maxRunSecs` to match and bound the spend with `ACTOR_MAX_TOTAL_CHARGE_USD`. At $0.0006 per page the whole site is $3.

**What happens if one site in a list fails?**
The others still run. The failed one produces a free diagnostic row and the run finishes `SUCCEEDED`. If *every* site fails, the run still finishes `SUCCEEDED`, with zero pages, a status message that says so ("0 pages extracted. N diagnostic row(s) explain why …") and nothing billed at all. Only an input with nothing fetchable in it finishes `FAILED`.

**Is the data fresh?**
Live. Every run reads the site at that moment; nothing is cached, and there is no index behind this.

### Legal and data-protection notes

- **Public pages only.** Every URL fetched is one a browser would load without signing in. The Actor never logs in, never accepts cookies or session tokens, never takes an API key belonging to anyone else, and never touches anything behind authentication.
- **It obeys the site's own instructions by default.** `robots.txt` is read before anything else and its `Disallow` rules are respected unless you deliberately turn that off for a site you control.
- **It is a polite crawler.** Requests to one host are spaced 250–600 ms apart, each has a 20-second ceiling, and the whole run has a budget. A refusal is retried once from a different address and then left alone.
- **Content belongs to whoever published it.** This Actor reformats a public web page; it does not license it. Copyright, terms of use and any database rights are unchanged by passing the page through a Markdown converter, and what you may do with the output is between you and the publisher.
- **Comment threads and forums can contain personal data.** The extractor keeps the text a page shows a reader, which on a forum includes user names and what they wrote. If you crawl that kind of site, that is personal data and your own obligations apply to it.
- **Not affiliated with any website read or named in these docs.** All trademarks belong to their respective owners.

### Our other Actors

Every Insight Solutions Actor is pay-per-result with no browser, no login and no API key, and every one of them returns free diagnostic rows instead of billing for failures. Prices are per 1,000 results.

**Video, audio & social**

- [YouTube Transcript API](https://apify.com/insight.solutions/youtube-transcript-api) — captions as timed segments, text, SRT or VTT, with language fallback and translation.
- [YouTube Comments API](https://apify.com/insight.solutions/youtube-comments-api) — comments and replies with likes, pinned and hearted flags, newest or top sort.
- [YouTube Channel API](https://apify.com/insight.solutions/youtube-channel-api) — a channel's videos, Shorts and live streams, plus YouTube search.
- [Podcast Search, Episodes & Charts API](https://apify.com/insight.solutions/podcast-api) — Apple Podcasts search, charts and full episode feeds.
- [Bluesky Scraper](https://apify.com/insight.solutions/bluesky-scraper) — profiles, posts, followers and follows from the public AT Protocol API.
- [Telegram Channel Scraper](https://apify.com/insight.solutions/telegram-channel-scraper) — posts, views and channel stats from public Telegram channels.
- [Substack Scraper](https://apify.com/insight.solutions/substack-scraper) — posts with full free text, comments and publication profiles.
- [Hacker News API](https://apify.com/insight.solutions/hacker-news-api) — stories, comments, users, front page and a structured "Who is hiring?" parser from the official HN APIs.
- [GitHub Repos, Trending & Releases API](https://apify.com/insight.solutions/github-repos-api) — repository facts, trending lists, releases and search, no token needed for small runs.
- [Discourse Forum API](https://apify.com/insight.solutions/discourse-forum-api) — topics, posts and categories from any Discourse community via its own JSON endpoints, usernames only.

**News, documents & the web**

- [Google News Search, Topics & Real Article URLs](https://apify.com/insight.solutions/google-news-api) — news search and topic feeds with the publisher's real URL decoded.
- [WordPress Posts & Pages API](https://apify.com/insight.solutions/wordpress-posts-api) — posts and pages from any WordPress site as clean text or Markdown, with categories, tags and images.
- [Website Metadata API](https://apify.com/insight.solutions/website-metadata-api) — any URL's meta tags, Open Graph, Twitter cards, hreflang, feeds, JSON-LD and microdata, with normalised Product, Article, Event and FAQ entities.
- [Subdomain Finder API](https://apify.com/insight.solutions/subdomain-finder-api) — every subdomain of a domain from certificate-transparency logs, resolved over DNS, with an optional live HTTP check.
- [Internet Archive API](https://apify.com/insight.solutions/internet-archive-api) — archive.org search, item metadata, files and reviews.
- [Wayback Machine Toolkit](https://apify.com/insight.solutions/wayback-toolkit) — archived URL inventories, snapshots and text diffs between dates.
- [Website Technology Detector](https://apify.com/insight.solutions/website-tech-detector) — the tech stack behind any site, with the evidence for each detection.
- [Domain Intelligence API](https://apify.com/insight.solutions/domain-intelligence-api) — DNS, RDAP registration, TLS certificate and HTTP facts in one row per domain.
- [SEO Page Audit](https://apify.com/insight.solutions/seo-page-audit) — sitemap crawl with on-page checks, structured data and broken-link reports.
- [Keyword Suggestions API](https://apify.com/insight.solutions/keyword-suggestions-api) — Google, YouTube, Bing, Amazon and eBay autocomplete with alphabet and question expansions.
- [Website Contact Extractor](https://apify.com/insight.solutions/website-contact-extractor) — emails, phone numbers and social profiles from any list of websites.
- [Web Search Results API](https://apify.com/insight.solutions/web-search-api) — Bing and DuckDuckGo organic results with snippets, no key, no browser.
- [Company Enrichment API](https://apify.com/insight.solutions/company-enrichment-api) — a domain in, a company profile out: firmographics, contacts, tech stack, DNS and hiring signal.
- [Company Dossier API](https://apify.com/insight.solutions/company-dossier-api) — one company in, twelve sections out: profile, tech, contacts, DNS, open roles, news, SEC filings, federal awards, recalls, YC batch and apps.
- [Press Releases API](https://apify.com/insight.solutions/press-releases-api) — GlobeNewswire and PR Newswire releases plus any newsroom feed, by keyword, company, ticker or subject.
- [Federal Register API](https://apify.com/insight.solutions/federal-register-api) — rules, proposed rules, notices and the Public Inspection desk with dockets, comment deadlines and CFR references.
- [Academic Papers Search API](https://apify.com/insight.solutions/academic-papers-api) — OpenAlex, Crossref, arXiv and PubMed in one row per paper: abstract, citations, open-access PDF, authors and venue.
- [RSS & Atom Feed Monitor](https://apify.com/insight.solutions/rss-feed-monitor) — any RSS, Atom or JSON feed (or an OPML file) in, only the new items out, with keyword filters and a webhook.
- [Website Change Monitor](https://apify.com/insight.solutions/website-change-monitor) — watch any pages, diff the text between runs, get change rows with added/removed lines, keyword alerts and a webhook.
- [Wikipedia & Wikidata API](https://apify.com/insight.solutions/wikipedia-api) — article text, search, daily pageviews and Wikidata entity facts, any language edition.

**Business, finance & jobs**

- [Congress & Insider Trades API](https://apify.com/insight.solutions/congress-insider-trades-api) — STOCK Act periodic transaction reports and SEC Form 4 insider trades in one schema.
- [Federal Contracts, Grants & Lobbying API](https://apify.com/insight.solutions/federal-contracts-grants-api) — SAM.gov opportunities, USAspending awards, Grants.gov notices and Senate lobbying filings in one schema.
- [SEC EDGAR API](https://apify.com/insight.solutions/sec-edgar-api) — filings, XBRL financials and full-text search by ticker or CIK.
- [Clinical Trials & FDA API](https://apify.com/insight.solutions/clinical-trials-fda-api) — ClinicalTrials.gov studies plus openFDA recalls, labels, approvals, 510(k)s and adverse-event reports.
- [Product & Vehicle Recalls API](https://apify.com/insight.solutions/product-recalls-api) — CPSC, NHTSA, FDA and USDA recalls, vehicle complaints and ratings, plus a VIN decoder.
- [Y Combinator Companies, Batches & Founders](https://apify.com/insight.solutions/yc-companies-directory) — the YC directory with founders and social links, filterable by batch, industry and hiring status.
- [Career Site Jobs API](https://apify.com/insight.solutions/ats-jobs-api) — jobs straight from Greenhouse, Lever, Ashby, Workable and 10+ other ATS career sites.
- [Workday Jobs API](https://apify.com/insight.solutions/workday-jobs-api) — every posting on any Workday career site, with facets, search, posted dates and full descriptions.
- [New Job Postings Monitor](https://apify.com/insight.solutions/job-postings-monitor) — new, closed and changed postings on the career sites you watch.
- [Hiring Signals API — Open Roles & Hiring Surge by Company](https://apify.com/insight.solutions/hiring-signals-api) — one row per company per run: open roles, what opened and closed, department and seniority breakdowns, and a hiring-surge flag.
- [Remote Jobs API](https://apify.com/insight.solutions/remote-jobs-api) — RemoteOK, Remotive, We Work Remotely, Himalayas, Jobicy and more in one schema, deduplicated.
- [Shopify Products API](https://apify.com/insight.solutions/shopify-products-api) — any Shopify store's catalogue, variants, prices and stock signals.
- [WooCommerce Products API](https://apify.com/insight.solutions/woocommerce-products-api) — any WooCommerce store's catalogue, prices, variants and stock through its public Store API.
- [Shopify App Store API](https://apify.com/insight.solutions/shopify-app-store-api) — Shopify App Store listings by category, collection or developer: pricing plans, ratings and merchant reviews.
- [Luma Events API](https://apify.com/insight.solutions/luma-events-api) — upcoming Luma events by city, category or calendar: dates, venue, price, capacity and organiser calendar, no host names.
- [Shopify Store Monitor](https://apify.com/insight.solutions/shopify-store-monitor) — price drops, sales, restocks, sell-outs and new products on any Shopify store, one row per change.
- [Public Tenders API](https://apify.com/insight.solutions/public-tenders-api) — EU TED, UK Find a Tender and Contracts Finder notices by keyword, CPV code, country, stage and deadline.
- [Nonprofit & IRS 990 Lookup API](https://apify.com/insight.solutions/nonprofit-990-api) — search US nonprofits and get EIN, NTEE code and multi-year Form 990 financials.
- [OpenStreetMap Places API](https://apify.com/insight.solutions/osm-places-api) — businesses and points of interest by category and area from OpenStreetMap: name, address, coordinates, website, phone, opening hours.

**Apps & games**

- [App Store & Google Play Reviews API](https://apify.com/insight.solutions/app-reviews-api) — reviews from both stores with ratings, versions and developer replies.
- [App Store Top Charts & App Search API](https://apify.com/insight.solutions/app-charts-api) — Apple top charts by country and genre, plus app search and details.
- [App Store Keyword Rank Tracker](https://apify.com/insight.solutions/app-store-keyword-rank-tracker) — where any app ranks for any keyword on the App Store and Google Play, with rank changes and ASO suggestions.
- [Steam Reviews API](https://apify.com/insight.solutions/steam-reviews-api) — Steam reviews with playtime, helpfulness and game details.
- [Steam Game Data API](https://apify.com/insight.solutions/steam-store-stats-api) — prices, tags, review scores, live player counts and top charts.
- [Sports Scores & Schedules API](https://apify.com/insight.solutions/sports-scores-api) — scores, schedules, standings, teams and headlines for the NFL, NBA, MLB, NHL, soccer and more, no key.
- [App Store Monitor](https://apify.com/insight.solutions/app-store-monitor) — App Store and Google Play listings watched between runs: new versions with release notes, price, rating and listing changes, one row per change.

# Actor input Schema

## `startUrls` (type: `array`):

What to read. An entry with **no path** — `apify.com` — means *the site*: the Actor reads `robots.txt`, follows the `Sitemap:` lines it finds, and extracts up to **Maximum pages** of it; a site that publishes no sitemap is crawled breadth-first from its home page. An entry **with a path** — `https://docs.apify.com/platform/actors` — means *that page*, plus the pages it links to when **Follow links** is on. Entries on one host are one website, one crawl, one summary row.

## `crawlLinks` (type: `boolean`):

On by default. Every page that is extracted has its same-host links read and queued, down to **Maximum depth**. Turn it off to extract exactly the URLs you listed and nothing else — useful when you already have the list and want the cost to equal the number of URLs in it. Links are read from the whole page, including the navigation this Actor strips out of the text: that is where a site keeps the links to the rest of itself.

## `maxPages` (type: `integer`):

How many pages to extract per website. This is the number that sets the price of a run: every extracted page is one `page` event. Sitemap URLs are taken in the order the sitemap lists them; a link crawl takes them breadth-first, shallowest links first. Pages that turn out to be PDFs, error pages, bot walls or empty shells are not counted against this and are not charged.

## `maxDepth` (type: `integer`):

How far the link crawl may travel from where it started. A URL you listed, and a URL that came out of a sitemap, is depth 0; a link found on it is depth 1. Set it to 0 to follow nothing, 1 for a page and its immediate links. Only used when **Follow links** is on.

## `output` (type: `string`):

`both` gives each row a `markdown` field and a `text` field. `markdown` fills only `markdown` — headings, lists, tables, fenced code, links. `text` fills only `text` — the same content with the syntax stripped, paragraphs separated by blank lines. Both are produced from one walk of the page, so they can never disagree about what it said; choosing one only leaves the other column null and the dataset smaller.

## `includeUrlPatterns` (type: `array`):

Optional. When any pattern is given, a discovered URL must match at least one of them to be extracted. Two spellings are accepted: a **glob** matched against the whole URL (`https://example.com/blog/*`, `*/docs/*`), where `*` is any run of characters and `?` is one; or a **regular expression** wrapped in slashes (`/\/blog\/\d{4}\//i`). Patterns apply to URLs the Actor *discovers* — never to a URL you typed into **Start URLs**, which is always read.

## `excludeUrlPatterns` (type: `array`):

Optional, and it wins over the include list. Same glob-or-regex spelling. Use it to keep tag archives, paginated listings, author pages and print views out of a crawl: `*/tag/*`, `*?page=*`, `/\/(author|print)\//`. A pattern that will not compile as a regular expression gets its own free diagnostic row rather than being silently ignored.

## `includeHtml` (type: `boolean`):

Off by default. When on, each row carries an `html` field holding the markup of the content region the Actor chose — not the whole document, and with scripts, styles, navigation and the rest of the boilerplate already removed. Useful when you want to run your own parser over the same region this Actor picked. It makes the dataset several times larger.

## `keepLinks` (type: `boolean`):

On by default: an anchor becomes `[text](https://absolute/url)` in the Markdown, with relative URLs resolved against the page they were found on. Turn it off and the anchor text stays while the URL goes — often what you want before embedding, because a URL costs tokens and carries little meaning. The `links` column lists every link either way, so nothing is lost.

## `keepImages` (type: `boolean`):

On by default: an image becomes `![alt](https://absolute/src)`. Turn it off and only the alt text survives in the body. Spacers, 1×1 tracking beacons and `data:` URIs are dropped either way. The `images` column lists up to 100 images with their alt text regardless of this setting.

## `chunkSize` (type: `integer`):

Leave at 0 for no chunking. Set it between 500 and 10,000 and each row gains a `chunks` array — `{index, text, headingPath, charCount, tokenEstimate}` — ready to embed. Chunks break at block boundaries, so a chunk never starts mid-sentence; a fenced code block or a table is never split; and `headingPath` carries the real `H1 > H2 > H3` trail. Chunking is included in the per-page price. See *Chunking for RAG* in the README for the one documented size exception.

## `chunkOverlap` (type: `integer`):

How much of the end of one chunk to repeat at the start of the next, so a fact that straddles a boundary is retrievable from either side. Only used when **Chunk size** is set, and capped at half the chunk size. The repeated text is cut back to a word boundary, and never opens a code fence it does not close.

## `respectRobots` (type: `boolean`):

On by default, and worth leaving on. The Actor reads the `User-agent: *` group of each site's `robots.txt` and skips any URL it disallows, using the same longest-match-wins rule search engines use, with `*` and `$` patterns supported. Skipped URLs are counted on the `site` row as `robotsBlocked` and are never fetched or billed. Turn it off only for a site you own or have permission to crawl.

## `maxSitemapUrls` (type: `integer`):

How many URLs to read out of a site's sitemaps before stopping. Collection also stops as soon as there are enough URLs to fill **Maximum pages**, because reading a news site's whole sitemap tree to extract fifty pages would be slow as well as rude. Sitemap indexes are followed three levels deep, gzip children included, and at most 50 sitemap files are fetched per website.

## `maxConcurrency` (type: `integer`):

How many pages of one website are fetched at once. Each parallel worker keeps its own proxy session, so a page that refuses one exit IP does not burn the address the others are using. Requests to a single host are spaced 250–600 ms apart whatever this is set to, so raising it buys parallel parsing rather than a burst of simultaneous requests. Websites themselves are always read one at a time.

## `maxRunSecs` (type: `integer`):

Wall-clock budget for the whole run. When it is reached the Actor stops fetching, keeps and bills for every row it already wrote, and files a free diagnostic row saying how many URLs it never reached. Nothing is charged for work that did not happen.

## `proxyConfiguration` (type: `object`):

Ordinary websites do not refuse datacenter address ranges the way the big platforms do, so the default is Apify's datacenter proxy and its cost is already inside the per-page price. If you are reading a site behind an aggressive bot wall, switch the group to RESIDENTIAL here and nothing else changes — a page that comes back as an anti-bot interstitial is already retried once from a fresh exit IP. Clearing the field sends requests from the run's own address.

## Actor input object example

```json
{
  "startUrls": [
    "https://docs.apify.com/platform/actors",
    "basecamp.com"
  ],
  "crawlLinks": true,
  "maxPages": 20,
  "maxDepth": 2,
  "output": "both",
  "includeUrlPatterns": [
    "https://example.com/blog/*"
  ],
  "excludeUrlPatterns": [
    "*/tag/*",
    "*?page=*"
  ],
  "includeHtml": false,
  "keepLinks": true,
  "keepImages": true,
  "chunkSize": 0,
  "chunkOverlap": 0,
  "respectRobots": true,
  "maxSitemapUrls": 5000,
  "maxConcurrency": 8,
  "maxRunSecs": 240,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `results` (type: `string`):

One row per extracted page — Markdown, plain text, metadata, headings, links, images and optional chunks — plus a summary row per website. Delivered as JSON items in the default dataset.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://docs.apify.com/platform/actors"
    ],
    "crawlLinks": true,
    "maxPages": 20,
    "maxDepth": 2,
    "output": "both",
    "includeUrlPatterns": [],
    "excludeUrlPatterns": [],
    "includeHtml": false,
    "keepLinks": true,
    "keepImages": true,
    "chunkSize": 0,
    "chunkOverlap": 0,
    "respectRobots": true,
    "maxSitemapUrls": 5000,
    "maxConcurrency": 8,
    "maxRunSecs": 240,
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("insight.solutions/website-content-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://docs.apify.com/platform/actors"],
    "crawlLinks": True,
    "maxPages": 20,
    "maxDepth": 2,
    "output": "both",
    "includeUrlPatterns": [],
    "excludeUrlPatterns": [],
    "includeHtml": False,
    "keepLinks": True,
    "keepImages": True,
    "chunkSize": 0,
    "chunkOverlap": 0,
    "respectRobots": True,
    "maxSitemapUrls": 5000,
    "maxConcurrency": 8,
    "maxRunSecs": 240,
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("insight.solutions/website-content-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://docs.apify.com/platform/actors"
  ],
  "crawlLinks": true,
  "maxPages": 20,
  "maxDepth": 2,
  "output": "both",
  "includeUrlPatterns": [],
  "excludeUrlPatterns": [],
  "includeHtml": false,
  "keepLinks": true,
  "keepImages": true,
  "chunkSize": 0,
  "chunkOverlap": 0,
  "respectRobots": true,
  "maxSitemapUrls": 5000,
  "maxConcurrency": 8,
  "maxRunSecs": 240,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call insight.solutions/website-content-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,insight.solutions/website-content-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/s7eOt4DcligonfCY5/builds/YzTQ1EJcvIImlsqgL/openapi.json