| ๐ Page URL and final URL (`crawl.loadedUrl`) |
๐ Main content as Markdown (`markdown`) |
๐ Main content as plain text (`text`) |
| ๐ท๏ธ Title and meta description |
๐ค Author, publisher, copyright |
๐
Published and modified dates |
| ๐ Language code and canonical URL |
๐ผ๏ธ Open Graph tags, image and favicon |
๐งฉ JSON-LD structured data (`metadata.jsonLd`) |
| ๐ Every link with its text (`links`) |
๐ Heading outline (`headings`) |
๐ Tables as Markdown (`tables`) |
| ๐๏ธ Images and embedded videos |
๐ข Word count and reading time |
๐งฎ Content hash for change tracking |
| โ๏ธ RAG chunks with heading paths (`chunks`) |
๐ PDF / DOCX text and file link (`file`) |
๐ธ Screenshot link (`screenshotUrl`) |
On a 100-page crawl of docs.apify.com every row had a title, description, language, canonical URL, Markdown, text, links and word count (100%); headings 99%, images 59%. Pages from nine search results also carried author (56%), published date (56%), modified date (78%), publisher (89%) and JSON-LD (78%), because those sites publish them.
### How to crawl a website with Website Content Crawler
1. [Create a free Apify account](https://console.apify.com/sign-up) - the free plan includes monthly credit, no card needed.
2. Open [Website Content Crawler](https://apify.com/topapi/website-content-crawler) and click **Try for free**.
3. Paste the address of the docs, blog or site section into **Start URLs** and set **Max pages**.
4. Click **Start**. Pages stream into the dataset as they are crawled.
5. Open the **Storage** tab and **Export** as JSON, CSV or Excel, or send the dataset to LangChain, LlamaIndex or a vector database.
### โฌ๏ธ Input
The only setting you need is **Start URLs**; everything else has a working default. Settings are grouped the same way in the form.
| Parameter | Type | Description |
|---|---|---|
| **๐ What to crawl** | | |
| `startUrls` | List | Pages where the crawl starts. Sub-pages under the same path are crawled too. A linked text file of URLs also works. |
| `searchQueries` | List | Optional search queries; the top results of each are crawled. |
| `searchEngine` | Select | `bing` (default) or `duckduckgo`. When it returns fewer results than asked, the other engine tops up the list. |
| `maxResultsPerSearch` | Integer | Results crawled per query (1-50). Default `3`. |
| `useSitemaps` / `sitemapUrls` | Boolean / List | Seed the crawl from robots.txt sitemaps, `/sitemap.xml`, indexes and `.gz` files, or from sitemap files you name. |
| `useLlmsTxt` | Boolean | Also crawl `/llms.txt`, `/llms-full.txt` and the Markdown links in them. |
| **๐ฏ Crawl scope** | | |
| `crawlMode` | Select | Pages under the start URL path (default), whole hostname, whole domain, every link, or only the start URLs. |
| `includeUrlGlobs` / `excludeUrlGlobs` | Globs | Crawl or skip links matching patterns such as `https://example.com/blog/**`. |
| `maxCrawlDepth` | Integer | Links followed from the start URLs. Default `20`; `0` = start URLs only. |
| `maxCrawlPages` | Integer | Pages crawled per run. Default `200`; `0` = no limit. |
| `maxPagesPerStartUrl`, `linkSelector` | Integer / Text | Per-start-URL cap; CSS selector of links to follow (default `a[href]`). |
| **โ๏ธ Refine results** | | |
| `onlyCrawlPagesModifiedSince` | Text | `2026-09-01` or `7 days`: only pages changed since then (304 answers, sitemap `lastmod`, page dates). |
| `minWords`, `articlesOnly`, `mustHaveDate` | | Skip short pages, non-article pages, or pages without a publish date. |
| `respectRobotsTxtFile` | Boolean | Skip URLs disallowed by robots.txt. Default `false`. |
| `keepUrlFragments`, `ignoreCanonicalUrl` | Boolean | Treat `#fragments` as pages; keep pages whose canonical URL was already delivered. |
| **๐ฅ๏ธ JavaScript rendering** | | |
| `renderJavaScript` | Select | `auto` (default: browser only for pages that are empty over HTTP), `never`, `always`. |
| `maxBrowserPages` | Integer | Cap on browser-rendered pages. `0` = 20% of Max pages (5 to 500). |
| `dynamicContentWaitSecs`, `waitForSelector`, `softWaitForSelector`, `maxScrollHeightPixels`, `clickElementsCssSelector`, `removeCookieWarnings`, `blockMedia`, `expandIframes` | | Browser behaviour, same names as Apify's Website Content Crawler. |
| **๐ Content extraction** | | |
| `htmlTransformer` | Select | `readableTextIfPossible` (default), `readableText`, `extractus`, `defuddle`, `none`. |
| `removeElementsCssSelector` / `keepElementsCssSelector` | Text | Elements removed before extraction, or the only elements kept. |
| `aggressivePrune` | Boolean | Drop text lines already seen on earlier pages (menus, footers). |
| **๐ฆ Output** | | |
| `outputFormats` | Multi-select | `markdown`, `text` (default), `html`, `raw`. |
| `chunkSize` / `chunkOverlap` | Integer | RAG chunk size in characters (`0` = off) and overlap (default `200`). |
| `saveHtmlAsFile`, `saveFiles`, `saveScreenshots` | Boolean | Store cleaned HTML, linked documents, screenshots in the key-value store. |
| `maxResults`, `compact`, `customMapFunction` | | Delivered-page cap; core fields only; a JavaScript function that reshapes each row. |
| **๐ Monitoring** | | |
| `monitorMode`, `monitorStoreName` | Boolean / Text | Only pages that are new or changed since the last run. Pair with a schedule. |
| **โ๏ธ Advanced** | | |
| `proxyConfiguration` | Proxy | Default: direct, with Apify Proxy for any site that refuses. Pick a group to proxy everything. |
| `customHttpHeaders`, `device`, `maxConcurrency`, `requestTimeoutSecs`, `maxRequestRetries`, `minFileDownloadSpeedKBps`, `ignoreHttpsErrors`, `storeSkippedUrls`, `debugMode` | | Network and debugging. |
#### Input JSON
```json
{
"startUrls": [{ "url": "https://docs.apify.com/academy/scraping-basics-javascript" }],
"maxCrawlPages": 100,
"outputFormats": ["markdown", "text"],
"chunkSize": 1500
}
```
#### Start at the section you want, not the homepage
The crawler follows links that sit under the start URL's path. **Do:** `https://example.com/docs/` to get the docs. **Don't:** `https://example.com/` when you only want the docs - it crawls marketing pages first. For a whole site, set **Which links to follow** to *Whole hostname*.
#### Use search queries for questions, start URLs for sites
A search query crawls only its top results (3 by default) and does not follow their links. Use it to collect fresh pages about a topic; use start URLs to index a known site.
#### Leave rendering on Auto
Most sites serve their content in the HTML, which is read in a fraction of a second. Auto sends only pages that come back empty to the browser. Choose *Always* for sites whose text appears only after their scripts run, on every page.
### โฌ๏ธ Output
Every crawled page becomes one dataset item. Here is the table view of a real run on docs.apify.com:
And one row as JSON (shortened):
```json
{
"url": "https://docs.apify.com/academy/scraping-basics-javascript",
"crawl": { "loadedUrl": "https://docs.apify.com/academy/scraping-basics-javascript", "depth": 0, "httpStatusCode": 200, "renderedWith": "http", "transformer": "readableText" },
"metadata": {
"title": "Web scraping basics for JavaScript devs | Academy | Apify Documentation",
"description": "Learn how to use JavaScript to extract information from websites in this practical course, starting from the absolute basics.",
"languageCode": "en",
"canonicalUrl": "https://docs.apify.com/academy/scraping-basics-javascript"
},
"markdown": "# Web scraping basics for JavaScript devs | Academy | Apify Documentation\n\n**Learn how to use JavaScript to extract information...",
"text": "Learn how to use JavaScript to extract information from websites in this practical course...",
"links": [{ "url": "https://docs.apify.com/academy/scraping-basics-javascript/devtools-inspecting", "text": "DevTools: Inspecting", "rel": null, "isInternal": true }],
"headings": [{ "level": 2, "text": "What we'll do" }],
"wordCount": 939,
"readingTimeSec": 237,
"contentHash": "168358ce2b473827c01e88cafd1720cf"
}
```
#### Fields
| Field | Type | What it is | Fill (100-page run) |
|---|---|---|---|
| `url` | string | URL as queued | 100% |
| `crawl` | object | `loadedUrl`, `loadedTime`, `referrerUrl`, `depth`, `httpStatusCode`, `contentType`, `contentLengthBytes`, `renderedWith`, `durationMs`, `transformer` | 100% |
| `metadata` | object | `title`, `description`, `author`, `keywords`, `languageCode`, `canonicalUrl`, `publishedAt`, `modifiedAt`, `publisher`, `copyright`, `favicon`, `image`, `type`, `tags`, `openGraph`, `jsonLd`, `headers` | title/description 100% |
| `markdown` / `text` | string | Main content | 100% |
| `html` / `htmlUrl` / `rawHtml` | string | Cleaned or original HTML, when requested | on request |
| `links` | array | `url`, `text`, `rel`, `isInternal` | 100% |
| `headings` / `tables` | array | Outline; tables as GFM Markdown | 99% / 3% |
| `images` / `videos` | array | Content images (`url`, `alt`); embedded videos | 59% / 1% |
| `wordCount`, `readingTimeSec`, `contentHash` | number / string | Size, reading time, change hash | 100% |
| `chunks` | array | `index`, `text`, `startChar`, `endChar`, `headingPath` | when `chunkSize` > 0 |
| `searchResult` | object | `query`, `title`, `description`, `url`, `rank`, `engine` | search results only |
| `file` | object | `type`, `pages`, `bytes`, `kvUrl` for PDF / DOCX | documents only |
| `isNew`, `changed`, `previousHash`, `previousCrawledAt` | | Monitor mode | monitor runs |
### How can I use website content data?
- ๐ค **AI and RAG teams** - index product docs, help centers and wikis into a vector database; read `markdown` or `chunks` with `headingPath` for citations
- ๐ฌ **Chatbot builders** - keep a support bot's knowledge current with a weekly monitor run that returns only changed pages
- ๐ **SEO and content teams** - audit titles, descriptions, canonical URLs, headings and word counts across a site
- ๐ **Researchers** - collect articles on a topic with search queries and keep their publish dates and authors
- ๐งพ **Compliance and legal teams** - archive terms, policies and PDFs and detect when their text changes (`contentHash`)
- ๐ ๏ธ **Developers** - get any page as Markdown in one API call for an agent or tool
### How much does it cost to crawl a website?
Website Content Crawler uses pay-per-event pricing; Apify platform usage is included in the event prices. There is no start fee.
| Event | Price | When it is charged |
|---|---:|---|
| Page crawled | **$0.0012** ($1.20 per 1,000 pages) | Once per page or document written to the dataset, read over HTTP or in the browser |
| Search query | $0.0033 | Once per search query that returns results |
| Screenshot | $0.001 | Once per stored screenshot (only with Save screenshots) |
| Pages | Cost |
|---:|---:|
| 10 | $0.012 |
| 100 | $0.12 |
| 1,000 | $1.20 |
| 10,000 | $12.00 |
Pages that fail to load, are blocked, redirect to a page already delivered, or have no text are not written to the dataset and are not charged. A search that returns nothing is not charged. Set **Max pages** (or a maximum cost per run in the run options) to cap a run's cost.
Measured runs on the default 1,024 MB: 10 pages in 6 s, 100 pages in 34 s, 975 pages in 15 minutes (a docs site with heavy API-reference pages); JavaScript pages rendered in the browser took about 2 seconds each.
### Integrations, API, scheduling & MCP
- **Schedules:** run Website Content Crawler daily or weekly from **Schedules**; turn on **Monitor mode** to get only new and changed pages.
- **AI frameworks:** load the dataset with the Apify integrations for [LangChain](https://docs.apify.com/platform/integrations/langchain), [LlamaIndex](https://docs.apify.com/platform/integrations/llama-index), [Pinecone](https://docs.apify.com/platform/integrations/pinecone) and [Qdrant](https://docs.apify.com/platform/integrations/qdrant).
- **No-code:** Make, Zapier, n8n, Google Sheets, Slack and webhooks via [Apify integrations](https://docs.apify.com/platform/integrations).
- **API:** call it from Node.js or Python with `apify-client` - code on the [API tab](https://apify.com/topapi/website-content-crawler/api/python).
- **MCP / AI agents:** connect Claude, Cursor or any MCP client through the [MCP tab](https://apify.com/topapi/website-content-crawler/api/mcp) and ask for a site's content in plain language; use `compact` for small rows.
### More data tools by TopApi