# 🌐 Website Content Crawler (`citrine_venus/website-content-crawler`) Actor

Crawl any website into clean text, Markdown & rich metadata — AI-ready output for RAG, LLMs & vector databases. Streams to your dataset in real time.

- **URL**: https://apify.com/citrine\_venus/website-content-crawler.md
- **Developed by:** [Data Minds](https://apify.com/citrine_venus) (community)
- **Categories:** AI, Developer tools, SEO tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $5.00 / 1,000 row\_results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md):

```bash
npm install apify-client
```

In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md):

```bash
pip install apify-client
```

In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md):

````bash
# MacOS / Linux
curl -fsSL https://apify.com/install-cli.sh | bash
# Windows
irm https://apify.com/install-cli.ps1 | iex
```bash

In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md).

If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).


# README

## 🌐 Website Content Crawler — Web Scraping & AI Content Extraction Actor

**Website Content Crawler** is a powerful [Apify Actor](https://docs.apify.com/platform/actors) for **website crawling**, **web scraping**, **content extraction**, and **AI data preparation**. It crawls any website starting from your seed URLs, follows internal links, extracts clean **plain text**, **Markdown**, and rich **metadata** (title, description, canonical URL, Open Graph, JSON-LD, language, headers), and writes structured page data to an Apify **Dataset** in real time — perfect for **RAG pipelines**, **LLM knowledge bases**, **vector database ingestion**, **semantic search**, and **SEO content audits**.

> 💡 **Need a custom version, private integration, or a tailored crawler?** Email **[hello.dataminds@gmail.com](mailto:hello.dataminds@gmail.com)**.

It is a focused alternative to the official [apify/website-content-crawler](https://apify.com/apify/website-content-crawler): a tighter feature surface, predictable cost, and AI-first output (text + Markdown + metadata) ready for **LangChain**, **LlamaIndex**, **OpenAI**, **Pinecone**, **Qdrant**, and **Weaviate** pipelines.

---

### 📑 Table of contents

- [What is Website Content Crawler?](#-what-is-website-content-crawler)
- [Main features](#-main-features)
- [Who is this Actor for?](#-who-is-this-actor-for)
- [Built for generative AI, LLMs, and RAG](#-built-for-generative-ai-llms-and-rag)
- [What the crawler does](#%EF%B8%8F-what-the-crawler-does)
- [Crawling configuration](#-crawling-configuration)
- [HTML cleaning & content extraction](#-html-cleaning--content-extraction)
- [Output format (Dataset)](#-output-format-dataset)
- [Example output (JSON)](#-example-output-json)
- [Quick start](#-quick-start)
- [Input parameters reference](#%EF%B8%8F-input-parameters-reference)
- [Integrations: LangChain, LlamaIndex, OpenAI, Pinecone](#-integrations-langchain-llamaindex-openai-pinecone)
- [Pricing & how to control cost](#-pricing--how-to-control-cost)
- [Frequently asked questions (FAQ)](#-frequently-asked-questions-faq)
- [Troubleshooting](#%EF%B8%8F-troubleshooting)
- [Help, support & custom builds](#-help-support--custom-builds)
- [Is web scraping legal?](#%EF%B8%8F-is-web-scraping-legal)
- [SEO keywords targeted](#-seo-keywords-targeted)

---

### 🌐 What is Website Content Crawler?

**Website Content Crawler** is an Apify-hosted **web scraping API** and **website crawler** that turns any public website into clean, structured, AI-ready data. Unlike generic HTTP scrapers, this Actor:

- **Crawls** an entire site from one or more start URLs, staying on the same registrable domain.
- **Renders JavaScript** automatically when a page needs it, so single-page apps still yield real content.
- **Cleans HTML** by stripping navigation, footers, scripts, cookie banners, modals, and other noise.
- **Extracts content** as readable plain text and optional **Markdown** for token-efficient LLM ingestion.
- **Captures metadata** (title, description, canonical, Open Graph, JSON-LD, headers, language).
- **Saves results** as structured rows in an Apify **Dataset**, exportable to JSON, CSV, Excel, or via the [Apify API](https://docs.apify.com/api/v2).

If you need a **website-to-dataset** pipeline for **AI agents**, **chatbot grounding**, **GPT knowledge files**, **RAG retrieval**, or **content analytics**, this is the Actor.

---

### ✨ Main features

- 🌱 **Multi-URL crawl seeds** — start from one URL or hundreds; the crawler stays on the same domain by default.
- 🎯 **Glob-based URL filtering** — `includeUrlGlobs` / `excludeUrlGlobs` for surgical scope (`/blog/*`, `/docs/**`, skip `/admin/**`). Path-only patterns work as written; `*` stops at `/`, `**` crosses folders.
- 🗺️ **Discovery sources** — seed the queue from `sitemap.xml` (`useSitemaps`, follows sitemap indexes) and `llms.txt` (`useLlmsTxt`) to reach pages that aren't linked, and honour `robots.txt` (`respectRobotsTxtFile`).
- 🛡️ **Apify Proxy support** — datacenter or residential groups, country targeting, or your own proxy URLs, with automatic fallback.
- 🍪 **Initial cookies** — `initialCookies` sent on every request and loaded into the browser for session or light-login access.
- 🖱️ **Rendered-page interaction** — `maxScrollHeightPixels` auto-scroll for lazy/infinite-scroll content, `clickElementsCssSelector` to expand accordions / "load more", and `blockMedia` to skip images/fonts/video for faster, cheaper renders.
- 🚫 **Ad/tracker blocking** — `blockAds` aborts known ad and tracker domains on rendered pages, on top of `blockMedia`.
- 📱 **Mobile emulation** — `mobileView` renders with a real mobile device profile (viewport, UA, touch) instead of desktop.
- 🕵️ **PII redaction (best-effort)** — `redactPii` masks emails, phone numbers, card-like numbers, and SSNs in `text`/`markdown`.
- 📐 **Screenshot mode** — `screenshotFullPage` (default on) toggles full-page vs viewport-only screenshots.
- 📕 **PDF text extraction** — `extractPdfText` opens discovered PDF links and extracts real text into `text`, instead of only listing them as file links.
- 📂 **Depth & page limits** — `maxCrawlDepth`, `maxCrawlPages`, `maxResults` to keep cost and runtime predictable.
- ⚡ **Async concurrency** — tune `maxConcurrency` from 1 to 50 parallel requests.
- 🖥️ **Three rendering modes** — `auto` (render only when a page needs it), `always`, or `never` for the cheapest possible run.
- 🪄 **Smart retry** — pages that come back incomplete are automatically retried for a fuller result.
- 🧼 **HTML noise removal** — strip nav, footer, modals, ads, and cookie banners via CSS selectors before extraction.
- 📝 **Markdown export** — token-efficient Markdown alongside plain text, ideal for **LLM context windows**.
- 🏷️ **Rich metadata extraction** — title, description, canonical URL, Open Graph, Twitter Card, JSON-LD, author, keywords, publish/modified dates, language code.
- 🧲 **Keep-only extraction** — `keepElementsCssSelector` to keep just an article body (`article, main`) and discard everything else before extraction.
- 🧾 **Skipped-URL audit** — `storeSkippedUrls` writes every discovered-but-not-crawled URL (with a reason) to `skipped-urls.json`.
- 📸 **HTML snapshots & full-page screenshots** stored in the run's key-value store, surfaced as **direct clickable URLs** on every row (original HTML, noise-removed HTML, readable HTML, PNG).
- ⏱️ **Politeness controls** — `rateLimitDelaySecs`, retry counts, request timeouts, and dynamic-content waits.
- 🎯 **Wait-for-selector** — block extraction until a CSS selector renders (great for React/Vue/Angular SPAs).
- 🛡️ **Custom HTTP headers** — User-Agent, sec-ch-ua, DNT, plus arbitrary cookies and auth headers via JSON.
- 🌐 **Canonical & fragment handling** — dedupe by canonical link or treat `#anchors` as distinct pages.
- 📦 **Real-time Dataset writes** — every page lands in your Dataset the moment it is ready, not at the end of the run.
- 🔧 **Verbose diagnostics** — link counts, status codes, content lengths, and snapshot links on every row.

---

### 👥 Who is this Actor for?

This Apify Actor is built for:

- 🤖 **AI engineers** building **RAG**, **retrieval-augmented generation**, or **LLM grounding** pipelines.
- 🧠 **Data scientists** assembling **training corpora**, **fine-tuning datasets**, or **embedding indexes**.
- 🛠️ **Backend developers** integrating **website-to-database** ETL flows or **knowledge sync** jobs.
- 📈 **SEO professionals** auditing **titles**, **meta descriptions**, **canonical URLs**, **Open Graph**, **schema.org** markup, and **internal linking** at scale.
- 📚 **Documentation teams** building **searchable docs portals** or **AI-powered help bots**.
- 🛒 **Content strategists & competitive analysts** monitoring competitor blogs, landing pages, and changelogs.
- 🔬 **Researchers** collecting **public web text data** for NLP, classification, or sentiment models.

If you've ever needed to "turn this whole website into a clean JSON file my LLM can read," this Actor is the answer.

---

### 🤖 Built for generative AI, LLMs, and RAG

This Actor is designed from the ground up for **AI workflows**:

- **RAG pipelines** — crawl docs, knowledge bases, FAQs, and blogs into chunkable Markdown.
- **LLM knowledge bases** — feed `text` + `metadata` into vector stores for grounded chatbot answers.
- **Semantic search engines** — index page content with embeddings for relevance-ranked retrieval.
- **Vector database ingestion** — drop directly into **Pinecone**, **Qdrant**, **Weaviate**, **Chroma**, **Milvus**, or **pgvector**.
- **Custom GPT knowledge files** — export Dataset JSON and upload to **ChatGPT custom GPTs**, **Claude Projects**, or **Anthropic Files API**.
- **AI agent training & grounding** — build **autonomous agents** with up-to-date web context.
- **Content operations** — summarization, translation, rewrite-assist, content moderation pipelines.
- **Competitive intelligence** — periodic crawls feeding analytics dashboards.

#### Practical AI use cases

1. **Customer support chatbot grounding** — crawl product docs, help center, and FAQ pages; embed `text + metadata` for retrieval.
2. **Documentation intelligence & semantic search** — turn product docs and changelogs into a searchable, embedding-indexed corpus.
3. **SEO and content audit automation** — analyze titles, meta descriptions, canonical links, and structured data sitewide.
4. **Custom GPT / Claude Project knowledge files** — generate a dataset JSON and upload to your AI assistant of choice.
5. **Internal LLM knowledge base** — sync intranet, wiki, or Notion-published content into a private vector store.
6. **Lead enrichment & company intelligence** — crawl prospect websites for descriptions, services, and contact pages.
7. **Compliance and content monitoring** — track changes to legal pages, terms of service, or pricing pages.

---

### ⚙️ What the crawler does

- **Crawls** from your seed URLs, follows internal links within domain scope, applies your `includeUrlGlobs` / `excludeUrlGlobs` filters, and respects `maxCrawlDepth`, `maxCrawlPages`, and `maxResults`.
- **Handles JavaScript sites** according to your `renderingMode` — `auto` renders only the pages that need it, `always` renders everything, `never` is the fastest and cheapest.
- **Cleans each page** with your CSS selectors (nav, footer, cookie banners, modals) before extracting text and Markdown.
- **Writes every row to the Apify Dataset the moment it is ready** — `url`, flat metadata columns, `crawl` info, `metadata`, `text`, `markdown`, snapshot links, and `screenshotUrl`. Downstream systems can consume rows while the crawl is still running.
- **Keeps going when a page fails** — bad URLs, non-HTML files, blocked pages, and oversized pages are handled per page, never by aborting the run.

---

### 🧭 Crawling configuration

- **Start URLs** are the only required input. Add as many as you like.
- By default, the crawler stays on the **same registrable domain** as your seeds — no accidental crawling of `facebook.com` from your blog's share buttons.
- **`includeUrlGlobs`** — keep only matching paths (e.g., `/blog/*`, `**/blog/**`).
- **`excludeUrlGlobs`** — skip paths (e.g., `/admin/**`, `**/login`, `**/*.pdf`).
- Filters apply to **discovered links only** — your start URLs are always crawled, so a narrow include pattern can't accidentally produce an empty run.
- **`useSitemaps`** — also seed the queue from `robots.txt`-advertised sitemaps and `/sitemap.xml` (following sitemap indexes). Reaches orphan pages that aren't linked from your seeds. Your include/exclude filters still apply.
- **`useLlmsTxt`** — also seed the queue from URLs listed in each start domain's `/llms.txt`.
- **`respectRobotsTxtFile`** — skip discovered URLs disallowed by the site's `robots.txt` (your explicit start URLs are always crawled).
- **`keepUrlFragments`** — treat `#anchors` as distinct URLs (off by default).
- **`ignoreCanonicalUrl`** — disable canonical-based dedup if you want every URL crawled distinctly.
- **`maxCrawlDepth`** — `0` = only seeds; `1` = seeds + their direct links; etc.
- **`maxCrawlPages`** — total unique pages opened (`0` = unlimited).
- **`maxResults`** — total successful records saved.

🧪 Test glob patterns with the [DigitalOcean glob tester](https://www.digitalocean.com/community/tools/glob).

---

### 🧼 HTML cleaning & content extraction

The Actor produces **clean, readable text** suitable for AI consumption:

- **`keepElementsCssSelector`** — keep **only** the elements matching this selector (e.g. `article, main`) and discard everything else. Applied before the remove list below; empty keeps the whole page.
- **`removeElementsCssSelector`** — comma-separated CSS selectors to strip before extraction (defaults remove nav, footer, scripts, cookie banners, modals, alerts).
- **`readableTextCharThreshold`** — how much text a page must have before it counts as complete.
- **`waitForSelector`** — pause until a key CSS selector appears (perfect for SPAs that mount content lazily).
- **`dynamicContentWaitSecs`** — extra settle time for late-arriving dynamic content.
- **`saveMarkdown`** — include a Markdown version of the page alongside plain text.

The result: **token-efficient, noise-free content** ready for embedding, indexing, or LLM context.

#### Rendered-page interaction

These apply only to rendered pages (`renderingMode: auto` when a page is rendered, or `always`):

- **`maxScrollHeightPixels`** — auto-scroll down up to this many pixels to trigger lazy-loaded / infinite-scroll content (`0` = off).
- **`clickElementsCssSelector`** — best-effort click on matching elements (accordions, "show more", expanders) before extracting; e.g. `[aria-expanded="false"]`.
- **`blockMedia`** — skip downloading images, fonts, and video while rendering for faster, cheaper runs (text and links unaffected).

---

### 📤 Output format (Dataset)

Results land in the default [Apify Dataset](https://docs.apify.com/platform/storage/dataset). You can:

- Browse rows in the Apify Console with multiple **prebuilt views** (Quick View, Crawl Details, Metadata, Content, Storage & Debug).
- Export to **JSON**, **CSV**, **Excel**, **XML**, **HTML**, or **RSS**.
- Pull data programmatically via the [Apify API](https://docs.apify.com/api/v2).
- Stream into **AI**, **ETL**, or **analytics** pipelines while the run is still active.

Each row includes:

| Field | Description |
|---|---|
| `orderNo` | 1-based order in which the row was saved during the run. |
| `url` | **The page's own URL** — its canonical URL when one exists (and `ignoreCanonicalUrl` is off), otherwise the loaded URL. |
| `loadedUrl` | The URL actually fetched, after redirects. |
| `referrerUrl` | The page that linked to this one (`referrerUrl` input for start URLs). |
| `depth` | Link distance from the nearest start URL (`0` for seeds). |
| `httpStatusCode` | HTTP status of the response. |
| `title`, `description`, `canonicalUrl`, `author`, `keywords`, `datePublished`, `dateModified`, `languageCode` | Flat metadata aliases so Dataset **table views** and CSV exports work without nesting. |
| `crawl` | Nested `loadedUrl`, `loadedTime`, `referrerUrl`, `depth`, `httpStatusCode`. |
| `metadata` | `canonicalUrl`, `title`, `description`, `author`, `keywords`, `datePublished`, `dateModified`, `languageCode`, `openGraph` (Open Graph + article + Twitter Card tags), `jsonLd`, `headers`. |
| `text` | Clean, readable plain text with paragraph structure preserved. |
| `markdown` | Real Markdown — headings, links, bold/italic, nested lists, blockquotes, GFM tables, fenced code (when `saveMarkdown` is on). |
| `html` | Cleaned readable HTML inline (when `saveHtml` is on). |
| `htmlUrl` | **Direct URL** to the readable HTML snapshot in the key-value store. |
| `screenshotUrl` | **Direct URL** to the full-page PNG (when `saveScreenshots` is on). |
| `files` | Non-HTML links found on the page (PDF, DOCX, XLSX, ZIP, …), or `null`. |
| `debug` | Per-row diagnostics — page/file link counts, response content type, snapshot URLs, and content byte lengths. |
| `debugTotalLinks`, `debugStatusCode` | Flat debug aliases for table views. |

> ℹ️ Rows over Apify's 9 MB item limit are trimmed automatically (`html` → `markdown` → `text`) and flagged with `debug.truncated: true`, so a single huge page never fails the run.

> 🧾 With `storeSkippedUrls: true`, discovered-but-not-crawled URLs are written to `skipped-urls.json` in the run's key-value store, each with a `reason` (`off-domain`, `excluded`, `not-included`, `robots-disallow`, `non-http`).

---

### 🧪 Example output (JSON)

Real output from a run against `https://docs.apify.com/` (text and Markdown abbreviated):

```json
{
  "orderNo": 4,
  "url": "https://docs.apify.com/proxy",
  "loadedUrl": "https://docs.apify.com/proxy",
  "loadedTime": "2026-07-23T14:31:52.015Z",
  "referrerUrl": "https://docs.apify.com/academy/scraping-basics-javascript",
  "depth": 1,
  "httpStatusCode": 200,
  "title": "Proxy | Platform | Apify Documentation",
  "description": "Learn to anonymously access websites in scraping/automation jobs.",
  "canonicalUrl": "https://docs.apify.com/proxy",
  "author": null,
  "keywords": null,
  "datePublished": null,
  "dateModified": null,
  "languageCode": "en",
  "crawl": {
    "loadedUrl": "https://docs.apify.com/proxy",
    "loadedTime": "2026-07-23T14:31:52.015Z",
    "referrerUrl": "https://docs.apify.com/academy/scraping-basics-javascript",
    "depth": 1,
    "httpStatusCode": 200
  },
  "metadata": {
    "canonicalUrl": "https://docs.apify.com/proxy",
    "title": "Proxy | Platform | Apify Documentation",
    "description": "Learn to anonymously access websites in scraping/automation jobs.",
    "author": null,
    "keywords": null,
    "datePublished": null,
    "dateModified": null,
    "languageCode": "en",
    "openGraph": [
      { "property": "og:url", "content": "https://docs.apify.com/proxy" },
      { "property": "og:locale", "content": "en" }
    ],
    "jsonLd": null,
    "headers": { "content-type": "text/html; charset=utf-8", "…": "…" }
  },
  "text": "Proxy\n\nApify Proxy lets you rotate IP addresses when scraping to avoid geographic blocking. …",
  "markdown": "# Proxy\n\n[Apify Proxy](https://apify.com/proxy) lets you rotate IP addresses when scraping …",
  "htmlUrl": "https://api.apify.com/v2/key-value-stores/<storeId>/records/kv__bfc3652cf15f843d170b_readable.html",
  "screenshotUrl": null,
  "files": null,
  "debug": {
    "totalLinks": 59,
    "pageLinks": 59,
    "fileLinks": 0,
    "response": { "statusCode": 200, "contentType": "text/html; charset=utf-8" },
    "snapshots": {
      "originalHtmlUrl": "https://api.apify.com/v2/key-value-stores/<storeId>/records/kv__bfc3652cf15f843d170b_original.html",
      "removedElementsHtmlUrl": "https://api.apify.com/v2/key-value-stores/<storeId>/records/kv__bfc3652cf15f843d170b_removed.html",
      "readableTextHtmlUrl": "https://api.apify.com/v2/key-value-stores/<storeId>/records/kv__bfc3652cf15f843d170b_readable.html"
    },
    "lengths": {
      "originalHtml": 75279,
      "readableHtml": 43168,
      "readableText": 10763,
      "markdown": 19203
    }
  }
}
````

***

### 🚀 Quick start

#### Run in Apify Console

1. Open the Actor in [Apify Console](https://console.apify.com/).
2. Paste one or more **start URLs** (e.g., `https://docs.apify.com/`).
3. (Optional) Set **`maxResults`**, **`maxCrawlDepth`**, **`maxCrawlPages`**, and URL filters.
4. Click **Start** and watch rows stream into your Dataset.
5. Export as **JSON / CSV** or pull via the [Apify API](https://docs.apify.com/api/v2).

#### Run via API

```bash
curl -X POST "https://api.apify.com/v2/acts/<your-actor-id>/runs?token=<APIFY_TOKEN>"   -H "Content-Type: application/json"   -d '{
        "startUrls": ["https://docs.apify.com/"],
        "maxResults": 25,
        "maxCrawlDepth": 2,
        "renderingMode": "auto",
        "saveMarkdown": true
      }'
```

Then read the rows from the run's dataset:

```bash
curl "https://api.apify.com/v2/actor-runs/<runId>/dataset/items?format=json&token=<APIFY_TOKEN>"
```

***

### 🛠️ Input parameters reference

| Parameter | Type | Default | Description |
|---|---|---|---|
| `startUrls` | array | — | **Required.** Seed URLs. Always crawled — URL filters apply to discovered links only. |
| `referrerUrl` | string | `""` | Value recorded as `referrerUrl` on rows produced from the start URLs. |
| `blockAds` | bool | `false` | Abort known ad/tracker domains on rendered pages. |
| `mobileView` | bool | `false` | Render with a mobile device profile (viewport, UA, touch) instead of desktop. |
| `redactPii` | bool | `false` | Mask emails, phone numbers, card-like numbers, and SSNs in `text`/`markdown` (best-effort). |
| `screenshotFullPage` | bool | `true` | Full-page vs viewport-only screenshots. |
| `extractPdfText` | bool | `false` | Open discovered PDF links and extract real text into `text` instead of only listing them as files. |
| `proxyConfiguration` | object | `{}` | Apify Proxy (datacenter/residential/country) or your own proxy URLs. |
| `includeUrlGlobs` | array | `[]` | Keep only discovered URLs matching these globs (`/blog/*`, `/docs/**`). |
| `excludeUrlGlobs` | array | `[]` | Skip discovered URLs matching these globs (`/admin/**`, `**/*.pdf`). |
| `useSitemaps` | bool | `false` | Seed the queue from `robots.txt`-advertised sitemaps and `/sitemap.xml` (follows sitemap indexes). |
| `useLlmsTxt` | bool | `false` | Seed the queue from URLs listed in each start domain's `/llms.txt`. |
| `respectRobotsTxtFile` | bool | `false` | Skip discovered URLs disallowed by `robots.txt` (seeds always crawled). |
| `keepUrlFragments` | bool | `false` | Treat `#anchors` as distinct URLs. |
| `ignoreCanonicalUrl` | bool | `false` | Use the loaded URL as `url` instead of the canonical one. |
| `maxCrawlDepth` | int | `2` | Link-following depth from seeds (`0` = seeds only). |
| `maxCrawlPages` | int | `0` | Hard cap on unique pages opened (`0` = unlimited). |
| `maxResults` | int | `10` | Cap on successful saved rows. |
| `maxConcurrency` | int | `10` | Parallel page workers (1–50). |
| `rateLimitDelaySecs` | string | `0.1` | Throttle before each fetch; accepts decimals. |
| `maxRequestRetries` | int | `3` | Total attempts per URL (backoff; honours `Retry-After` on 429). |
| `requestTimeoutSecs` | int | `60` | Request / navigation timeout. |
| `dynamicContentWaitSecs` | int | `3` | **Real** settle time in seconds after a rendered page loads. |
| `waitForSelector` | string | — | Wait for this CSS selector; a missing selector triggers a more thorough retry. |
| `renderingMode` | enum | `auto` | `auto` (render only when a page needs it) · `always` · `never` (cheapest). |
| `retryFailedWithRendering` | bool | `true` | Give a failed page one more, more thorough attempt. |
| `blockMedia` | bool | `false` | Skip images/fonts/video on rendered pages for faster, cheaper runs. |
| `maxScrollHeightPixels` | int | `0` | Auto-scroll rendered pages up to N pixels for lazy/infinite-scroll content (`0` = off). |
| `clickElementsCssSelector` | string | — | Best-effort click on matching elements (expanders / "load more") before extracting rendered pages. |
| `ignoreHttpsErrors` | bool | `false` | Accept bad certificates (dev/staging only). |
| `keepElementsCssSelector` | string | — | Keep only elements matching this selector (e.g. `article, main`); empty keeps the whole page. |
| `removeElementsCssSelector` | string | (sensible default) | CSS selectors stripped before extraction. |
| `readableTextCharThreshold` | int | `100` | Below this many characters a page counts as "thin" and is retried. |
| `kvDir` | string | `kv` | Storage prefix for snapshots & screenshots (empty = skip storage writes). |
| `saveScreenshots` | bool | `false` | Full-page PNG per row. **Requires rendering** every page. |
| `saveMarkdown` | bool | `true` | Include Markdown in each row. |
| `saveHtml` | bool | `true` | Include cleaned HTML inline; turn off for much smaller rows. |
| `headerAcceptLanguage` | string | `en-US,en;q=0.9` | `Accept-Language` header. |
| `customHttpHeaders` | object | `{}` | Extra HTTP headers (cookies, auth, lang) — these override everything else. |
| `initialCookies` | array | `[]` | Cookies sent on every request and loaded into the browser: `[{"name":"...","value":"...","domain":"...","path":"/"}]`. |
| `debugLog` | bool | `true` | Drops the run log to DEBUG level. |
| `storeSkippedUrls` | bool | `false` | Write discovered-but-not-crawled URLs (with reasons) to `skipped-urls.json` in the key-value store. |

Plus browser-identity headers (`headerUserAgent`, `headerSecChUa`, `headerSecChUaMobile`, `headerSecChUaPlatform`, `headerDnt`, `headerUpgradeInsecureRequests`). The defaults give the best success rate on most sites; set a custom `headerUserAgent` or `headerSecChUa` and your values are used verbatim instead.

***

### 🔌 Integrations: LangChain, LlamaIndex, OpenAI, Pinecone

This Actor's Dataset output plugs directly into the modern AI stack:

- 🦜 **LangChain** — use `ApifyDatasetLoader` or load JSON into `Document` objects with `text` + `metadata`.
- 🦙 **LlamaIndex** — feed Markdown rows into `MarkdownReader` or build a `VectorStoreIndex`.
- 🧠 **OpenAI Embeddings & Assistants** — chunk `text` / `markdown`, embed with `text-embedding-3`, and upload as **OpenAI File** for Assistants API or Custom GPT knowledge.
- 🤖 **Anthropic Claude** — upload Dataset JSON to **Claude Projects**, the **Files API**, or use as RAG context.
- 📍 **Pinecone**, **Qdrant**, **Weaviate**, **Chroma**, **Milvus**, **pgvector** — embed and upsert with `metadata` filters.
- 📊 **n8n**, **Make.com**, **Zapier** — wire crawl results into automations and notifications.
- 📦 **Snowflake**, **BigQuery**, **PostgreSQL**, **MongoDB**, **S3** — load Dataset JSON/CSV into your warehouse or lake.
- 🧰 **Apify API & SDKs** — orchestrate runs from your own services in any language.

See [Apify Publishing & integration docs](https://docs.apify.com/platform/actors/publishing) for platform-level guidance.

***

### 💸 Pricing & how to control cost

#### Billable event

This Actor charges one **pay-per-event** event:

| Event | When it fires | What you get |
|---|---|---|
| `row_result` | Once per page record written to the Dataset | A complete page row — text, Markdown, metadata, snapshot links |

Pages that fail, are skipped (non-HTML, `404`, filtered by globs), or are dropped after the `maxResults` cap **are not charged**. If the run hits its charging limit (`ACTOR_MAX_TOTAL_CHARGE_USD`), the crawl stops cleanly and everything collected so far stays in the Dataset.

> ⚙️ **Publishing note:** the `row_result` event must exist in *Apify Console → your Actor → Publication → Monetization* with the same name. Until pay-per-event is configured, the Actor still runs and saves data — it just logs `Ignored attempt to charge for an event`.

#### Compute cost

Cost on Apify also scales with **compute units** consumed during a run. The biggest drivers are:

- 📄 **Number of pages crawled** (`maxResults`, `maxCrawlPages`).
- 🖥️ **Rendering mode** — `always` is far heavier than `auto`, and `auto` is heavier than `never`.
- 🧮 **Concurrency** (`maxConcurrency`) and timeout settings.
- 🌐 **Page complexity** and network conditions.

#### To control cost, tune:

- `maxResults` — hard cap on saved rows.
- `maxCrawlPages` — hard cap on URLs opened.
- `maxCrawlDepth` — keep crawls shallow when possible.
- `maxConcurrency` — balance speed vs. politeness.
- `renderingMode: "never"` — the cheapest possible run.
- `blockMedia: true` — skip images/fonts/video on rendered pages to cut render time and bandwidth.
- `blockAds: true` — skip known ad/tracker requests on rendered pages.
- `screenshotFullPage: false` — capture only the viewport instead of the whole scrolled page.
- `retryFailedWithRendering: false` — skip the second attempt on pages that fail once.
- `rateLimitDelaySecs` — throttle to avoid being rate-limited (which wastes retries).
- `saveScreenshots` — leave **off** unless you need images; enabling it renders every page.
- `saveHtml: false` — drops the inline `html` field for much smaller (and cheaper to store) rows.

> 💡 **Tip:** start with `maxResults: 10` for testing, then scale up once your filters and selectors are dialed in.

***

### ❓ Frequently asked questions (FAQ)

#### What is the best web scraper for AI and LLM training data?

A web scraper for **AI / LLM training data** must produce **clean text or Markdown**, preserve **metadata**, handle **JavaScript-rendered pages**, and export to a structured format (JSON, JSONL, Parquet). Website Content Crawler does all four — it strips boilerplate HTML, outputs both plain text and Markdown, captures rich metadata, and writes directly to an Apify Dataset that exports as JSON/CSV.

#### How do I scrape a website for a RAG pipeline?

1. Run Website Content Crawler with your docs/help/blog URLs as `startUrls`.
2. Set `saveMarkdown: true` and tune `removeElementsCssSelector` to strip nav/footer.
3. Export the Dataset as JSON.
4. Chunk each `markdown` field, embed with OpenAI / Cohere / Voyage / local models, and upsert to Pinecone/Qdrant/Weaviate.
5. At query time, retrieve top-K chunks and pass them as LLM context.

#### Can this Actor scrape JavaScript-heavy single-page applications (SPAs)?

Yes. `renderingMode: "auto"` already handles pages that need rendering automatically; set `renderingMode: "always"` when every page must be rendered. Use `waitForSelector` for content that mounts after the initial load, and `dynamicContentWaitSecs` for late-arriving content — it is a genuine wait in seconds.

#### How do I export the crawl results?

Open the run in Apify Console → **Dataset** tab → **Export** → choose JSON, CSV, Excel, XML, HTML, or RSS. You can also pull data programmatically via the [Apify Dataset API](https://docs.apify.com/api/v2#/reference/datasets).

#### Does this Actor respect robots.txt?

Optionally, yes — set `respectRobotsTxtFile: true` and discovered URLs disallowed by the site's `robots.txt` are skipped (your explicit start URLs are always crawled). It is **off by default**. Either way, **you** remain responsible for compliance with `robots.txt`, website terms of service, and applicable laws; you can also add `excludeUrlGlobs` to skip specific paths.

#### How do I avoid being rate-limited or blocked?

- Lower `maxConcurrency` (start at 2–5).
- Increase `rateLimitDelaySecs` (e.g., `1.0`).
- Set realistic `headerUserAgent` and `customHttpHeaders` (cookies, language).
- Enable `proxyConfiguration`, or set `renderingMode: "always"` for sites with aggressive bot detection.

#### Can I crawl multiple websites in one run?

Yes — add multiple URLs to `startUrls`. By default the crawler stays on **each seed's own domain**, so you'll get multi-domain coverage.

#### How is this different from the official `apify/website-content-crawler`?

This Actor keeps the input surface small and the cost predictable: three rendering modes instead of a matrix of engines, Markdown and metadata by default, screenshots and HTML snapshots as direct links, and one billable event you can reason about. The official Apify Actor has a much larger feature surface. Choose this one when you want **simpler input, predictable cost, and AI-first defaults**; choose the official one when you need its full enterprise feature set.

#### What output formats are supported?

JSON, JSONL, CSV, Excel, XML, HTML, RSS — all via standard Apify Dataset exports.

#### Does it support proxies?

Yes — use the **`proxyConfiguration`** input to pick [Apify Proxy](https://docs.apify.com/platform/proxy) datacenter or residential groups, a country, or your own proxy URLs. Requests are spread across several proxy sessions automatically. If the chosen proxy is unavailable, the run continues instead of failing.

#### Can I schedule recurring crawls?

Yes — use [Apify Schedules](https://docs.apify.com/platform/schedules) to run this Actor hourly, daily, or weekly, and pipe new rows into your warehouse, vector DB, or notification channel.

***

### 🛠️ Troubleshooting

| Symptom | Likely cause | Fix |
|---|---|---|
| **No rows saved** | Bad `startUrls`, over-restrictive filters, target offline | Verify `startUrls`, relax `includeUrlGlobs` / `excludeUrlGlobs`, check site availability |
| **Crawl too slow** | Rendering every page + high retries | Lower `maxConcurrency`, switch `renderingMode` to `auto` or `never` |
| **Missing content / empty `text`** | JS-rendered SPA, content behind selector | Set `waitForSelector`, increase `dynamicContentWaitSecs`, set `renderingMode: "always"` |
| **Boilerplate in output** | Default `removeElementsCssSelector` doesn't cover site's nav | Add site-specific selectors (e.g., `.sidebar, .related-posts, .ad-slot`) |
| **Duplicate rows** | Canonical handling off, fragments treated as distinct | Enable canonical (default), set `keepUrlFragments: false` |
| **Cert errors on staging** | Self-signed / expired cert | Set `ignoreHttpsErrors: true` (dev only) |
| **Blocked / 403 / 429** | Bot detection, rate limit | Enable `proxyConfiguration` (residential helps most), increase `rateLimitDelaySecs`, lower `maxConcurrency`, try `renderingMode: "always"` |
| **No screenshots in output** | `saveScreenshots` off, `kvDir` empty, or `renderingMode: "never"` | Set `saveScreenshots: true`, keep a `kvDir` prefix, and use `auto` or `always` |
| **Only the start URL is crawled** | `includeUrlGlobs` doesn't match any discovered link, or `maxCrawlDepth: 0` | Widen the globs (`/section/**`) and raise `maxCrawlDepth` |
| **Rows are huge / slow to export** | `html` field included | Set `saveHtml: false` and use `htmlUrl` instead |

***

### 💬 Help, support & custom builds

For **custom solutions**, **private integrations**, **white-label crawlers**, or **implementation help**:

📧 **<hello.dataminds@gmail.com>**

Useful Apify documentation:

- 📘 [Apify documentation](https://docs.apify.com/)
- 🤖 [Actors overview](https://docs.apify.com/platform/actors)
- 🔌 [Apify API reference](https://docs.apify.com/api/v2)
- 📚 [Apify Academy — web scraping](https://docs.apify.com/academy)
- 🧰 [Apify SDKs](https://docs.apify.com/sdk)

***

### ⚖️ Is web scraping legal?

Use this Actor **only on pages you are allowed to access**. Respect:

- ✅ `robots.txt` directives,
- ✅ website terms of service,
- ✅ rate limits and politeness norms,
- ✅ data-protection laws (GDPR, CCPA, etc.) in your jurisdiction,
- ✅ copyright and licensing of extracted content.

You are responsible for **compliant data collection, storage, and use**. When in doubt, consult a qualified attorney.

***

### 🔍 SEO keywords targeted

This README is intentionally keyword-rich for discovery on **Google Search**, **Apify Store search**, and **Bing**. Primary and secondary terms covered include:

**Primary:** website content crawler · web scraping · website crawler · content extraction · web scraping API · Apify Actor · website scraper · site crawler

**Content & format:** Markdown extraction · plain text extraction · metadata extraction · structured data extraction · HTML cleaner · readable text extractor · canonical URL extraction · Open Graph scraper · JSON-LD extraction · schema.org scraper

**AI & LLM:** RAG pipeline · retrieval-augmented generation · LLM training data scraper · vector database ingestion · semantic search crawler · embedding pipeline · custom GPT knowledge file · Claude Projects data · LangChain data loader · LlamaIndex reader · AI agent grounding

**Capabilities:** headless rendering · JavaScript rendering scraper · SPA scraper · React scraper · dynamic content scraper · concurrent crawler · anti-blocking scraper · sitemap crawler · llms.txt crawler · robots.txt compliant crawler · infinite scroll scraper

**Use cases:** documentation scraper · blog crawler · SEO content audit · competitive intelligence crawler · website-to-JSON · website-to-database · website-to-Markdown · knowledge base scraper · help center crawler · changelog scraper

**Integrations:** Pinecone ingestion · Qdrant ingestion · Weaviate · Chroma · pgvector · Snowflake · BigQuery · n8n · Make.com · Zapier · OpenAI Embeddings · Anthropic Files API

***

*Reference patterns for structure and section ordering:*

- 📘 Public Actor: [apify/website-content-crawler](https://apify.com/apify/website-content-crawler)
- 📗 Apify publishing guidance: [Publishing and monetization](https://docs.apify.com/platform/actors/publishing)
- 🔐 Console (access-gated): [Console README](https://console.apify.com/actors/aYG0l9s7dbB7j3gbS/information/version-0/readme)

# Actor input Schema

## `startUrls` (type: `array`):

📝 **Required.** Paste the page(s) where crawling should begin. Example: your docs home, a blog index, or a product category. URL filters below are applied to *discovered* links only — your seeds are always crawled.

## `referrerUrl` (type: `string`):

🏷️ Recorded as `referrerUrl` on rows produced from your start URLs (they have no natural “came from” page). Discovered pages always record the page that linked to them. Leave blank for an empty referrer.

## `blockAds` (type: `boolean`):

🛡️ Abort requests to known ad/tracker hosts (doubleclick, Google Ads, Taboola, Criteo, and more) while rendering — cleaner, faster pages. Only affects rendered pages.

## `mobileView` (type: `boolean`):

📲 Render pages with a mobile viewport and device profile instead of desktop. Useful for sites that serve different content to mobile visitors. A custom User-Agent above still overrides the mobile one.

## `redactPii` (type: `boolean`):

🔒 Mask emails, phone numbers, credit-card-like numbers, and SSNs in `text` and `markdown` before saving. Regex-based heuristics — not a compliance guarantee.

## `screenshotFullPage` (type: `boolean`):

🖼️ When **on** (default), screenshots capture the entire scrolled page. Turn **off** to capture just the visible viewport — faster and smaller. Only applies when `saveScreenshots` is on.

## `extractPdfText` (type: `boolean`):

📄 When **on**, discovered PDF links are opened and their text is extracted into `text` (counted toward your page/result limits like any other page) instead of only being listed as a file link.

## `includeUrlGlobs` (type: `array`):

🧩 **Include patterns.** If you add any, *only* discovered URLs matching at least one pattern are queued. Examples: `/blog/*` · `/docs/**` · `https://site.com/product/**`. Empty = no extra filter.

## `excludeUrlGlobs` (type: `array`):

🛑 URLs matching **any** pattern here are skipped — handy for login, admin, or file folders. Examples: `/admin/**` · `/login` · `**/*.pdf`.

## `useSitemaps` (type: `boolean`):

When **on**, the crawler reads `robots.txt`-advertised sitemaps and `/sitemap.xml` (following sitemap indexes) and queues those URLs — great for reaching pages that aren't linked from your start page. Your include/exclude filters still apply.

## `useLlmsTxt` (type: `boolean`):

When **on**, the crawler reads `/llms.txt` at each start domain and queues the URLs it lists. Your include/exclude filters still apply.

## `respectRobotsTxtFile` (type: `boolean`):

When **on**, discovered URLs disallowed by the site's `robots.txt` are skipped. Your explicit start URLs are always crawled.

## `keepUrlFragments` (type: `boolean`):

📌 Normally `#section` is ignored. Turn **on** if you need each fragment to count as its own URL.

## `ignoreCanonicalUrl` (type: `boolean`):

🔀 When **off**, the page's canonical URL becomes the row's `url` and drives deduplication. When **on**, the loaded URL wins.

## `maxCrawlDepth` (type: `integer`):

0️⃣ = only your start URLs · 1️⃣ = their direct links · 2️⃣ = one more hop… and so on. Great for staying on-topic!

## `maxCrawlPages` (type: `integer`):

🛡️ Hard cap on how many **unique pages** may be opened, including pages that turn out to be empty or blocked. **0** means “no extra cap” (max rows below still applies).

## `maxResults` (type: `integer`):

🎁 Hard stop on how many **successful page records** are saved — perfect for sampling or quick tests.

## `maxConcurrency` (type: `integer`):

🔄 How many pages load **at the same time**. Higher = faster, but be kind to small sites — start moderate.

## `rateLimitDelaySecs` (type: `string`):

🌊 Throttle applied before each fetch, e.g. `0.1` or `1.5`. Use `0` to go as fast as your other limits allow. Non-numeric values fall back to `0`.

## `maxRequestRetries` (type: `integer`):

♻️ Total fetch **attempts** for a URL before giving up (`3` = one try plus two retries, with exponential backoff). `429` responses honour `Retry-After`.

## `requestTimeoutSecs` (type: `integer`):

⏲️ Max time to wait for a page to respond — useful on slow or heavy sites.

## `dynamicContentWaitSecs` (type: `integer`):

📺 Real settle time **after** a rendered page finishes loading, so late-arriving content is captured. Only applies to rendered pages. Higher values cost more compute.

## `waitForSelector` (type: `string`):

✨ Wait until this **CSS selector** appears before extracting — ideal for key content that pops in late. A page missing the selector is treated as incomplete and collected again more thoroughly.

## `renderingMode` (type: `string`):

**Auto** is recommended: most pages are collected quickly, and only the ones that need it are rendered. **Always** renders every page (most reliable for JavaScript-heavy sites, slowest). **Never** is the cheapest and fastest, but cannot capture screenshots or JavaScript-built content.

## `retryFailedWithRendering` (type: `boolean`):

💫 If a page cannot be collected on the first attempt, try it once more with rendering. Great for flaky or protected pages. Ignored when the rendering mode is *Never*.

## `blockMedia` (type: `boolean`):

⚡ Skip downloading images, fonts and video while rendering — faster and cheaper on heavy pages. Only affects rendered pages; text and links are unaffected.

## `maxScrollHeightPixels` (type: `integer`):

🖱️ On rendered pages, scroll down up to this many pixels to trigger lazy-loaded / infinite-scroll content before extracting. `0` disables scrolling. Only applies to rendered pages.

## `clickElementsCssSelector` (type: `string`):

🔽 On rendered pages, click elements matching this selector (accordions, “show more”, expanders) before extracting. Best-effort — unclickable elements are skipped. Example: `[aria-expanded="false"]`.

## `ignoreHttpsErrors` (type: `boolean`):

🔓 **Off** by default (secure). Turn **on** only for local/dev sites with certificate issues — not for production.

## `keepElementsCssSelector` (type: `string`):

🧲 If set, **only** the elements matching this selector are kept — everything else is discarded before extraction. Ideal for grabbing just an article body, e.g. `article, main`. Leave empty to keep the whole page. Applied before the remove list below.

## `removeElementsCssSelector` (type: `string`):

Comma-separated selectors. Commas inside `[attr="a,b"]` or `:is(a,b)` are handled correctly, and an invalid selector is skipped with a warning instead of failing the page.

## `readableTextCharThreshold` (type: `integer`):

📊 A page whose cleaned text is shorter than this counts as **too thin** and, in *Auto* rendering mode, is collected again more thoroughly.

## `kvDir` (type: `string`):

Prefix for saved records (e.g. `kv`). Characters that are illegal in key-value store keys are replaced automatically. Leave **empty** to skip writing files entirely.

## `saveScreenshots` (type: `boolean`):

🖼️ Captures a full-page **PNG** for every saved row. ⚠️ Screenshots require rendering every page, so runs are slower and cost more. Needs a storage prefix and a rendering mode other than *Never*.

## `saveMarkdown` (type: `boolean`):

📒 Adds a real **Markdown** rendering (headings, links, lists, tables, code fences) alongside plain text — turn off if you only need raw text.

## `saveHtml` (type: `boolean`):

🧱 Embeds the cleaned readable HTML in the `html` field. Turn **off** for much smaller dataset rows — the snapshot links in `htmlUrl` still work.

## `proxyConfiguration` (type: `object`):

Pick Apify Proxy (datacenter or residential) or supply custom proxy URLs. If the selected proxy is unavailable, the run continues rather than failing.

## `headerAcceptLanguage` (type: `string`):

Preferred content language, e.g. `en-US,en;q=0.9` or `de-DE,de;q=0.9`.

## `headerDnt` (type: `string`):

Typical value `1`. Leave empty to omit the header.

## `headerUpgradeInsecureRequests` (type: `string`):

Standard browser hint — usually `1`. Leave empty to omit.

## `headerUserAgent` (type: `string`):

🎭 Leave at the default for the best success rate. Change it only if a site needs a specific User-Agent — a custom value is then used verbatim.

## `headerSecChUa` (type: `string`):

Client hint header — keep the default unless you also customise the User-Agent.

## `headerSecChUaMobile` (type: `string`):

Usually `?0` for desktop mode.

## `headerSecChUaPlatform` (type: `string`):

e.g. `"Windows"` for desktop.

## `customHttpHeaders` (type: `object`):

🧩 Merge **custom headers** (cookies, auth, language). These win over every header above. Invalid JSON fails validation.

## `initialCookies` (type: `array`):

🔐 Cookies sent with every request and loaded into the browser — useful for session or light login. Each item: `{ "name": "...", "value": "...", "domain": "...", "path": "/" }` (`domain`/`path` optional). Example: `[{"name":"sessionid","value":"abc123"}]`.

## `debugLog` (type: `boolean`):

📣 When **on**, the run log drops to DEBUG level and prints queue decisions, fallbacks, and parse notes. Noisier, but great for troubleshooting.

## `storeSkippedUrls` (type: `boolean`):

📄 When **on**, URLs that were discovered but not crawled (off-domain, filtered out, robots-disallowed) are written to `skipped-urls.json` in the key-value store, with a reason for each.

## Actor input object example

```json
{
  "startUrls": [
    "https://docs.apify.com/academy/scraping-basics-javascript"
  ],
  "blockAds": false,
  "mobileView": false,
  "redactPii": false,
  "screenshotFullPage": true,
  "extractPdfText": false,
  "includeUrlGlobs": [],
  "excludeUrlGlobs": [],
  "useSitemaps": false,
  "useLlmsTxt": false,
  "respectRobotsTxtFile": false,
  "keepUrlFragments": false,
  "ignoreCanonicalUrl": false,
  "maxCrawlDepth": 2,
  "maxCrawlPages": 0,
  "maxResults": 10,
  "maxConcurrency": 10,
  "rateLimitDelaySecs": "0.1",
  "maxRequestRetries": 3,
  "requestTimeoutSecs": 60,
  "dynamicContentWaitSecs": 3,
  "renderingMode": "auto",
  "retryFailedWithRendering": true,
  "blockMedia": false,
  "maxScrollHeightPixels": 0,
  "ignoreHttpsErrors": false,
  "removeElementsCssSelector": "nav, footer, script, style, noscript, svg, img[src^='data:'], [role=\"alert\"], [role=\"banner\"], [role=\"dialog\"], [role=\"alertdialog\"], [role=\"region\"][aria-label*=\"skip\" i], [aria-modal=\"true\"]",
  "readableTextCharThreshold": 100,
  "kvDir": "kv",
  "saveScreenshots": false,
  "saveMarkdown": true,
  "saveHtml": true,
  "proxyConfiguration": {
    "useApifyProxy": true
  },
  "headerAcceptLanguage": "en-US,en;q=0.9",
  "headerDnt": "1",
  "headerUpgradeInsecureRequests": "1",
  "headerUserAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36",
  "headerSecChUa": "\"Google Chrome\";v=\"143\", \"Chromium\";v=\"143\", \"Not A(Brand\";v=\"24\"",
  "headerSecChUaMobile": "?0",
  "headerSecChUaPlatform": "\"Windows\"",
  "customHttpHeaders": {},
  "initialCookies": [],
  "debugLog": true,
  "storeSkippedUrls": false
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://docs.apify.com/academy/scraping-basics-javascript"
    ],
    "referrerUrl": "",
    "includeUrlGlobs": [],
    "excludeUrlGlobs": [],
    "rateLimitDelaySecs": "0.1",
    "waitForSelector": "",
    "clickElementsCssSelector": "",
    "keepElementsCssSelector": "",
    "kvDir": "kv",
    "proxyConfiguration": {
        "useApifyProxy": true
    },
    "headerAcceptLanguage": "en-US,en;q=0.9",
    "headerDnt": "1",
    "headerUpgradeInsecureRequests": "1",
    "headerUserAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36",
    "headerSecChUa": "\"Google Chrome\";v=\"143\", \"Chromium\";v=\"143\", \"Not A(Brand\";v=\"24\"",
    "headerSecChUaMobile": "?0",
    "headerSecChUaPlatform": "\"Windows\"",
    "customHttpHeaders": {},
    "initialCookies": []
};

// Run the Actor and wait for it to finish
const run = await client.actor("citrine_venus/website-content-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://docs.apify.com/academy/scraping-basics-javascript"],
    "referrerUrl": "",
    "includeUrlGlobs": [],
    "excludeUrlGlobs": [],
    "rateLimitDelaySecs": "0.1",
    "waitForSelector": "",
    "clickElementsCssSelector": "",
    "keepElementsCssSelector": "",
    "kvDir": "kv",
    "proxyConfiguration": { "useApifyProxy": True },
    "headerAcceptLanguage": "en-US,en;q=0.9",
    "headerDnt": "1",
    "headerUpgradeInsecureRequests": "1",
    "headerUserAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36",
    "headerSecChUa": "\"Google Chrome\";v=\"143\", \"Chromium\";v=\"143\", \"Not A(Brand\";v=\"24\"",
    "headerSecChUaMobile": "?0",
    "headerSecChUaPlatform": "\"Windows\"",
    "customHttpHeaders": {},
    "initialCookies": [],
}

# Run the Actor and wait for it to finish
run = client.actor("citrine_venus/website-content-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://docs.apify.com/academy/scraping-basics-javascript"
  ],
  "referrerUrl": "",
  "includeUrlGlobs": [],
  "excludeUrlGlobs": [],
  "rateLimitDelaySecs": "0.1",
  "waitForSelector": "",
  "clickElementsCssSelector": "",
  "keepElementsCssSelector": "",
  "kvDir": "kv",
  "proxyConfiguration": {
    "useApifyProxy": true
  },
  "headerAcceptLanguage": "en-US,en;q=0.9",
  "headerDnt": "1",
  "headerUpgradeInsecureRequests": "1",
  "headerUserAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36",
  "headerSecChUa": "\\"Google Chrome\\";v=\\"143\\", \\"Chromium\\";v=\\"143\\", \\"Not A(Brand\\";v=\\"24\\"",
  "headerSecChUaMobile": "?0",
  "headerSecChUaPlatform": "\\"Windows\\"",
  "customHttpHeaders": {},
  "initialCookies": []
}' |
apify call citrine_venus/website-content-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=citrine_venus/website-content-crawler",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

```json
{
    "openapi": "3.0.1",
    "info": {
        "title": "🌐 Website Content Crawler",
        "description": "Crawl any website into clean text, Markdown & rich metadata — AI-ready output for RAG, LLMs & vector databases. Streams to your dataset in real time.",
        "version": "1.1",
        "x-build-id": "uPZ6EcAT89iYOKx10"
    },
    "servers": [
        {
            "url": "https://api.apify.com/v2"
        }
    ],
    "paths": {
        "/acts/citrine_venus~website-content-crawler/run-sync-get-dataset-items": {
            "post": {
                "operationId": "run-sync-get-dataset-items-citrine_venus-website-content-crawler",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK"
                    }
                }
            }
        },
        "/acts/citrine_venus~website-content-crawler/runs": {
            "post": {
                "operationId": "runs-sync-citrine_venus-website-content-crawler",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor and returns information about the initiated run in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK",
                        "content": {
                            "application/json": {
                                "schema": {
                                    "$ref": "#/components/schemas/runsResponseSchema"
                                }
                            }
                        }
                    }
                }
            }
        },
        "/acts/citrine_venus~website-content-crawler/run-sync": {
            "post": {
                "operationId": "run-sync-citrine_venus-website-content-crawler",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK"
                    }
                }
            }
        }
    },
    "components": {
        "schemas": {
            "inputSchema": {
                "type": "object",
                "required": [
                    "startUrls"
                ],
                "properties": {
                    "startUrls": {
                        "title": "🌱 Start URLs",
                        "type": "array",
                        "description": "📝 **Required.** Paste the page(s) where crawling should begin. Example: your docs home, a blog index, or a product category. URL filters below are applied to *discovered* links only — your seeds are always crawled.",
                        "items": {
                            "type": "string"
                        }
                    },
                    "referrerUrl": {
                        "title": "🔗 Default referrer (optional)",
                        "type": "string",
                        "description": "🏷️ Recorded as `referrerUrl` on rows produced from your start URLs (they have no natural “came from” page). Discovered pages always record the page that linked to them. Leave blank for an empty referrer."
                    },
                    "blockAds": {
                        "title": "🚫 Block ad & tracker domains (rendered pages)",
                        "type": "boolean",
                        "description": "🛡️ Abort requests to known ad/tracker hosts (doubleclick, Google Ads, Taboola, Criteo, and more) while rendering — cleaner, faster pages. Only affects rendered pages.",
                        "default": false
                    },
                    "mobileView": {
                        "title": "📱 Emulate a mobile browser (rendered pages)",
                        "type": "boolean",
                        "description": "📲 Render pages with a mobile viewport and device profile instead of desktop. Useful for sites that serve different content to mobile visitors. A custom User-Agent above still overrides the mobile one.",
                        "default": false
                    },
                    "redactPii": {
                        "title": "🕵️ Redact personal info (best-effort)",
                        "type": "boolean",
                        "description": "🔒 Mask emails, phone numbers, credit-card-like numbers, and SSNs in `text` and `markdown` before saving. Regex-based heuristics — not a compliance guarantee.",
                        "default": false
                    },
                    "screenshotFullPage": {
                        "title": "📐 Full-page screenshots",
                        "type": "boolean",
                        "description": "🖼️ When **on** (default), screenshots capture the entire scrolled page. Turn **off** to capture just the visible viewport — faster and smaller. Only applies when `saveScreenshots` is on.",
                        "default": true
                    },
                    "extractPdfText": {
                        "title": "📕 Extract text from linked PDFs",
                        "type": "boolean",
                        "description": "📄 When **on**, discovered PDF links are opened and their text is extracted into `text` (counted toward your page/result limits like any other page) instead of only being listed as a file link.",
                        "default": false
                    },
                    "includeUrlGlobs": {
                        "title": "✅ Only crawl URLs matching…",
                        "type": "array",
                        "description": "🧩 **Include patterns.** If you add any, *only* discovered URLs matching at least one pattern are queued. Examples: `/blog/*` · `/docs/**` · `https://site.com/product/**`. Empty = no extra filter.",
                        "items": {
                            "type": "string"
                        }
                    },
                    "excludeUrlGlobs": {
                        "title": "🚫 Skip URLs matching…",
                        "type": "array",
                        "description": "🛑 URLs matching **any** pattern here are skipped — handy for login, admin, or file folders. Examples: `/admin/**` · `/login` · `**/*.pdf`.",
                        "items": {
                            "type": "string"
                        }
                    },
                    "useSitemaps": {
                        "title": "🗺️ Seed from sitemap.xml",
                        "type": "boolean",
                        "description": "When **on**, the crawler reads `robots.txt`-advertised sitemaps and `/sitemap.xml` (following sitemap indexes) and queues those URLs — great for reaching pages that aren't linked from your start page. Your include/exclude filters still apply.",
                        "default": false
                    },
                    "useLlmsTxt": {
                        "title": "📄 Seed from llms.txt",
                        "type": "boolean",
                        "description": "When **on**, the crawler reads `/llms.txt` at each start domain and queues the URLs it lists. Your include/exclude filters still apply.",
                        "default": false
                    },
                    "respectRobotsTxtFile": {
                        "title": "🤖 Respect robots.txt",
                        "type": "boolean",
                        "description": "When **on**, discovered URLs disallowed by the site's `robots.txt` are skipped. Your explicit start URLs are always crawled.",
                        "default": false
                    },
                    "keepUrlFragments": {
                        "title": "🔖 Treat `#anchors` as different pages",
                        "type": "boolean",
                        "description": "📌 Normally `#section` is ignored. Turn **on** if you need each fragment to count as its own URL.",
                        "default": false
                    },
                    "ignoreCanonicalUrl": {
                        "title": "🧭 Ignore canonical `<link>` tags",
                        "type": "boolean",
                        "description": "🔀 When **off**, the page's canonical URL becomes the row's `url` and drives deduplication. When **on**, the loaded URL wins.",
                        "default": false
                    },
                    "maxCrawlDepth": {
                        "title": "📂 Max crawl depth",
                        "minimum": 0,
                        "maximum": 50,
                        "type": "integer",
                        "description": "0️⃣ = only your start URLs · 1️⃣ = their direct links · 2️⃣ = one more hop… and so on. Great for staying on-topic!",
                        "default": 2
                    },
                    "maxCrawlPages": {
                        "title": "🔢 Max pages to visit (0 = unlimited)",
                        "minimum": 0,
                        "maximum": 1000000,
                        "type": "integer",
                        "description": "🛡️ Hard cap on how many **unique pages** may be opened, including pages that turn out to be empty or blocked. **0** means “no extra cap” (max rows below still applies).",
                        "default": 0
                    },
                    "maxResults": {
                        "title": "📬 Max rows in your dataset",
                        "minimum": 1,
                        "maximum": 100000,
                        "type": "integer",
                        "description": "🎁 Hard stop on how many **successful page records** are saved — perfect for sampling or quick tests.",
                        "default": 10
                    },
                    "maxConcurrency": {
                        "title": "⚡ Parallel pages",
                        "minimum": 1,
                        "maximum": 50,
                        "type": "integer",
                        "description": "🔄 How many pages load **at the same time**. Higher = faster, but be kind to small sites — start moderate.",
                        "default": 10
                    },
                    "rateLimitDelaySecs": {
                        "title": "⏳ Pause between requests (seconds)",
                        "type": "string",
                        "description": "🌊 Throttle applied before each fetch, e.g. `0.1` or `1.5`. Use `0` to go as fast as your other limits allow. Non-numeric values fall back to `0`."
                    },
                    "maxRequestRetries": {
                        "title": "🔁 Attempts per page",
                        "minimum": 1,
                        "maximum": 20,
                        "type": "integer",
                        "description": "♻️ Total fetch **attempts** for a URL before giving up (`3` = one try plus two retries, with exponential backoff). `429` responses honour `Retry-After`.",
                        "default": 3
                    },
                    "requestTimeoutSecs": {
                        "title": "⌛ Page load timeout (seconds)",
                        "minimum": 5,
                        "maximum": 300,
                        "type": "integer",
                        "description": "⏲️ Max time to wait for a page to respond — useful on slow or heavy sites.",
                        "default": 60
                    },
                    "dynamicContentWaitSecs": {
                        "title": "🎬 Extra wait for dynamic sites (seconds)",
                        "minimum": 0,
                        "maximum": 120,
                        "type": "integer",
                        "description": "📺 Real settle time **after** a rendered page finishes loading, so late-arriving content is captured. Only applies to rendered pages. Higher values cost more compute.",
                        "default": 3
                    },
                    "waitForSelector": {
                        "title": "🎯 Wait for CSS selector (optional)",
                        "type": "string",
                        "description": "✨ Wait until this **CSS selector** appears before extracting — ideal for key content that pops in late. A page missing the selector is treated as incomplete and collected again more thoroughly."
                    },
                    "renderingMode": {
                        "title": "🎮 Rendering mode",
                        "enum": [
                            "auto",
                            "always",
                            "never"
                        ],
                        "type": "string",
                        "description": "**Auto** is recommended: most pages are collected quickly, and only the ones that need it are rendered. **Always** renders every page (most reliable for JavaScript-heavy sites, slowest). **Never** is the cheapest and fastest, but cannot capture screenshots or JavaScript-built content.",
                        "default": "auto"
                    },
                    "retryFailedWithRendering": {
                        "title": "🆘 Second chance if a page won't load",
                        "type": "boolean",
                        "description": "💫 If a page cannot be collected on the first attempt, try it once more with rendering. Great for flaky or protected pages. Ignored when the rendering mode is *Never*.",
                        "default": true
                    },
                    "blockMedia": {
                        "title": "🚫 Block images, fonts & media (rendered pages)",
                        "type": "boolean",
                        "description": "⚡ Skip downloading images, fonts and video while rendering — faster and cheaper on heavy pages. Only affects rendered pages; text and links are unaffected.",
                        "default": false
                    },
                    "maxScrollHeightPixels": {
                        "title": "📜 Auto-scroll height (pixels, 0 = off)",
                        "minimum": 0,
                        "maximum": 200000,
                        "type": "integer",
                        "description": "🖱️ On rendered pages, scroll down up to this many pixels to trigger lazy-loaded / infinite-scroll content before extracting. `0` disables scrolling. Only applies to rendered pages.",
                        "default": 0
                    },
                    "clickElementsCssSelector": {
                        "title": "👆 Click elements before extracting (CSS selector)",
                        "type": "string",
                        "description": "🔽 On rendered pages, click elements matching this selector (accordions, “show more”, expanders) before extracting. Best-effort — unclickable elements are skipped. Example: `[aria-expanded=\"false\"]`."
                    },
                    "ignoreHttpsErrors": {
                        "title": "⚠️ Allow insecure / bad certs (dev only)",
                        "type": "boolean",
                        "description": "🔓 **Off** by default (secure). Turn **on** only for local/dev sites with certificate issues — not for production.",
                        "default": false
                    },
                    "keepElementsCssSelector": {
                        "title": "🎯 Keep only elements (CSS selectors)",
                        "type": "string",
                        "description": "🧲 If set, **only** the elements matching this selector are kept — everything else is discarded before extraction. Ideal for grabbing just an article body, e.g. `article, main`. Leave empty to keep the whole page. Applied before the remove list below."
                    },
                    "removeElementsCssSelector": {
                        "title": "🗑️ Remove elements (CSS selectors)",
                        "type": "string",
                        "description": "Comma-separated selectors. Commas inside `[attr=\"a,b\"]` or `:is(a,b)` are handled correctly, and an invalid selector is skipped with a warning instead of failing the page.",
                        "default": "nav, footer, script, style, noscript, svg, img[src^='data:'], [role=\"alert\"], [role=\"banner\"], [role=\"dialog\"], [role=\"alertdialog\"], [role=\"region\"][aria-label*=\"skip\" i], [aria-modal=\"true\"]"
                    },
                    "readableTextCharThreshold": {
                        "title": "📏 “Enough text?” sensitivity",
                        "minimum": 0,
                        "maximum": 100000,
                        "type": "integer",
                        "description": "📊 A page whose cleaned text is shorter than this counts as **too thin** and, in *Auto* rendering mode, is collected again more thoroughly.",
                        "default": 100
                    },
                    "kvDir": {
                        "title": "📁 Storage prefix",
                        "type": "string",
                        "description": "Prefix for saved records (e.g. `kv`). Characters that are illegal in key-value store keys are replaced automatically. Leave **empty** to skip writing files entirely."
                    },
                    "saveScreenshots": {
                        "title": "📸 Save full-page screenshots",
                        "type": "boolean",
                        "description": "🖼️ Captures a full-page **PNG** for every saved row. ⚠️ Screenshots require rendering every page, so runs are slower and cost more. Needs a storage prefix and a rendering mode other than *Never*.",
                        "default": false
                    },
                    "saveMarkdown": {
                        "title": "📝 Include Markdown in each row",
                        "type": "boolean",
                        "description": "📒 Adds a real **Markdown** rendering (headings, links, lists, tables, code fences) alongside plain text — turn off if you only need raw text.",
                        "default": true
                    },
                    "saveHtml": {
                        "title": "📄 Include cleaned HTML in each row",
                        "type": "boolean",
                        "description": "🧱 Embeds the cleaned readable HTML in the `html` field. Turn **off** for much smaller dataset rows — the snapshot links in `htmlUrl` still work.",
                        "default": true
                    },
                    "proxyConfiguration": {
                        "title": "🌍 Proxy configuration",
                        "type": "object",
                        "description": "Pick Apify Proxy (datacenter or residential) or supply custom proxy URLs. If the selected proxy is unavailable, the run continues rather than failing.",
                        "default": {}
                    },
                    "headerAcceptLanguage": {
                        "title": "🗣️ Accept-Language",
                        "type": "string",
                        "description": "Preferred content language, e.g. `en-US,en;q=0.9` or `de-DE,de;q=0.9`."
                    },
                    "headerDnt": {
                        "title": "🛡️ DNT (Do Not Track)",
                        "type": "string",
                        "description": "Typical value `1`. Leave empty to omit the header."
                    },
                    "headerUpgradeInsecureRequests": {
                        "title": "⬆️ Upgrade insecure requests",
                        "type": "string",
                        "description": "Standard browser hint — usually `1`. Leave empty to omit."
                    },
                    "headerUserAgent": {
                        "title": "🧩 User-Agent string",
                        "type": "string",
                        "description": "🎭 Leave at the default for the best success rate. Change it only if a site needs a specific User-Agent — a custom value is then used verbatim."
                    },
                    "headerSecChUa": {
                        "title": "🔒 sec-ch-ua",
                        "type": "string",
                        "description": "Client hint header — keep the default unless you also customise the User-Agent."
                    },
                    "headerSecChUaMobile": {
                        "title": "📱 sec-ch-ua-mobile",
                        "type": "string",
                        "description": "Usually `?0` for desktop mode."
                    },
                    "headerSecChUaPlatform": {
                        "title": "🪟 sec-ch-ua-platform",
                        "type": "string",
                        "description": "e.g. `\"Windows\"` for desktop."
                    },
                    "customHttpHeaders": {
                        "title": "➕ Extra HTTP headers (JSON)",
                        "type": "object",
                        "description": "🧩 Merge **custom headers** (cookies, auth, language). These win over every header above. Invalid JSON fails validation."
                    },
                    "initialCookies": {
                        "title": "🍪 Initial cookies (JSON)",
                        "type": "array",
                        "description": "🔐 Cookies sent with every request and loaded into the browser — useful for session or light login. Each item: `{ \"name\": \"...\", \"value\": \"...\", \"domain\": \"...\", \"path\": \"/\" }` (`domain`/`path` optional). Example: `[{\"name\":\"sessionid\",\"value\":\"abc123\"}]`.",
                        "items": {
                            "type": "object"
                        }
                    },
                    "debugLog": {
                        "title": "🗣️ Verbose logging",
                        "type": "boolean",
                        "description": "📣 When **on**, the run log drops to DEBUG level and prints queue decisions, fallbacks, and parse notes. Noisier, but great for troubleshooting.",
                        "default": true
                    },
                    "storeSkippedUrls": {
                        "title": "🧾 Save skipped URLs",
                        "type": "boolean",
                        "description": "📄 When **on**, URLs that were discovered but not crawled (off-domain, filtered out, robots-disallowed) are written to `skipped-urls.json` in the key-value store, with a reason for each.",
                        "default": false
                    }
                }
            },
            "runsResponseSchema": {
                "type": "object",
                "properties": {
                    "data": {
                        "type": "object",
                        "properties": {
                            "id": {
                                "type": "string"
                            },
                            "actId": {
                                "type": "string"
                            },
                            "userId": {
                                "type": "string"
                            },
                            "startedAt": {
                                "type": "string",
                                "format": "date-time",
                                "example": "2025-01-08T00:00:00.000Z"
                            },
                            "finishedAt": {
                                "type": "string",
                                "format": "date-time",
                                "example": "2025-01-08T00:00:00.000Z"
                            },
                            "status": {
                                "type": "string",
                                "example": "READY"
                            },
                            "meta": {
                                "type": "object",
                                "properties": {
                                    "origin": {
                                        "type": "string",
                                        "example": "API"
                                    },
                                    "userAgent": {
                                        "type": "string"
                                    }
                                }
                            },
                            "stats": {
                                "type": "object",
                                "properties": {
                                    "inputBodyLen": {
                                        "type": "integer",
                                        "example": 2000
                                    },
                                    "rebootCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "restartCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "resurrectCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "computeUnits": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            },
                            "options": {
                                "type": "object",
                                "properties": {
                                    "build": {
                                        "type": "string",
                                        "example": "latest"
                                    },
                                    "timeoutSecs": {
                                        "type": "integer",
                                        "example": 300
                                    },
                                    "memoryMbytes": {
                                        "type": "integer",
                                        "example": 1024
                                    },
                                    "diskMbytes": {
                                        "type": "integer",
                                        "example": 2048
                                    }
                                }
                            },
                            "buildId": {
                                "type": "string"
                            },
                            "defaultKeyValueStoreId": {
                                "type": "string"
                            },
                            "defaultDatasetId": {
                                "type": "string"
                            },
                            "defaultRequestQueueId": {
                                "type": "string"
                            },
                            "buildNumber": {
                                "type": "string",
                                "example": "1.0.0"
                            },
                            "containerUrl": {
                                "type": "string"
                            },
                            "usage": {
                                "type": "object",
                                "properties": {
                                    "ACTOR_COMPUTE_UNITS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_WRITES": {
                                        "type": "integer",
                                        "example": 1
                                    },
                                    "KEY_VALUE_STORE_LISTS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_INTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_EXTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_SERPS": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            },
                            "usageTotalUsd": {
                                "type": "number",
                                "example": 0.00005
                            },
                            "usageUsd": {
                                "type": "object",
                                "properties": {
                                    "ACTOR_COMPUTE_UNITS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_WRITES": {
                                        "type": "number",
                                        "example": 0.00005
                                    },
                                    "KEY_VALUE_STORE_LISTS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_INTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_EXTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_SERPS": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}
```