# Website to RAG Dataset (`sebastian-actors/website-to-rag-dataset`) Actor

Convert public websites, docs, blogs, and XML sitemaps into clean Markdown, structured metadata, and stable chunks for RAG pipelines and vector databases.

- **URL**: https://apify.com/sebastian-actors/website-to-rag-dataset.md
- **Developed by:** [Sebastian](https://apify.com/sebastian-actors) (community)
- **Categories:** AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.50 / 1,000 processed pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md):

```bash
npm install apify-client
```

In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md):

```bash
pip install apify-client
```

In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md):

````bash
# MacOS / Linux
curl -fsSL https://apify.com/install-cli.sh | bash
# Windows
irm https://apify.com/install-cli.ps1 | iex
```bash

In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md).

If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).


# README

## Website to RAG Dataset

Convert public websites, documentation, blogs, and XML sitemaps into clean Markdown, structured page metadata, and stable chunks for RAG pipelines and vector databases.

The Actor uses bounded static HTTP fetching. It does not render JavaScript, follow page links recursively, create embeddings, or use proxies.

### What it does

- Accepts direct page URLs, an XML sitemap, or a sitemap index.
- Enforces page, sitemap, concurrency, timeout, redirect, and response-size limits.
- Blocks private, loopback, link-local, reserved, and mixed public/private DNS destinations.
- Respects `robots.txt` by default.
- Removes common navigation and layout noise.
- Converts the selected page content to Markdown.
- Extracts canonical URLs, titles, descriptions, headings, language, robots metadata, Open Graph data, and Twitter metadata.
- Extracts JSON-LD types, breadcrumbs, article dates, HTTP `Last-Modified`, and sitemap `lastmod` values.
- Creates deterministic, query-sensitive chunk IDs and bounded overlapping chunks.
- Writes pages/errors and RAG chunks to separate datasets.
- Saves an aggregate run summary to the `OUTPUT` key-value-store record.

### Input

Provide `urls`, `sitemapUrl`, or both:

```json
{
  "sitemapUrl": "https://example.com/sitemap.xml",
  "maxPages": 100,
  "maxSitemaps": 100,
  "sameDomainOnly": true,
  "respectRobotsTxt": true,
  "outputMode": "pages_and_chunks",
  "includeMarkdown": true,
  "includeSchema": true,
  "includeFreshnessSignals": true,
  "chunkSize": 1200,
  "chunkOverlap": 150,
  "maxConcurrency": 5
}
````

At least one URL or sitemap is required. Only public HTTP and HTTPS destinations are allowed. The Actor revalidates every redirect and every DNS result before connecting.

#### Output modes

- `pages`: successful page rows and skipped/error rows in the default dataset.
- `chunks`: successful chunks in the `chunks` dataset; skipped/error page rows remain in the default dataset.
- `pages_and_chunks`: successful pages in the default dataset and successful chunks in the `chunks` dataset.

Chunk creation is derived from `outputMode`; there is no separate chunk toggle.

#### Important controls

`maxPages` limits eligible source pages. `maxSitemaps` independently limits XML sitemap files, including child sitemaps.

`urlAllowlistPatterns` and `urlBlocklistPatterns` are regular expressions applied before page fetching. Filtered sitemap URLs produce capped skipped rows so filtering remains inspectable without creating an unbounded dataset.

`onlySelectors` selects known content containers. `removeSelectors` removes unwanted elements after content selection.

`chunkSize` and `chunkOverlap` are approximate tokens using a characters/4 estimate. Oversized paragraphs are split so emitted chunks remain within `chunkSize`.

### Outputs

#### Pages and errors

The default dataset contains page-level results and all skipped/error rows:

```json
{
  "itemType": "page",
  "url": "https://example.com/docs/getting-started",
  "finalUrl": "https://example.com/docs/getting-started",
  "canonicalUrl": "https://example.com/docs/getting-started",
  "status": "ok",
  "statusCode": 200,
  "metadata": {
    "title": "Getting Started",
    "description": "Learn how to get started.",
    "language": "en",
    "h1": "Getting Started",
    "headings": [
      { "level": 1, "text": "Getting Started" },
      { "level": 2, "text": "Install" }
    ]
  },
  "freshness": {
    "lastModifiedHeader": "2026-06-21T15:42:00.000Z",
    "sitemapLastmod": "2026-06-20T00:00:00.000Z",
    "dateModified": "2026-06-21T00:00:00.000Z"
  },
  "content": {
    "markdown": "# Getting Started\n\nThis guide explains...",
    "contentChars": 4820,
    "estimatedTokens": 1205,
    "textHash": "sha256:...",
    "markdownHash": "sha256:...",
    "lowContent": false
  },
  "chunks": {
    "count": 2,
    "chunkIds": ["example-com-docs-getting-started-a1b2c3d4e5f6-000"]
  },
  "error": null
}
```

Unsafe, filtered, robots-disallowed, duplicate-canonical, budget-limited, empty chunks-only, non-HTML, and failed targets use `status: "skipped"` or `status: "error"` with a structured `error` object.

#### RAG chunks

The `chunks` dataset contains only chunk rows:

```json
{
  "itemType": "chunk",
  "url": "https://example.com/docs/getting-started",
  "canonicalUrl": "https://example.com/docs/getting-started",
  "chunkId": "example-com-docs-getting-started-a1b2c3d4e5f6-000",
  "chunkIndex": 0,
  "chunkText": "# Getting Started\n\nThis guide explains...",
  "headingPath": ["Getting Started"],
  "charLength": 3900,
  "estimatedTokens": 975,
  "textHash": "sha256:...",
  "metadata": {
    "title": "Getting Started",
    "language": "en",
    "schemaTypes": ["TechArticle", "BreadcrumbList"]
  }
}
```

Chunk IDs combine a readable URL prefix, a hash of the complete normalized source URL, and the chunk index. Query-string variants therefore do not collide.

#### Run summary

The `OUTPUT` record reports discovery, selection, processing, filtering, billing, chunks, errors, domains, warnings, and applied cost guards:

```json
{
  "pagesDiscovered": 120,
  "pagesSelected": 100,
  "pagesAttempted": 100,
  "pagesProcessed": 96,
  "pagesFiltered": 20,
  "pagesSkipped": 2,
  "errors": 2,
  "chunksCreated": 214,
  "chargedPages": 96,
  "billingEnabled": true,
  "sitemapsFetched": 3,
  "averageContentChars": 4210,
  "domains": ["example.com"],
  "warnings": [],
  "costGuardrails": {
    "maxPagesApplied": true,
    "maxSitemapsApplied": false,
    "filteredRowsTruncated": true,
    "browserRenderingUsed": false,
    "proxyUsed": false,
    "recursiveCrawlingUsed": false,
    "embeddingsUsed": false
  }
}
```

`chargedPages` is zero during local and non-PPE runs. It must equal the number of successfully delivered source pages in a configured PPE cloud run.

### Content extraction

The Actor selects content in this order:

1. User-provided `onlySelectors`
2. `<main>`
3. `<article>`
4. `[role="main"]`
5. Common documentation, post, and content selectors
6. `<body>`

It removes navigation, headers, footers, scripts, styles, forms, iframes, and configured selectors before Markdown conversion.

### Pricing

The publication model is pay per successfully processed source page using the custom `page-processed` event. Errors, skips, duplicate canonicals, empty chunks-only pages, and budget-limited targets are not charged. Chunks are never charged individually.

The Actor does not manually charge the synthetic `apify-actor-start` event and must not enable `apify-default-dataset-item`, which would incorrectly charge default-dataset error rows.

The pre-publish cloud profiles processed 100 pages each with no errors:

| Profile | Platform cost | Cost per page |
| --- | ---: | ---: |
| Apify Docs, pages and chunks | $0.005891 | $0.0000589 |
| Apify Blog, pages and chunks | $0.006108 | $0.0000611 |
| Apify Docs, chunks only | $0.004385 | $0.0000438 |

The guarded cost is `1.5 x $0.0000611 = $0.0000916` per page. Twice that guarded cost rounds up to the minimum `$0.0005` increment, so the final `page-processed` prices remain `$0.004/$0.003/$0.002/$0.0015` for the FREE/BRONZE/SILVER/GOLD tiers. These prices exceed the required 30% post-share margin under the guarded cost.

### Limitations

- Static server-rendered HTML only; no JavaScript rendering.
- No login-gated, private-network, paywalled, or authenticated pages.
- No recursive page-link crawling.
- No PDFs, browser screenshots, embeddings, vector-database writes, proxies, webhooks, or historical diffs.
- Sites may still reject the Actor's user agent or rate-limit requests.
- Main-content extraction is heuristic; use selectors when a site needs explicit targeting.

### Legal and responsible use

Only process public pages you are allowed to use. Respect website terms, copyright, privacy, database rights, and applicable law. `robots.txt` checks are enabled by default but do not replace the user's legal responsibilities.

### Troubleshooting

If output is empty, inspect the default page/error dataset first. It records URL safety blocks, filters, robots decisions, HTTP failures, non-HTML responses, canonical duplicates, and chunks-only pages without usable content.

If Markdown contains layout noise, add `removeSelectors`. If the main article is missing, add a precise `onlySelectors` value.

If a sitemap stops early, inspect `warnings`, `maxPagesApplied`, and `maxSitemapsApplied` in `OUTPUT`.

### Local development

```bash
npm install
npm test
npm run build
apify validate-schema
apify run --purge
```

# Actor input Schema

## `urls` (type: `array`):

List of public web pages to process. You can provide individual URLs, docs pages, blog posts, or landing pages.

## `sitemapUrl` (type: `string`):

Optional XML sitemap or sitemap index URL. If provided, URLs from the sitemap are processed up to Max pages.

## `maxPages` (type: `integer`):

Maximum number of pages to process.

## `sameDomainOnly` (type: `boolean`):

Only process URLs on the same hostname as the first URL or sitemap URL.

## `respectRobotsTxt` (type: `boolean`):

Check robots.txt and skip URLs disallowed for the configured user agent.

## `outputMode` (type: `string`):

Choose whether to output page rows, chunk rows, or both.

## `includeMarkdown` (type: `boolean`):

Include cleaned Markdown in page output rows.

## `includeSchema` (type: `boolean`):

Extract JSON-LD schema types and key schema fields where available.

## `includeFreshnessSignals` (type: `boolean`):

Extract Last-Modified, sitemap lastmod, datePublished, and dateModified signals when available.

## `includeLinks` (type: `boolean`):

Include internal and external links found in the main content.

## `includeHtml` (type: `boolean`):

Include cleaned HTML in page rows. Disabled by default to keep output smaller.

## `chunkSize` (type: `integer`):

Approximate chunk size in estimated tokens. Token estimate uses a simple characters/4 heuristic.

## `chunkOverlap` (type: `integer`):

Approximate overlap between chunks in estimated tokens.

## `minContentChars` (type: `integer`):

Pages with less cleaned content than this value are marked as low-content.

## `maxConcurrency` (type: `integer`):

Number of pages fetched in parallel.

## `requestTimeoutSecs` (type: `integer`):

Timeout for each page request.

## `maxSitemaps` (type: `integer`):

Maximum number of sitemap and sitemap-index files to fetch.

## `userAgent` (type: `string`):

User agent used for requests and robots.txt checks.

## `urlAllowlistPatterns` (type: `array`):

Optional regex patterns. If provided, only matching URLs are processed.

## `urlBlocklistPatterns` (type: `array`):

Optional regex patterns. Matching URLs are skipped.

## `removeSelectors` (type: `array`):

CSS selectors to remove before Markdown conversion.

## `onlySelectors` (type: `array`):

Optional CSS selectors for the content area. If provided, only matched elements are used.

## `dedupeByCanonical` (type: `boolean`):

Avoid processing duplicate pages with the same canonical URL.

## `saveRunSummaryToKeyValueStore` (type: `boolean`):

Save run-level summary statistics to the default key-value store under OUTPUT.

## Actor input object example

```json
{
  "urls": [
    "https://docs.apify.com/actors"
  ],
  "sitemapUrl": "https://docs.apify.com/sitemap.xml",
  "maxPages": 100,
  "sameDomainOnly": true,
  "respectRobotsTxt": true,
  "outputMode": "pages_and_chunks",
  "includeMarkdown": true,
  "includeSchema": true,
  "includeFreshnessSignals": true,
  "includeLinks": false,
  "includeHtml": false,
  "chunkSize": 1200,
  "chunkOverlap": 150,
  "minContentChars": 250,
  "maxConcurrency": 5,
  "requestTimeoutSecs": 20,
  "maxSitemaps": 100,
  "userAgent": "WebsiteToRagDatasetBot/1.0 (+https://apify.com)",
  "urlAllowlistPatterns": [],
  "urlBlocklistPatterns": [],
  "removeSelectors": [
    "nav",
    "footer",
    "header",
    "aside",
    "script",
    "style",
    "noscript",
    "iframe",
    "form"
  ],
  "onlySelectors": [],
  "dedupeByCanonical": true,
  "saveRunSummaryToKeyValueStore": true
}
```

# Actor output Schema

## `pages` (type: `string`):

Successful page rows plus skipped and failed URL rows.

## `chunks` (type: `string`):

Chunk-only dataset for RAG and vector database ingestion.

## `runSummary` (type: `string`):

Aggregate run summary stored in the default key-value store under OUTPUT.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://docs.apify.com/actors"
    ],
    "sitemapUrl": "https://docs.apify.com/sitemap.xml"
};

// Run the Actor and wait for it to finish
const run = await client.actor("sebastian-actors/website-to-rag-dataset").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "urls": ["https://docs.apify.com/actors"],
    "sitemapUrl": "https://docs.apify.com/sitemap.xml",
}

# Run the Actor and wait for it to finish
run = client.actor("sebastian-actors/website-to-rag-dataset").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://docs.apify.com/actors"
  ],
  "sitemapUrl": "https://docs.apify.com/sitemap.xml"
}' |
apify call sebastian-actors/website-to-rag-dataset --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=sebastian-actors/website-to-rag-dataset",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

```json
{
    "openapi": "3.0.1",
    "info": {
        "title": "Website to RAG Dataset",
        "description": "Convert public websites, docs, blogs, and XML sitemaps into clean Markdown, structured metadata, and stable chunks for RAG pipelines and vector databases.",
        "version": "0.1",
        "x-build-id": "d10vubhUFofNsbdg6"
    },
    "servers": [
        {
            "url": "https://api.apify.com/v2"
        }
    ],
    "paths": {
        "/acts/sebastian-actors~website-to-rag-dataset/run-sync-get-dataset-items": {
            "post": {
                "operationId": "run-sync-get-dataset-items-sebastian-actors-website-to-rag-dataset",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK"
                    }
                }
            }
        },
        "/acts/sebastian-actors~website-to-rag-dataset/runs": {
            "post": {
                "operationId": "runs-sync-sebastian-actors-website-to-rag-dataset",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor and returns information about the initiated run in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK",
                        "content": {
                            "application/json": {
                                "schema": {
                                    "$ref": "#/components/schemas/runsResponseSchema"
                                }
                            }
                        }
                    }
                }
            }
        },
        "/acts/sebastian-actors~website-to-rag-dataset/run-sync": {
            "post": {
                "operationId": "run-sync-sebastian-actors-website-to-rag-dataset",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK"
                    }
                }
            }
        }
    },
    "components": {
        "schemas": {
            "inputSchema": {
                "type": "object",
                "properties": {
                    "urls": {
                        "title": "URLs",
                        "type": "array",
                        "description": "List of public web pages to process. You can provide individual URLs, docs pages, blog posts, or landing pages.",
                        "items": {
                            "type": "string"
                        },
                        "default": []
                    },
                    "sitemapUrl": {
                        "title": "Sitemap URL",
                        "type": "string",
                        "description": "Optional XML sitemap or sitemap index URL. If provided, URLs from the sitemap are processed up to Max pages."
                    },
                    "maxPages": {
                        "title": "Max pages",
                        "minimum": 1,
                        "maximum": 10000,
                        "type": "integer",
                        "description": "Maximum number of pages to process.",
                        "default": 100
                    },
                    "sameDomainOnly": {
                        "title": "Same domain only",
                        "type": "boolean",
                        "description": "Only process URLs on the same hostname as the first URL or sitemap URL.",
                        "default": true
                    },
                    "respectRobotsTxt": {
                        "title": "Respect robots.txt",
                        "type": "boolean",
                        "description": "Check robots.txt and skip URLs disallowed for the configured user agent.",
                        "default": true
                    },
                    "outputMode": {
                        "title": "Output mode",
                        "enum": [
                            "pages",
                            "chunks",
                            "pages_and_chunks"
                        ],
                        "type": "string",
                        "description": "Choose whether to output page rows, chunk rows, or both.",
                        "default": "pages_and_chunks"
                    },
                    "includeMarkdown": {
                        "title": "Include Markdown",
                        "type": "boolean",
                        "description": "Include cleaned Markdown in page output rows.",
                        "default": true
                    },
                    "includeSchema": {
                        "title": "Include schema metadata",
                        "type": "boolean",
                        "description": "Extract JSON-LD schema types and key schema fields where available.",
                        "default": true
                    },
                    "includeFreshnessSignals": {
                        "title": "Include freshness signals",
                        "type": "boolean",
                        "description": "Extract Last-Modified, sitemap lastmod, datePublished, and dateModified signals when available.",
                        "default": true
                    },
                    "includeLinks": {
                        "title": "Include links",
                        "type": "boolean",
                        "description": "Include internal and external links found in the main content.",
                        "default": false
                    },
                    "includeHtml": {
                        "title": "Include cleaned HTML",
                        "type": "boolean",
                        "description": "Include cleaned HTML in page rows. Disabled by default to keep output smaller.",
                        "default": false
                    },
                    "chunkSize": {
                        "title": "Chunk size",
                        "minimum": 200,
                        "maximum": 4000,
                        "type": "integer",
                        "description": "Approximate chunk size in estimated tokens. Token estimate uses a simple characters/4 heuristic.",
                        "default": 1200
                    },
                    "chunkOverlap": {
                        "title": "Chunk overlap",
                        "minimum": 0,
                        "maximum": 1000,
                        "type": "integer",
                        "description": "Approximate overlap between chunks in estimated tokens.",
                        "default": 150
                    },
                    "minContentChars": {
                        "title": "Minimum content characters",
                        "minimum": 0,
                        "maximum": 10000,
                        "type": "integer",
                        "description": "Pages with less cleaned content than this value are marked as low-content.",
                        "default": 250
                    },
                    "maxConcurrency": {
                        "title": "Max concurrency",
                        "minimum": 1,
                        "maximum": 25,
                        "type": "integer",
                        "description": "Number of pages fetched in parallel.",
                        "default": 5
                    },
                    "requestTimeoutSecs": {
                        "title": "Request timeout seconds",
                        "minimum": 5,
                        "maximum": 120,
                        "type": "integer",
                        "description": "Timeout for each page request.",
                        "default": 20
                    },
                    "maxSitemaps": {
                        "title": "Max sitemaps",
                        "minimum": 1,
                        "maximum": 1000,
                        "type": "integer",
                        "description": "Maximum number of sitemap and sitemap-index files to fetch.",
                        "default": 100
                    },
                    "userAgent": {
                        "title": "User agent",
                        "type": "string",
                        "description": "User agent used for requests and robots.txt checks.",
                        "default": "WebsiteToRagDatasetBot/1.0 (+https://apify.com)"
                    },
                    "urlAllowlistPatterns": {
                        "title": "URL allowlist patterns",
                        "type": "array",
                        "description": "Optional regex patterns. If provided, only matching URLs are processed.",
                        "items": {
                            "type": "string"
                        },
                        "default": []
                    },
                    "urlBlocklistPatterns": {
                        "title": "URL blocklist patterns",
                        "type": "array",
                        "description": "Optional regex patterns. Matching URLs are skipped.",
                        "items": {
                            "type": "string"
                        },
                        "default": []
                    },
                    "removeSelectors": {
                        "title": "Remove CSS selectors",
                        "type": "array",
                        "description": "CSS selectors to remove before Markdown conversion.",
                        "items": {
                            "type": "string"
                        },
                        "default": [
                            "nav",
                            "footer",
                            "header",
                            "aside",
                            "script",
                            "style",
                            "noscript",
                            "iframe",
                            "form"
                        ]
                    },
                    "onlySelectors": {
                        "title": "Only CSS selectors",
                        "type": "array",
                        "description": "Optional CSS selectors for the content area. If provided, only matched elements are used.",
                        "items": {
                            "type": "string"
                        },
                        "default": []
                    },
                    "dedupeByCanonical": {
                        "title": "Dedupe by canonical URL",
                        "type": "boolean",
                        "description": "Avoid processing duplicate pages with the same canonical URL.",
                        "default": true
                    },
                    "saveRunSummaryToKeyValueStore": {
                        "title": "Save run summary",
                        "type": "boolean",
                        "description": "Save run-level summary statistics to the default key-value store under OUTPUT.",
                        "default": true
                    }
                }
            },
            "runsResponseSchema": {
                "type": "object",
                "properties": {
                    "data": {
                        "type": "object",
                        "properties": {
                            "id": {
                                "type": "string"
                            },
                            "actId": {
                                "type": "string"
                            },
                            "userId": {
                                "type": "string"
                            },
                            "startedAt": {
                                "type": "string",
                                "format": "date-time",
                                "example": "2025-01-08T00:00:00.000Z"
                            },
                            "finishedAt": {
                                "type": "string",
                                "format": "date-time",
                                "example": "2025-01-08T00:00:00.000Z"
                            },
                            "status": {
                                "type": "string",
                                "example": "READY"
                            },
                            "meta": {
                                "type": "object",
                                "properties": {
                                    "origin": {
                                        "type": "string",
                                        "example": "API"
                                    },
                                    "userAgent": {
                                        "type": "string"
                                    }
                                }
                            },
                            "stats": {
                                "type": "object",
                                "properties": {
                                    "inputBodyLen": {
                                        "type": "integer",
                                        "example": 2000
                                    },
                                    "rebootCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "restartCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "resurrectCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "computeUnits": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            },
                            "options": {
                                "type": "object",
                                "properties": {
                                    "build": {
                                        "type": "string",
                                        "example": "latest"
                                    },
                                    "timeoutSecs": {
                                        "type": "integer",
                                        "example": 300
                                    },
                                    "memoryMbytes": {
                                        "type": "integer",
                                        "example": 1024
                                    },
                                    "diskMbytes": {
                                        "type": "integer",
                                        "example": 2048
                                    }
                                }
                            },
                            "buildId": {
                                "type": "string"
                            },
                            "defaultKeyValueStoreId": {
                                "type": "string"
                            },
                            "defaultDatasetId": {
                                "type": "string"
                            },
                            "defaultRequestQueueId": {
                                "type": "string"
                            },
                            "buildNumber": {
                                "type": "string",
                                "example": "1.0.0"
                            },
                            "containerUrl": {
                                "type": "string"
                            },
                            "usage": {
                                "type": "object",
                                "properties": {
                                    "ACTOR_COMPUTE_UNITS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_WRITES": {
                                        "type": "integer",
                                        "example": 1
                                    },
                                    "KEY_VALUE_STORE_LISTS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_INTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_EXTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_SERPS": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            },
                            "usageTotalUsd": {
                                "type": "number",
                                "example": 0.00005
                            },
                            "usageUsd": {
                                "type": "object",
                                "properties": {
                                    "ACTOR_COMPUTE_UNITS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_WRITES": {
                                        "type": "number",
                                        "example": 0.00005
                                    },
                                    "KEY_VALUE_STORE_LISTS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_INTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_EXTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_SERPS": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}
```
