# Website Embeddings for RAG - Vector DB, Pinecone, Qdrant (`tidytools/website-to-embeddings`) Actor

Crawl a site or docs, chunk the pages and get 1024-dim multilingual bge-m3 embeddings with stable IDs for Pinecone, Qdrant, pgvector or any vector database. No OpenAI key. From $0.20/1k chunks.

- **URL**: https://apify.com/tidytools/website-to-embeddings.md
- **Developed by:** [Yukai Lin](https://apify.com/tidytools) (community)
- **Categories:** AI, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

**No OpenAI / Cohere API key needed.** One run = crawler + chunker + embedding API: you get vectors with stable IDs, ready to upsert into Pinecone, Qdrant, Weaviate or pgvector.

**Input:** website or documentation URLs (or your own texts, or another Actor's dataset ID). **Output:** one row per text chunk with its text, heading path, page metadata and a 1024-dimension bge-m3 vector with a stable UUID. **Price:** $0.20 per 1,000 chunks plus $1.00 per 1,000 pages (fast mode; $2.50 in browser mode), no start fee, failed pages free (a 100-page docs site is about $0.26).

### What does Website Embeddings for RAG do?

Website embeddings in one run: it crawls a website or documentation portal, converts every page to clean text, **splits it into chunks** and returns a **1024-dimension embedding for each chunk**, ready to load into Pinecone, Qdrant, Weaviate, Milvus, Chroma, pgvector or any other vector database. One run replaces a crawler, a chunker and an embedding API.

- 🌍 **Multilingual embeddings**: BAAI **bge-m3**, strong across 100+ languages (an English and a Chinese sentence with the same meaning land close together)
- 🔑 **No API key, no prepaid credits**: the embedding model runs on our side; you only pay per page and per chunk
- 🆔 **Upsert-ready**: every chunk has a stable UUID `id` (same URL + position = same id on every run), a `contentHash` and a `pageContentHash`, so re-runs update instead of duplicating
- 🕷️ **Crawls for you**: same folder, whole site, or only the URLs you list; include/exclude patterns
- 🌐 **Several websites at once**: with several start URLs, up to 5 websites are crawled in parallel (**Parallel websites**), each with its own **Time limit per website** (default 1,800 s) and optional **Max pages per start URL**. Pages are taken in the same order as Website to Markdown Crawler: links in the page content first, then menus, then the sitemap, then other languages and tag or author pages. `SUMMARY.sites` reports each website
- 🗺️ **Sitemap seeding and robots.txt**: reads the site's `sitemap.xml` so JavaScript menus don't hide pages, and respects robots.txt by default
- ✂️ **Paragraph-aware chunking** with overlap (you choose the size); each chunk carries its `headingPath` (h1 > h2 > h3) for better retrieval and citations, and a heading is never left alone at the end of a chunk (it moves to the start of the next chunk, with its text)
- 🏷️ **Metadata for filtering**: `language`, `canonicalUrl`, `description`, `wordCount`, `crawledAt`
- 📝 **Optional full-page Markdown** (`includePageMarkdown`) in the same run, at no extra cost
- 📄 **Documents too**: linked PDF, Word, Excel and CSV files are embedded as well
- 🧹 **No duplicate content**: identical pages under different URLs are embedded and charged once
- ✏️ **Your own texts**: embed a list of texts without crawling
- 🔗 **Embed another Actor's dataset**: already crawled with Apify's Website Content Crawler (or any scraper)? Pass its dataset ID and pay only the chunk price, no page fee
- 🧹 **Clean chunks by default**: **Remove navigation, headers and footers** keeps menus, footers, language lists and "Skip to main content" links out of your vectors (when a page has one main content element, such as `article` or `main`, only that element is converted), and **Tidy the Markdown** drops skip links left at the top, leftover `<!doctype>` lines, empty headings, tracking pixels, schema.org JSON-LD blocks and sections repeated word for word. Exact duplicate chunks (e.g. the same footer text on every page) are embedded once per run; repeats are skipped and not charged. With it on, tag, category and author link lines (and "By … " bylines with their avatars) above blog posts, page-end "Related tags", "Related posts", share, follow and newsletter boxes, and code-tab labels such as "JavaScriptTypeScript" are dropped as well, and a block repeated on many pages of the same site (a footer, a banner) is dropped from the fourth page on, so none of them becomes a charged chunk of its own

With `removeBoilerplate` on, hidden elements (`hidden` or inline `display:none`) are also removed before Markdown conversion and chunking.

- 🛟 **One bad page never stops the run**: a page that fails to load or embed gets its own error row, and the run goes on
- ⚡ **Fast**: in our test, 6 pages → 45 chunks were crawled and embedded in about 3 seconds

#### Website embeddings for RAG, semantic search and chatbots

An alternative to chaining a crawler (such as Website Content Crawler) with a Pinecone or Qdrant integration that calls an embedding API:

| | Crawler + embedding integration | This Actor |
|---|---|---|
| Embedding API key | Your own OpenAI (or similar) key | Not needed |
| Steps | Crawl, then pass the dataset to an embedding step | Crawl, chunk and embed in one run |
| Failed pages | Depends on each tool's pricing | Free (error row, not charged) |
| Price | Each tool's price plus embedding API tokens | $0.20 per 1,000 chunks plus $1.00 per 1,000 pages (fast mode) |

### How much does it cost?

| Event | Price |
|---|---|
| Page (fast mode, plain HTTP) | $1.00 / 1,000 pages |
| Page (browser mode, JavaScript) | $2.50 / 1,000 pages |
| Embedded chunk | **$0.20 / 1,000 chunks** |

Example: a 100-page docs site with ~8 chunks per page costs about $0.10 for pages + $0.16 for 800 chunks = **$0.26**. **No start fee.** Duplicate, blocked and failed pages are free, and so are pages disallowed by robots.txt, pages with no text and exact duplicate chunks. When you embed another Actor's dataset, there is no page fee: only the chunks are charged. Higher Apify plans get volume discounts (Scale 10%, Business and higher 20%).

For comparison (checked September 2026): Apify's Pinecone and Qdrant integrations are free to run but need your own OpenAI or Cohere embeddings API key and your own vector database, and you pay their platform usage. Here the embeddings are included in the chunk price.

The trade-off: if you already pay for an embeddings API, embedding with your own key can cost less per chunk than this Actor. What you get here is no key, no setup, and crawling + chunking + embedding in one run.

### Control your cost

- **What is charged:** each chunk embedded, and the page fee for each crawled page, **only after all of its chunks were embedded**. If embedding fails part-way, the chunks already embedded are charged but the page fee is not.
- **What is free:** failed, blocked and duplicate pages, pages with no text, exact duplicate chunks, chunks with fewer than 4 visible characters (a lone heading at the end of a page), pages disallowed by robots.txt and invalid input lines. Rows have `charged: true` or `false`.
- At the start, the run logs its worst case for pages (every page at the browser price), e.g. `Plan: 50 pages × $0.0025 = at most $0.125`, plus the chunk price, and warns when that is more than your **maximum charge per run**.

#### Why did my run stop at $X?

Every Apify run has a **maximum charge per run** (set in the run options, or `maxTotalChargeUsd` in the API; integrations often set a low one). When the next chunk or page would go over it, the run stops, keeps everything embedded so far, and says so in the status message. The `SUMMARY` record then has `status: "LIMIT_REACHED"` and `notProcessed` (how many queued pages were not crawled, and up to 100 of them). Raise the limit and run again: chunk `id`s are stable, so re-embedded chunks overwrite the same points in your vector database.

### How to use it

1. Enter **start URLs**, one per line (e.g. your docs home page; bare domains are fine), and **max pages**.
2. Choose a **chunk size** (1000–2000 characters works well) and overlap.
3. Keep **Remove navigation, headers and footers** on (the default) to keep menus out of your embeddings; a **content selector** such as `main` also works.
4. Optional: **Use the site's sitemap** is on (*Seed*) by default; *Only* embeds just the sitemap's pages.
5. Click **Start**, then read the dataset (JSON) and upsert it into your vector database.

#### Input example

```json
{
    "urls": ["https://docs.example.com/"],
    "maxPages": 200,
    "crawlScope": "path",
    "chunkSize": 1500,
    "chunkOverlap": 150,
    "removeBoilerplate": true,
    "useSitemap": "seed",
    "respectRobotsTxt": true,
    "includePageMarkdown": false
}
```

#### Output: one item per chunk

Real output from a test run on `https://docs.apify.com/academy/web-scraping-for-beginners` (text and vector shortened):

```json
{
    "id": "28eedd07-a3b0-52a2-b015-079e3cec641e",
    "type": "chunk",
    "source": "web",
    "url": "https://docs.apify.com/academy/actor-marketing-playbook/actor-basics/actors-and-emojis",
    "startUrl": "https://docs.apify.com/academy/web-scraping-for-beginners",
    "title": "Actors and emojis | Academy | Apify Documentation",
    "chunkIndex": 1,
    "text": "…to explain things about Actors, and we want to avoid users needing to open extra tabs or pages. …",
    "headingPath": ["Actors and emojis", "On the use of emojis in Actors"],
    "charCount": 1188,
    "wordCount": 198,
    "description": "Discover how emojis can boost your Actors by grabbing attention, simplifying navigation, and enhancing clarity. …",
    "language": "en",
    "canonicalUrl": "https://docs.apify.com/academy/actor-marketing-playbook/actor-basics/actors-and-emojis",
    "contentHash": "0484f7301da3d86ade63f5a7c135edaabf9ad668",
    "pageContentHash": "7cd55fb178f0a9d85e9f240c141752dd40cfad33",
    "via": "backend",
    "crawledAt": "2026-09-29T02:16:40.384Z",
    "embedding": [0.0123, -0.0456, "… 1024 numbers …"],
    "dimensions": 1024,
    "model": "bge-m3",
    "charged": true
}
```

- `id` is a UUID derived from the page URL and the chunk position, so it is the same on every run: use it as the primary key when you upsert.
- `contentHash` changes only when the chunk's text changes; `pageContentHash` only when the page changes. Compare them with your stored values to skip unchanged vectors.
- `wordCount` and `charCount` are for this chunk. `via` tells where the page was fetched from (`backend`, `direct` or `browser`).
- For your own `texts`, `source` is `text-0`, `text-1`… and the `id` is derived from the text itself.
- With **Also output each page's full Markdown**, each page adds one item with `"type": "page"`, the whole `markdown`, `wordCount` for the page and the same metadata (no extra charge).
- `startUrl` is the start URL a crawled page came from.
- A failed page is not charged: `{ "url": "…", "startUrl": "…", "success": false, "error": "target returned HTTP 404 (not charged)", "errorType": "not_found", "charged": false }`.
- The key-value store's `SUMMARY` record has `status` (`SUCCESS`, `PARTIAL_RESULTS`, `FAILED`, `NO_RESULTS` or `LIMIT_REACHED`), counts and `duplicateChunksSkipped`.
- For crawls, `SUMMARY.sites` has one entry per start URL: `status` (`ok`, `failed`, `timeout` or `not_processed`), `pages`, `chunks`, `failed`, `duplicates`, `notCrawled` (pages found but not crawled because of Max pages, Max pages per start URL or the time limit) and the error if any. Max pages is shared by all start URLs: with more start URLs than Max pages, the extra websites are `not_processed` (not charged). A website stopped at its time limit gets a row with `errorType: "timeout"` (not charged); the pages embedded before the limit stay (charged).

#### Already crawled with Website Content Crawler? Embed its dataset

Crawl with Apify's Website Content Crawler (or [Website to Markdown Crawler](https://apify.com/tidytools/website-markdown-crawler)), then pass the run's dataset ID. The `markdown` field is used (then `text`), long pages are chunked, and `url`, `metadata.title`, `metadata.description`, `metadata.languageCode` and `metadata.canonicalUrl` are carried over:

```json
{ "datasetId": "YOUR_WCC_RUN_DATASET_ID" }
```

Real output from a 3-page Website Content Crawler run on `https://docs.apify.com/academy/web-scraping-for-beginners`: 3 dataset items became 26 chunks (charged as 26 chunks, $0.0052, no page fee):

```json
{
    "id": "fcabdb9f-85c9-5155-875b-0b12afbe1807",
    "type": "chunk",
    "source": "dataset",
    "url": "https://docs.apify.com/academy/web-scraping-for-beginners",
    "title": "Web scraping basics for JavaScript devs | Academy | Apify Documentation",
    "chunkIndex": 0,
    "text": "# Web scraping basics for JavaScript devs | Academy\n\n**Learn how to use JavaScript to extract information from websites in this practical course, starting from …",
    "headingPath": ["Web scraping basics for JavaScript devs | Academy"],
    "charCount": 1278,
    "language": "en",
    "description": "Learn how to use JavaScript to extract information from websites in this practical course, starting from the absolute basics.",
    "canonicalUrl": "https://docs.apify.com/academy/scraping-basics-javascript",
    "embedding": [0.0123, -0.0456, "… 1024 numbers …"],
    "dimensions": 1024,
    "model": "bge-m3"
}
```

- Chunk IDs come from `url` + chunk position, exactly as in a crawl, so a WCC run and a crawl of the same page upsert the same points.
- Other datasets work too: set **Dataset text field** (e.g. `content.body`) and **Dataset URL field** if they are named differently. Items with `"success": false` are skipped.
- The `SUMMARY` record has a `dataset` object with `textField`, `itemsRead`, `itemsEmbedded` and `duplicatesSkipped`.
- From code, chain the two runs: start WCC, then call this Actor with `{"datasetId": run["defaultDatasetId"]}`.
- Dataset rows carry `datasetIndex` (the item's position in the source dataset). An item that cannot be embedded gets an error row and the run continues; after 5 failures in a row the dataset part stops, and `SUMMARY.dataset` keeps the counts of what was embedded.

#### Run automatically after a crawl (Apify integration)

1. Open your crawler (Website Content Crawler or Website to Markdown Crawler) or its saved task, go to **Integrations** and choose **Connect Actor or task** → **Website Embeddings for RAG**, triggered when the run succeeds.
2. In the input, clear **Start URLs** and set the dataset: `{"datasetId": "{{resource.defaultDatasetId}}"}`. `{{resource.defaultDatasetId}}` is Apify's variable for the triggering run's dataset.
3. Set a **maximum charge per run** that fits your crawl size (see *Why did my run stop at $X?* above).

#### Embedding search queries

Query and document vectors must come from the same model (bge-m3). Use this Actor for queries too, with the `texts` input:

```json
{ "texts": ["How do I scrape a page with JavaScript?"] }
```

#### Load into a vector database (Python example)

```python
from apify_client import ApifyClient
client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("tidytools/website-to-embeddings").call(run_input={"urls": ["https://docs.example.com/"], "maxPages": 100})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    if "embedding" in item:
        index.upsert([(item["id"], item["embedding"], {"url": item["url"], "text": item["text"], "headings": " > ".join(item["headingPath"])})])
```

Qdrant example (the `id` is a UUID, which Qdrant accepts as a point ID):

```python
from qdrant_client import QdrantClient, models
qdrant = QdrantClient(url="https://YOUR-CLUSTER.qdrant.io", api_key="YOUR_QDRANT_KEY")
points = [models.PointStruct(id=i["id"], vector=i["embedding"], payload={"url": i["url"], "text": i["text"]})
          for i in client.dataset(run["defaultDatasetId"]).iterate_items() if "embedding" in i]
qdrant.upsert(collection_name="docs", points=points)  # create it first with size=1024, distance=COSINE
```

To search, embed your queries with the same model (bge-m3) so query and document vectors are comparable (see *Embedding search queries* above).

### Use with AI agents (MCP)

Connect Apify's MCP server (https://mcp.apify.com?tools=tidytools/website-to-embeddings) to Claude, Cursor or any MCP client, then ask e.g. "Embed the first 50 pages of docs.example.com and give me the dataset ID."

Minimal input:

```json
{ "urls": ["https://docs.apify.com/academy"], "maxPages": 20 }
```

Failed items are not charged and carry an `errorType`.

### Use it from code and integrations

Run it from your own code with the Apify API. This call waits for the run and returns the results as JSON (replace `YOUR_TOKEN` with your Apify API token):

```bash
curl -X POST "https://api.apify.com/v2/acts/tidytools~website-to-embeddings/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://docs.apify.com/academy/web-scraping-for-beginners"],"maxPages":5,"removeBoilerplate":true}'
```

The synchronous endpoint waits up to 5 minutes. For bigger runs, start the run with `POST https://api.apify.com/v2/acts/tidytools~website-to-embeddings/runs` and read the dataset when it finishes, or use the `apify-client` package for JavaScript or Python.

**Schedules and integrations:** run it daily or weekly with Apify Schedules, get a webhook when a run finishes, or send the results to Zapier, Make, n8n, Google Sheets, Slack and other apps with Apify integrations. Results can be exported as JSON, CSV, Excel or XML.

### Limitations

- Only public pages; no logins.
- **Input mistakes never fail the run:** with no valid URL, text or dataset ID, the run ends normally with free error rows (`errorType: "invalid_input"`, `charged: false`) and `SUMMARY.status` `NO_RESULTS`. A dataset that cannot be read (wrong ID, no permission) gets one free error row too.
- **Errors are free rows:** an unknown domain or a server that does not answer gives `errorType: "unreachable"`, an invalid SSL certificate also `unreachable`, and a start URL that robots.txt disallows gives `errorType: "robots_disallowed"` (other disallowed pages found while crawling are skipped silently and counted in `SUMMARY.disallowedByRobotsTxt`). None of them is charged.
- A very long page (e.g. a whole book as one text file) produces many chunks, each charged at the chunk price; the start-of-run cost plan assumes about 2 chunks per page, so set a **maximum charge per run** when you crawl pages you have not seen.
- A start URL that redirects to another address (e.g. another host) is followed: links under the final address are crawled with the same scope, and `SUMMARY.redirectedStarts` lists the redirect.
- **Documentation sites** with *Same folder*: if the start page links to nothing inside its own folder (sites that give every page its own folder, such as `/en/getting-started/`), that website is crawled from the folder one level up (`/en/`, never the whole site) and SUMMARY `scopeWidened` lists it.
- **Calling the API?** Use the plain `urls` list: the older `startUrls` field still works, but Apify rejects the whole run (HTTP 400) when it holds a bare domain or a blank line. Clean-up and tidying are on unless you send `"removeBoilerplate": false` or `"tidyMarkdown": false`.
- If a site blocks our servers, **Advanced settings** → *Auto* retries from Apify's network; you can add an Apify proxy (billed to your Apify account).
- Each chunk is embedded as-is up to the model's input limit (long chunks are truncated).
- The model is fixed (bge-m3, 1024 dimensions); queries must be embedded with the same model.
- A dataset is read up to **Max dataset items** (default 1,000). The dataset must be one your Apify account can read.

### FAQ

**Does the Actor collect usage data?** The Actor sends one anonymous usage counter per run (input type, item count range, outcome) to improve the tool; no URLs, content or account data.

**Do I need an OpenAI or Cohere API key to get vector embeddings?** No. The embedding model (BAAI bge-m3, 1024 dimensions) runs on our side, so there is no API key and no prepaid credits: you only pay per page and per chunk.

**Can I load the output into Pinecone, Qdrant or another vector database?** Yes. Every chunk comes with a 1024-dimension embedding and a stable UUID `id` (same URL + position = same id on every run), ready to upsert into Pinecone, Qdrant, Weaviate, Milvus, Chroma, pgvector or any other vector database, so re-runs update instead of duplicating. Qdrant accepts the UUID `id` as a point ID.

**How is it different from Apify's Pinecone and Qdrant integrations?** Those integrations are free to run but need your own OpenAI or Cohere embeddings API key and your own vector database, and you pay their platform usage. Here crawling, chunking and embedding happen in one run and the embeddings are included in the chunk price; if you already pay for an embeddings API, embedding with your own key can cost less per chunk.

**What is charged and what is free?** You pay $0.20 per 1,000 embedded chunks plus a page fee ($1.00 / 1,000 pages in fast mode, $2.50 / 1,000 in browser mode), with no start fee. Failed, blocked and duplicate pages, pages with no text, pages disallowed by robots.txt and exact duplicate chunks are free, and with **Embed another Actor's dataset (dataset ID)** there is no page fee, only the chunk price.

### Support

Open an issue in the **Issues** tab with your start URL and input. Issues are checked regularly.

Found this useful? A short review on the Store helps other users find it.

# Actor input Schema

## `urls` (type: `array`):

Main input (fill this, or `texts` / `startUrls` / `datasetId`). Website or documentation to crawl and embed, one URL per line. Bare domains (example.com) are fine; blank or invalid lines are skipped and listed in the results (not charged).

## `texts` (type: `array`):

Embed your own texts instead of (or in addition to) crawling. Long texts are chunked too.

## `maxPages` (type: `integer`):

Stop crawling after this many pages.

## `maxPagesPerStartUrl` (type: `integer`):

With several start URLs: stop crawling one website after this many pages, so every website gets its share of Max pages. 0 = no per-website limit.

## `crawlScope` (type: `string`):

Which links to follow. "Same folder" keeps the crawl under the start URL's path; when the start page links to nothing in its own folder (docs sites that give every page its own folder, e.g. /en/getting-started/), the folder one level up is used for it, never the whole site. Values: path = Same folder as the start URL; domain = Whole website (incl. subdomains); page = Only the start URLs.

## `maxDepth` (type: `integer`):

How many clicks away from a start URL to go.

## `useSitemap` (type: `string`):

Read sitemap.xml (found via robots.txt or common locations) and queue its in-scope pages. Finds pages that JavaScript menus hide from link discovery. Ignored when the crawl scope is "Only the start URLs". Values: off = Off: follow links only; seed = Seed (recommended): sitemap URLs plus links; only = Only: sitemap URLs, don't follow links.

## `respectRobotsTxt` (type: `boolean`):

Skip pages the site's robots.txt disallows for crawlers. Skipped pages are not output, not charged and don't count toward Max pages. A start URL that robots.txt disallows gets one free error row (errorType robots_disallowed).

## `chunkSize` (type: `integer`):

Target size of each chunk; chunks are split at paragraph boundaries. 1000-2000 works well for most RAG setups.

## `chunkOverlap` (type: `integer`):

Characters repeated between consecutive chunks to keep context.

## `includePageMarkdown` (type: `boolean`):

Adds one extra item per page (type "page") with the whole Markdown and page metadata, so you don't need a separate crawl. No extra charge.

## `cssSelector` (type: `string`):

Optional, e.g. "main" or "article", to leave menus and footers out of the embeddings.

## `includeUrlPatterns` (type: `array`):

Optional glob patterns; only matching URLs are crawled.

## `excludeUrlPatterns` (type: `array`):

Optional glob patterns to skip.

## `includeDocuments` (type: `boolean`):

Also convert and embed linked PDF, Word, Excel and CSV files.

## `mode` (type: `string`):

"Auto" uses plain HTTP and a real browser only when needed. Values: auto = Auto (recommended); fast = Fast (HTTP only); browser = Browser (JavaScript rendering).

## `maxConcurrency` (type: `integer`):

How many pages are processed at the same time.

## `httpVia` (type: `string`):

Some sites block requests from data centers. Auto retries from a second network before falling back to a real browser. If both are refused, Auto also tries our second server (Oracle, different IP) before a real browser.

## `maxConcurrentSites` (type: `integer`):

With several start URLs: how many websites are crawled at the same time, so one slow website does not hold up the others. "Parallel pages" and Max pages are shared by all of them, and one page of Max pages is kept for every website that has not started yet, so each start page is read.

## `siteTimeoutSecs` (type: `integer`):

Crawling one start URL (its sitemap, pages and embeddings) stops after this long. Pages embedded before the limit stay in the results (charged); pages still being read are dropped (not charged), the website gets a row with errorType "timeout" (not charged) and the next website starts. 0 = no limit.

## `proxyConfiguration` (type: `object`):

Only used for requests sent from Apify's network. Apify proxy usage is billed to your Apify account.

## `removeBoilerplate` (type: `boolean`):

Recommended (on by default, also for API calls). Drops nav bars, site headers and footers, sidebars, breadcrumbs, cookie notices, "Skip to main content" links and table-of-contents widgets from the Markdown before it is chunked. Links for crawling are still taken from the whole page. Set to false to keep the full page.

## `removeSelectors` (type: `string`):

Optional comma-separated CSS selectors removed from the Markdown, e.g. ".related-posts, #comments, .ad". Works with or without the option above.

## `tidyMarkdown` (type: `boolean`):

Removes leftover <!doctype> lines, empty headings, 1×1 tracking-pixel images, schema.org JSON-LD code blocks and exact repeated sections before chunking (crawled pages and dataset items), so they are not embedded or charged. Set to false to embed the text unchanged.

## `startUrls` (type: `array`):

Same as the list above, in Apify's request-list format (also accepts a link to a text file of URLs). Kept for older inputs; the plain list above is recommended, especially for API calls.

## `datasetId` (type: `string`):

Optional. The ID of a dataset from another run, e.g. Website Content Crawler or Website to Markdown Crawler. Every item's text is chunked and embedded: only the chunk price applies, no page fee. Can be used instead of, or together with, start URLs.

## `datasetTextField` (type: `string`):

Field that holds the text to embed (dot paths allowed, e.g. "content.body"). Leave empty to use "markdown", then "text".

## `datasetUrlField` (type: `string`):

Field with the page URL, used for the stable chunk IDs and the "url" output field.

## `datasetMaxItems` (type: `integer`):

Stop after this many dataset items.

## Actor input object example

```json
{
  "urls": [
    "https://docs.apify.com/academy/web-scraping-for-beginners"
  ],
  "maxPages": 5,
  "maxPagesPerStartUrl": 0,
  "crawlScope": "path",
  "maxDepth": 5,
  "useSitemap": "seed",
  "respectRobotsTxt": true,
  "chunkSize": 1500,
  "chunkOverlap": 150,
  "includePageMarkdown": false,
  "includeDocuments": true,
  "mode": "auto",
  "maxConcurrency": 8,
  "httpVia": "auto",
  "maxConcurrentSites": 5,
  "siteTimeoutSecs": 1800,
  "removeBoilerplate": true,
  "tidyMarkdown": true,
  "datasetUrlField": "url",
  "datasetMaxItems": 1000
}
```

# Actor output Schema

## `chunks` (type: `string`):

No description

## `full` (type: `string`):

No description

## `details` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://docs.apify.com/academy/web-scraping-for-beginners"
    ],
    "maxPages": 5,
    "removeBoilerplate": true
};

// Run the Actor and wait for it to finish
const run = await client.actor("tidytools/website-to-embeddings").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "urls": ["https://docs.apify.com/academy/web-scraping-for-beginners"],
    "maxPages": 5,
    "removeBoilerplate": True,
}

# Run the Actor and wait for it to finish
run = client.actor("tidytools/website-to-embeddings").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://docs.apify.com/academy/web-scraping-for-beginners"
  ],
  "maxPages": 5,
  "removeBoilerplate": true
}' |
apify call tidytools/website-to-embeddings --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,tidytools/website-to-embeddings"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/69AbOL4uxESijjy3s/builds/9lqbo3xR5yuVe08La/openapi.json