Website Embeddings for RAG - Vector DB, Pinecone, Qdrant

No OpenAI / Cohere API key needed. One run = crawler + chunker + embedding API: you get vectors with stable IDs, ready to upsert into Pinecone, Qdrant, Weaviate or pgvector.

Input: website or documentation URLs (or your own texts, or another Actor's dataset ID). Output: one row per text chunk with its text, heading path, page metadata and a 1024-dimension bge-m3 vector with a stable UUID. Price: $0.20 per 1,000 chunks plus $1.00 per 1,000 pages (fast mode; $2.50 in browser mode), no start fee, failed pages free (a 100-page docs site is about $0.26).

What does Website Embeddings for RAG do?

Website embeddings in one run: it crawls a website or documentation portal, converts every page to clean text, splits it into chunks and returns a 1024-dimension embedding for each chunk, ready to load into Pinecone, Qdrant, Weaviate, Milvus, Chroma, pgvector or any other vector database. One run replaces a crawler, a chunker and an embedding API.

• 🌍 Multilingual embeddings: BAAI bge-m3, strong across 100+ languages (an English and a Chinese sentence with the same meaning land close together)
• 🔑 No API key, no prepaid credits: the embedding model runs on our side; you only pay per page and per chunk
• 🆔 Upsert-ready: every chunk has a stable UUID id (same URL + position = same id on every run), a contentHash and a pageContentHash, so re-runs update instead of duplicating
• 🕷️ Crawls for you: same folder, whole site, or only the URLs you list; include/exclude patterns
• 🌐 Several websites at once: with several start URLs, up to 5 websites are crawled in parallel (Parallel websites), each with its own Time limit per website (default 1,800 s) and optional Max pages per start URL. Pages are taken in the same order as Website to Markdown Crawler: links in the page content first, then menus, then the sitemap, then other languages and tag or author pages. SUMMARY.sites reports each website
• 🗺️ Sitemap seeding and robots.txt: reads the site's sitemap.xml so JavaScript menus don't hide pages, and respects robots.txt by default
• ✂️ Paragraph-aware chunking with overlap (you choose the size); each chunk carries its headingPath (h1 > h2 > h3) for better retrieval and citations, and a heading is never left alone at the end of a chunk (it moves to the start of the next chunk, with its text)
• 🏷️ Metadata for filtering: language, canonicalUrl, description, wordCount, crawledAt
• 📝 Optional full-page Markdown (includePageMarkdown) in the same run, at no extra cost
• 📄 Documents too: linked PDF, Word, Excel and CSV files are embedded as well
• 🧹 No duplicate content: identical pages under different URLs are embedded and charged once
• ✏️ Your own texts: embed a list of texts without crawling
• 🔗 Embed another Actor's dataset: already crawled with Apify's Website Content Crawler (or any scraper)? Pass its dataset ID and pay only the chunk price, no page fee
• 🧹 Clean chunks by default: Remove navigation, headers and footers keeps menus, footers, language lists and "Skip to main content" links out of your vectors (when a page has one main content element, such as article or main, only that element is converted), and Tidy the Markdown drops skip links left at the top, leftover <!doctype> lines, empty headings, tracking pixels, schema.org JSON-LD blocks and sections repeated word for word. Exact duplicate chunks (e.g. the same footer text on every page) are embedded once per run; repeats are skipped and not charged. With it on, tag, category and author link lines (and "By … " bylines with their avatars) above blog posts, page-end "Related tags", "Related posts", share, follow and newsletter boxes, and code-tab labels such as "JavaScriptTypeScript" are dropped as well, and a block repeated on many pages of the same site (a footer, a banner) is dropped from the fourth page on, so none of them becomes a charged chunk of its own

With removeBoilerplate on, hidden elements (hidden or inline display:none) are also removed before Markdown conversion and chunking.

• 🛟 One bad page never stops the run: a page that fails to load or embed gets its own error row, and the run goes on
• ⚡ Fast: in our test, 6 pages → 45 chunks were crawled and embedded in about 3 seconds

Website embeddings for RAG, semantic search and chatbots

An alternative to chaining a crawler (such as Website Content Crawler) with a Pinecone or Qdrant integration that calls an embedding API:

	Crawler + embedding integration	This Actor
Embedding API key	Your own OpenAI (or similar) key	Not needed
Steps	Crawl, then pass the dataset to an embedding step	Crawl, chunk and embed in one run
Failed pages	Depends on each tool's pricing	Free (error row, not charged)
Price	Each tool's price plus embedding API tokens	$0.20 per 1,000 chunks plus $1.00 per 1,000 pages (fast mode)

How much does it cost?

Event	Price
Page (fast mode, plain HTTP)	$1.00 / 1,000 pages
Page (browser mode, JavaScript)	$2.50 / 1,000 pages
Embedded chunk	$0.20 / 1,000 chunks

Example: a 100-page docs site with ~8 chunks per page costs about $0.10 for pages + $0.16 for 800 chunks = $0.26. No start fee. Duplicate, blocked and failed pages are free, and so are pages disallowed by robots.txt, pages with no text and exact duplicate chunks. When you embed another Actor's dataset, there is no page fee: only the chunks are charged. Higher Apify plans get volume discounts (Scale 10%, Business and higher 20%).

For comparison (checked September 2026): Apify's Pinecone and Qdrant integrations are free to run but need your own OpenAI or Cohere embeddings API key and your own vector database, and you pay their platform usage. Here the embeddings are included in the chunk price.

The trade-off: if you already pay for an embeddings API, embedding with your own key can cost less per chunk than this Actor. What you get here is no key, no setup, and crawling + chunking + embedding in one run.

Control your cost

• What is charged: each chunk embedded, and the page fee for each crawled page, only after all of its chunks were embedded. If embedding fails part-way, the chunks already embedded are charged but the page fee is not.
• What is free: failed, blocked and duplicate pages, pages with no text, exact duplicate chunks, chunks with fewer than 4 visible characters (a lone heading at the end of a page), pages disallowed by robots.txt and invalid input lines. Rows have charged: true or false.
• At the start, the run logs its worst case for pages (every page at the browser price), e.g. Plan: 50 pages × $0.0025 = at most $0.125, plus the chunk price, and warns when that is more than your maximum charge per run.

Why did my run stop at $X?

Every Apify run has a maximum charge per run (set in the run options, or maxTotalChargeUsd in the API; integrations often set a low one). When the next chunk or page would go over it, the run stops, keeps everything embedded so far, and says so in the status message. The SUMMARY record then has status: "LIMIT_REACHED" and notProcessed (how many queued pages were not crawled, and up to 100 of them). Raise the limit and run again: chunk ids are stable, so re-embedded chunks overwrite the same points in your vector database.

How to use it

1. Enter start URLs, one per line (e.g. your docs home page; bare domains are fine), and max pages.
2. Choose a chunk size (1000–2000 characters works well) and overlap.
3. Keep Remove navigation, headers and footers on (the default) to keep menus out of your embeddings; a content selector such as main also works.
4. Optional: Use the site's sitemap is on (Seed) by default; Only embeds just the sitemap's pages.
5. Click Start, then read the dataset (JSON) and upsert it into your vector database.

Input example

{
    "urls": ["https://docs.example.com/"],
    "maxPages": 200,
    "crawlScope": "path",
    "chunkSize": 1500,
    "chunkOverlap": 150,
    "removeBoilerplate": true,
    "useSitemap": "seed",
    "respectRobotsTxt": true,
    "includePageMarkdown": false
}

Output: one item per chunk

Real output from a test run on https://docs.apify.com/academy/web-scraping-for-beginners (text and vector shortened):

{
    "id": "28eedd07-a3b0-52a2-b015-079e3cec641e",
    "type": "chunk",
    "source": "web",
    "url": "https://docs.apify.com/academy/actor-marketing-playbook/actor-basics/actors-and-emojis",
    "startUrl": "https://docs.apify.com/academy/web-scraping-for-beginners",
    "title": "Actors and emojis | Academy | Apify Documentation",
    "chunkIndex": 1,
    "text": "…to explain things about Actors, and we want to avoid users needing to open extra tabs or pages. …",
    "headingPath": ["Actors and emojis", "On the use of emojis in Actors"],
    "charCount": 1188,
    "wordCount": 198,
    "description": "Discover how emojis can boost your Actors by grabbing attention, simplifying navigation, and enhancing clarity. …",
    "language": "en",
    "canonicalUrl": "https://docs.apify.com/academy/actor-marketing-playbook/actor-basics/actors-and-emojis",
    "contentHash": "0484f7301da3d86ade63f5a7c135edaabf9ad668",
    "pageContentHash": "7cd55fb178f0a9d85e9f240c141752dd40cfad33",
    "via": "backend",
    "crawledAt": "2026-09-29T02:16:40.384Z",
    "embedding": [0.0123, -0.0456, "… 1024 numbers …"],
    "dimensions": 1024,
    "model": "bge-m3",
    "charged": true
}

• id is a UUID derived from the page URL and the chunk position, so it is the same on every run: use it as the primary key when you upsert.
• contentHash changes only when the chunk's text changes; pageContentHash only when the page changes. Compare them with your stored values to skip unchanged vectors.
• wordCount and charCount are for this chunk. via tells where the page was fetched from (backend, direct or browser).
• For your own texts, source is text-0, text-1… and the id is derived from the text itself.
• With Also output each page's full Markdown, each page adds one item with "type": "page", the whole markdown, wordCount for the page and the same metadata (no extra charge).
• startUrl is the start URL a crawled page came from.
• A failed page is not charged: { "url": "…", "startUrl": "…", "success": false, "error": "target returned HTTP 404 (not charged)", "errorType": "not_found", "charged": false }.
• The key-value store's SUMMARY record has status (SUCCESS, PARTIAL_RESULTS, FAILED, NO_RESULTS or LIMIT_REACHED), counts and duplicateChunksSkipped.
• For crawls, SUMMARY.sites has one entry per start URL: status (ok, failed, timeout or not_processed), pages, chunks, failed, duplicates, notCrawled (pages found but not crawled because of Max pages, Max pages per start URL or the time limit) and the error if any. Max pages is shared by all start URLs: with more start URLs than Max pages, the extra websites are not_processed (not charged). A website stopped at its time limit gets a row with errorType: "timeout" (not charged); the pages embedded before the limit stay (charged).

Already crawled with Website Content Crawler? Embed its dataset

Crawl with Apify's Website Content Crawler (or Website to Markdown Crawler), then pass the run's dataset ID. The markdown field is used (then text), long pages are chunked, and url, metadata.title, metadata.description, metadata.languageCode and metadata.canonicalUrl are carried over:

{ "datasetId": "YOUR_WCC_RUN_DATASET_ID" }

Real output from a 3-page Website Content Crawler run on https://docs.apify.com/academy/web-scraping-for-beginners: 3 dataset items became 26 chunks (charged as 26 chunks, $0.0052, no page fee):

{
    "id": "fcabdb9f-85c9-5155-875b-0b12afbe1807",
    "type": "chunk",
    "source": "dataset",
    "url": "https://docs.apify.com/academy/web-scraping-for-beginners",
    "title": "Web scraping basics for JavaScript devs | Academy | Apify Documentation",
    "chunkIndex": 0,
    "text": "# Web scraping basics for JavaScript devs | Academy\n\n**Learn how to use JavaScript to extract information from websites in this practical course, starting from …",
    "headingPath": ["Web scraping basics for JavaScript devs | Academy"],
    "charCount": 1278,
    "language": "en",
    "description": "Learn how to use JavaScript to extract information from websites in this practical course, starting from the absolute basics.",
    "canonicalUrl": "https://docs.apify.com/academy/scraping-basics-javascript",
    "embedding": [0.0123, -0.0456, "… 1024 numbers …"],
    "dimensions": 1024,
    "model": "bge-m3"
}

• Chunk IDs come from url + chunk position, exactly as in a crawl, so a WCC run and a crawl of the same page upsert the same points.
• Other datasets work too: set Dataset text field (e.g. content.body) and Dataset URL field if they are named differently. Items with "success": false are skipped.
• The SUMMARY record has a dataset object with textField, itemsRead, itemsEmbedded and duplicatesSkipped.
• From code, chain the two runs: start WCC, then call this Actor with {"datasetId": run["defaultDatasetId"]}.
• Dataset rows carry datasetIndex (the item's position in the source dataset). An item that cannot be embedded gets an error row and the run continues; after 5 failures in a row the dataset part stops, and SUMMARY.dataset keeps the counts of what was embedded.

Run automatically after a crawl (Apify integration)

1. Open your crawler (Website Content Crawler or Website to Markdown Crawler) or its saved task, go to Integrations and choose Connect Actor or task → Website Embeddings for RAG, triggered when the run succeeds.
2. In the input, clear Start URLs and set the dataset: {"datasetId": "{{resource.defaultDatasetId}}"}. {{resource.defaultDatasetId}} is Apify's variable for the triggering run's dataset.
3. Set a maximum charge per run that fits your crawl size (see Why did my run stop at $X? above).

Embedding search queries

Query and document vectors must come from the same model (bge-m3). Use this Actor for queries too, with the texts input:

{ "texts": ["How do I scrape a page with JavaScript?"] }

Load into a vector database (Python example)

from apify_client import ApifyClient
client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("tidytools/website-to-embeddings").call(run_input={"urls": ["https://docs.example.com/"], "maxPages": 100})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    if "embedding" in item:
        index.upsert([(item["id"], item["embedding"], {"url": item["url"], "text": item["text"], "headings": " > ".join(item["headingPath"])})])

Qdrant example (the id is a UUID, which Qdrant accepts as a point ID):

from qdrant_client import QdrantClient, models
qdrant = QdrantClient(url="https://YOUR-CLUSTER.qdrant.io", api_key="YOUR_QDRANT_KEY")
points = [models.PointStruct(id=i["id"], vector=i["embedding"], payload={"url": i["url"], "text": i["text"]})
          for i in client.dataset(run["defaultDatasetId"]).iterate_items() if "embedding" in i]
qdrant.upsert(collection_name="docs", points=points)  # create it first with size=1024, distance=COSINE

To search, embed your queries with the same model (bge-m3) so query and document vectors are comparable (see Embedding search queries above).

Use with AI agents (MCP)

Connect Apify's MCP server (https://mcp.apify.com?tools=tidytools/website-to-embeddings) to Claude, Cursor or any MCP client, then ask e.g. "Embed the first 50 pages of docs.example.com and give me the dataset ID."

Minimal input:

{ "urls": ["https://docs.apify.com/academy"], "maxPages": 20 }

Failed items are not charged and carry an errorType.

Use it from code and integrations

Run it from your own code with the Apify API. This call waits for the run and returns the results as JSON (replace YOUR_TOKEN with your Apify API token):

curl -X POST "https://api.apify.com/v2/acts/tidytools~website-to-embeddings/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://docs.apify.com/academy/web-scraping-for-beginners"],"maxPages":5,"removeBoilerplate":true}'

The synchronous endpoint waits up to 5 minutes. For bigger runs, start the run with POST https://api.apify.com/v2/acts/tidytools~website-to-embeddings/runs and read the dataset when it finishes, or use the apify-client package for JavaScript or Python.

Schedules and integrations: run it daily or weekly with Apify Schedules, get a webhook when a run finishes, or send the results to Zapier, Make, n8n, Google Sheets, Slack and other apps with Apify integrations. Results can be exported as JSON, CSV, Excel or XML.

Limitations

• Only public pages; no logins.
• Input mistakes never fail the run: with no valid URL, text or dataset ID, the run ends normally with free error rows (errorType: "invalid_input", charged: false) and SUMMARY.status NO_RESULTS. A dataset that cannot be read (wrong ID, no permission) gets one free error row too.
• Errors are free rows: an unknown domain or a server that does not answer gives errorType: "unreachable", an invalid SSL certificate also unreachable, and a start URL that robots.txt disallows gives errorType: "robots_disallowed" (other disallowed pages found while crawling are skipped silently and counted in SUMMARY.disallowedByRobotsTxt). None of them is charged.
• A very long page (e.g. a whole book as one text file) produces many chunks, each charged at the chunk price; the start-of-run cost plan assumes about 2 chunks per page, so set a maximum charge per run when you crawl pages you have not seen.
• A start URL that redirects to another address (e.g. another host) is followed: links under the final address are crawled with the same scope, and SUMMARY.redirectedStarts lists the redirect.
• Documentation sites with Same folder: if the start page links to nothing inside its own folder (sites that give every page its own folder, such as /en/getting-started/), that website is crawled from the folder one level up (/en/, never the whole site) and SUMMARY scopeWidened lists it.
• Calling the API? Use the plain urls list: the older startUrls field still works, but Apify rejects the whole run (HTTP 400) when it holds a bare domain or a blank line. Clean-up and tidying are on unless you send "removeBoilerplate": false or "tidyMarkdown": false.
• If a site blocks our servers, Advanced settings → Auto retries from Apify's network; you can add an Apify proxy (billed to your Apify account).
• Each chunk is embedded as-is up to the model's input limit (long chunks are truncated).
• The model is fixed (bge-m3, 1024 dimensions); queries must be embedded with the same model.
• A dataset is read up to Max dataset items (default 1,000). The dataset must be one your Apify account can read.

FAQ

Does the Actor collect usage data? The Actor sends one anonymous usage counter per run (input type, item count range, outcome) to improve the tool; no URLs, content or account data.

Do I need an OpenAI or Cohere API key to get vector embeddings? No. The embedding model (BAAI bge-m3, 1024 dimensions) runs on our side, so there is no API key and no prepaid credits: you only pay per page and per chunk.

Can I load the output into Pinecone, Qdrant or another vector database? Yes. Every chunk comes with a 1024-dimension embedding and a stable UUID id (same URL + position = same id on every run), ready to upsert into Pinecone, Qdrant, Weaviate, Milvus, Chroma, pgvector or any other vector database, so re-runs update instead of duplicating. Qdrant accepts the UUID id as a point ID.

How is it different from Apify's Pinecone and Qdrant integrations? Those integrations are free to run but need your own OpenAI or Cohere embeddings API key and your own vector database, and you pay their platform usage. Here crawling, chunking and embedding happen in one run and the embeddings are included in the chunk price; if you already pay for an embeddings API, embedding with your own key can cost less per chunk.

What is charged and what is free? You pay $0.20 per 1,000 embedded chunks plus a page fee ($1.00 / 1,000 pages in fast mode, $2.50 / 1,000 in browser mode), with no start fee. Failed, blocked and duplicate pages, pages with no text, pages disallowed by robots.txt and exact duplicate chunks are free, and with Embed another Actor's dataset (dataset ID) there is no page fee, only the chunk price.

Support

Open an issue in the Issues tab with your start URL and input. Issues are checked regularly.

Found this useful? A short review on the Store helps other users find it.