Website to Knowledge Base Sync | Docs to LLM for RAG

What is Website to Knowledge Base Sync?

Turn any public website into a RAG knowledge base and keep it in sync: only changed chunks with stable IDs, $0.15 per 1,000 changed chunks with page checks, no login.

For AI teams keeping a chatbot or agent current. Prices: $0.10 per 1,000 changed chunks plus $0.50 per 1,000 pages checked; push changes into your own Pinecone or Qdrant index. Your first run with the example input costs about $0.043 (284 changed chunks in our test run).

▶ Watch the 75-second demo

Start from	Public start URLs or sitemap URLs
Returns	One row per chunk: op 'upsert' (text to embed) or 'delete' (chunk ID to remove)
IDs	Content-anchored chunk IDs and page IDs, valid UUIDs
Vector database	Optional push to Pinecone or Qdrant with your own key
Automate	Schedules, Zapier, Make, n8n, MCP for AI agents
Price	$0.15 per 1,000 changed chunks ($0.13 per 1,000 changed chunks on paid Apify plans) with page checks included, counting one page check per 10 changed chunks (first runs measured 14 to 26 chunks a page, so usually less; a quiet day costs only $0.50 per 1,000 pages checked)
Free to try	Runs on Apify's free plan: the $5 monthly credit covers about 33,000 changed chunks
Login needed	No

Daily monitor: only what's new, every morning

Set it once, get only the new changed chunks every morning. With Only changes after the first run on (the default), the first run returns every chunk; each later run returns only chunks to upsert and chunk IDs to delete, so you pay for the page checks plus what changed.

{
  "startUrls": [
    "https://docs.python.org/3/tutorial/"
  ],
  "syncId": "python-tutorial",
  "maxPages": 100,
  "incremental": true
}

1. Paste this input into the Input tab (JSON view) and click Save as a new task; it appears under Saved tasks.
2. Open the task from Saved tasks and click Schedule.
3. Choose Daily, pick the hour, and save.

Cost example: the Python tutorial (17 pages) with about 10 changed chunks a day ≈ $0.44/month on the free plan ($0.43 on paid plans): one run a day for 30 days, start fee included. The first run returns every chunk (about 300 for this site), so it costs about $0.044.

Why Website to Knowledge Base Sync users pick this one

• Re-embed only what changed. After the first run, only new, edited and deleted chunks.
• Stable chunk IDs. Anchored on heading path and text, valid UUIDs.
• Direct vector database sync. Pinecone or Qdrant with your own keys.
• No false deletes. Broken or blocked pages keep their chunks.

Get started in 3 steps

1. Paste public start URLs (or a sitemap URL) and set Max pages above the size of the site.
2. Run it once: every chunk comes back as 'upsert' with its stable chunk_id. Embed and load them, or let the Actor push them into Pinecone or Qdrant.
3. Schedule the same input daily. Each run returns only chunks to upsert and chunk IDs to delete; apply them by chunk_id in your vector database, or let the Actor do it.

▶ Try it with the example input

The form opens with an example input already filled in: that run costs about $0.043 on the free plan.

How much does it cost to sync a website to a knowledge base?

Pay per event: you pay for each page the run checks and each changed chunk it returns, plus a small start fee. Bronze, Silver, Gold and higher Apify plans pay less per result.

Changed chunks	Free plan	Paid plans (Bronze, Silver, Gold)
100	$0.02	$0.018
1,000	$0.16	$0.14
10,000	$1.51	$1.31

Totals include the $0.005 start fee per run and one $0.0005 page-checked fee per 10 changed chunks (first runs measured 14 to 26 chunks a page, so real totals are usually lower; a page with no change costs only its check), rounded to the cent (amounts under $0.10 to four decimals). Apify's free plan includes $5 of credit a month, enough for about 33,000 changed chunks.

A page check is charged once per page compared in a run (new, changed, deleted or unchanged), and each returned upsert or delete row is one changed chunk, so a quiet day with no edits costs only the page checks and the start fee. Not charged: pages that could not be read (errors, bot challenges, JavaScript-only pages; their chunks are kept), pages blocked by robots.txt, fetch errors, non-HTML responses and duplicates. A page is charged only when its page check and all its changed chunks fit under the maximum charge you set; anything not returned because of the maximum charge comes in the next run. Embedding and vector database costs are billed by those providers to your own account.

Billing events

Event	Free plan	Paid plans (Bronze, Silver, Gold)
chunk-change	$0.10 per 1,000 changed chunks	$0.08 per 1,000 changed chunks
page-checked	$0.50 per 1,000 pages checked	$0.50 per 1,000 pages checked
apify-actor-start	$0.005 per run	$0.005 per run

Benchmarks: how we compare

Test	Input	Result	Run time
First run	docs.apify.com/academy, Max pages 20	20 pages, 284 chunks to upsert	23 s
First run	fastapi.tiangolo.com/tutorial/, Max pages 70	45 pages, 800 chunks to upsert	176 s
Re-run with no edits	the same FastAPI sync	45 pages unchanged, 0 false changes; 6 pages found for the first time returned as new	164 s
Controlled edit and deletion	the Apify Academy sync with one stored chunk made stale and one stored page that now returns 404	exactly 1 chunk to upsert and 2 chunk IDs to delete; 19 pages unchanged	20 s
Slow site	a site asking for a 6-second Crawl-delay, Max pages 8	8 pages, 213 chunks, delay honoured	68 s

Apify cloud runs at the default 256 MB with plain HTTP and no proxy, one run per row. The controlled test edited the stored memory record before the run and compared the output with the expected chunk IDs. Pages are fetched at most two a second per site.

What data you get

Website to Knowledge Base Sync turns a public website or docs site into a RAG knowledge base and keeps it in sync. The first run returns every page's main content as text chunks with stable IDs; every later run re-checks the site and returns only the chunks to upsert (new or edited) and the chunk IDs to delete, one row per chunk with an 'op' field. Chunk IDs are anchored on content (page URL, heading path and a hash of the normalized text), so an unchanged section keeps its ID and you re-embed only what changed. Optionally it pushes the changes straight into your own Pinecone index or Qdrant collection with your own keys, embedding only new and edited chunks with your own OpenAI key.

• op: 'upsert' (add or replace this chunk) or 'delete' (remove this chunk ID from your index)
• chunk_id, doc_id (stable UUIDs), url (canonical page URL), chunk_url (with #anchor), title
• heading_path, chunk_index, text (Markdown), char_count, citation (upserts)
• reason (deletes: 'chunk changed or removed' or 'page returned HTTP 404'), sync_id, source_url, scraped_at (ISO 8601 UTC)
• embedding: only with your OpenAI key and no vector database, the chunk's vector
• Plus a SUMMARY record per run: pages checked, new, changed, deleted and unchanged, chunks upserted and deleted, index size, pages that could not be read, vector database result

op	url	heading_path	char_count	chunk_id
upsert	https://docs.apify.com/academy	["Apify Academy"]	125	2f2b178a-262e-52a2-8e63-40fe11712104
upsert	https://docs.apify.com/academy	["Apify Academy", "Beginner courses", "Web scraping with ...	174	e7b0e55b-909b-5bdf-824a-d81f49cdfe9f
upsert	https://docs.apify.com/academy	["Apify Academy", "Beginner courses", "Web scraping basic...	192	3fa16237-c166-577f-9709-514f29e55a77
upsert	https://docs.apify.com/academy	["Apify Academy", "Beginner courses", "Web scraping basic...	184	c67ccd3d-b552-5c87-81b2-10d7dcf8cc3a
upsert	https://docs.apify.com/academy	["Apify Academy", "Beginner courses", "Apify platform"]	228	e961c229-16d2-5b80-9a15-7c9c9143d7ea

One full changed chunk record, exactly as a real run returned it (long text and lists shortened here):

{
  "op": "upsert",
  "chunk_id": "e961c229-16d2-5b80-9a15-7c9c9143d7ea",
  "doc_id": "d67cabce-5772-5274-acc5-2b5cf3bf4520",
  "url": "https://docs.apify.com/academy",
  "chunk_url": "https://docs.apify.com/academy",
  "title": "Apify Academy | Academy | Apify Documentation",
  "heading_path": [
    "Apify Academy",
    "Beginner courses",
    "Apify platform"
  ],
  "chunk_index": 4,
  "text": "The Apify platform is the best place to run your scrapers and automations in the cloud. Learn what an actor is, how to t... [108 more characters]",
  "char_count": 228,
  "citation": "Apify Academy > Beginner courses > Apify platform (https://docs.apify.com/academy)",
  "reason": null,
  "sync_id": "site-487918db87be",
  "source_url": "https://docs.apify.com/academy",
  "scraped_at": "2026-11-14T14:06:14Z"
}

Use cases

• Keeping a RAG chatbot or support assistant in sync with a help center or documentation site
• Feeding docs to an LLM or AI agent as a knowledge base with source citations
• Daily vector database sync into Pinecone or Qdrant without re-embedding the whole site
• Building a website-to-knowledge-base pipeline in n8n, Make or your own code from one row per change

Use-case recipes

Keep a docs chatbot current

For AI developers. Run once to fill the index, then schedule the same input daily. For each 'upsert' row embed 'text' and upsert by 'chunk_id' with 'citation' as metadata; for each 'delete' row delete that chunk_id.

{
  "startUrls": [
    "https://docs.apify.com/academy"
  ],
  "syncId": "apify-academy",
  "maxPages": 50
}

Sync straight into Pinecone

For teams on Pinecone. Add your Pinecone key, index host and OpenAI key; only new and edited chunks are embedded and written, and gone chunks are deleted from the namespace.

{
  "startUrls": [
    "https://docs.python.org/3/tutorial/"
  ],
  "maxPages": 100,
  "vectorDatabase": "pinecone",
  "pineconeApiKey": "<your Pinecone key>",
  "pineconeIndexHost": "<your index host>",
  "pineconeNamespace": "python-docs",
  "openaiApiKey": "<your OpenAI key>"
}

Whole help center from its sitemap

For support teams. Use the sitemap so pages no menu links to are included, and exclude listing pages that change on every visit.

{
  "startUrls": [
    "https://docs.apify.com/sitemap.xml"
  ],
  "scope": "domain",
  "maxPages": 2000,
  "excludeGlobs": [
    "**/search**",
    "**/tags/**"
  ]
}

Input

Field	Type	What it does	Default
startUrls (Start URLs or sitemaps)	array	Public pages to start from, e.g. ['https://docs.apify.com/academy'], or sitemap URLs ending in .xml, e.g. ['https://example.com/sitemap.xml'].	["https://docs.apify.com/academy"]
maxPages (Max pages checked per run)	integer	Stop after this many pages are fetched in a run, e.g. 500 (pages that cannot be read count too, but...	500
incremental (Only changes after the first run)	boolean	On (default): the first run returns every chunk, each later run only chunks to upsert and chunk IDs to delete.	true
scope (Crawl scope)	string	'path' = only URLs under the start URL's folder (start https://example.com/docs/ syncs /docs/...), 'domain' = the whole site of each...	"path"
includeGlobs (Include URL patterns)	array	Sync only URLs matching one of these patterns, e.g. ['https://docs.example.com/guides/**'].
excludeGlobs (Exclude URL patterns)	array	Never sync URLs matching these patterns, e.g. ['/changelog/', '**?page=*'].

All 20 fields, with examples: see the Input tab.

Use with AI agents (MCP)

MCP (Model Context Protocol) is the open standard that lets AI assistants such as Claude, ChatGPT and Cursor call outside tools. Apify's hosted MCP server at https://mcp.apify.com/?tools=aurelix/rag-site-sync gives your assistant this Actor only: it fills in the input from your request, runs it and reads the changed chunks back, at the same per-result price as a run started by hand.

Example prompts

• "Sync https://docs.python.org/3/tutorial/ into knowledge base chunks, up to 50 pages, and show me the chunks with their citations"
• "Re-run the knowledge base sync for https://docs.apify.com/academy and tell me which chunks I need to upsert or delete in my vector database"
• "Turn the help center at https://example.com/help/ into RAG chunks of 800 characters with 100 characters of overlap"

Claude Desktop and claude.ai

Settings > Connectors > Add custom connector, name it Apify, paste https://mcp.apify.com/?tools=aurelix/rag-site-sync, keep the detected OAuth settings, select Add and sign in to Apify.

Claude Code

$claude mcp add --transport http apify "https://mcp.apify.com/?tools=aurelix/rag-site-sync"

Cursor and VS Code

Add this to ~/.cursor/mcp.json (Cursor); in VS Code's .vscode/mcp.json put the same entry under "servers" with "type": "http". Sign in to Apify when asked.

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com/?tools=aurelix/rag-site-sync"
    }
  }
}

ChatGPT

Turn on Developer mode, open Settings > Apps & Connectors > Create, paste the URL above as the MCP server URL, choose OAuth and authorize Apify.

Clients without sign-in send Authorization: Bearer <YOUR_APIFY_TOKEN> instead. No Apify account? This Actor supports Apify's agentic payments: an agent can pay per run with a prepaid, spend-capped token, bought with x402 stablecoin payments.

n8n AI agent

Give an AI Agent node an MCP Client Tool with the endpoint https://mcp.apify.com/?tools=aurelix/rag-site-sync and a Bearer credential holding your Apify token.

Use it from AI agents (MCP), the API or no-code

Copy this to your AI assistant

Use the Apify Actor aurelix/rag-site-sync (Website to Knowledge Base Sync) through Apify's MCP server or API. Main input: startUrls (Start URLs or sitemaps). Example input: {"startUrls": ["https://docs.apify.com/academy"], "maxPages": 300, "syncId": "apify-academy", "chunkSize": 1500}. Each result is one changed chunk with fields such as op, chunk_id, doc_id, url, chunk_url, title, heading_path, chunk_index. It costs $0.15 per 1,000 changed chunks on Apify's free plan on full result pages ($0.0001 per changed chunk plus $0.0005 per result page of 10) plus a $0.005 start fee per run, and needs no login.

MCP server URL for this Actor: https://mcp.apify.com/?tools=aurelix/rag-site-sync (setup for each client in the section above).

API

Start a run and read the dataset with Apify's REST API or the Python and JavaScript clients:

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("aurelix/rag-site-sync").call(run_input={'startUrls': ['https://docs.apify.com/academy'], 'maxPages': 300, 'syncId': 'apify-academy', 'chunkSize': 1500})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

No-code: schedule it in 3 clicks

Send the results to Google Sheets, Zapier, Make, n8n, Slack or any webhook URL.

1. Fill in the input, then click Save as a new task.
2. In the task, click Schedule and pick daily or weekly (Apify Schedules).
3. Turn on Only changes after the first run so each run returns only what is new.

Prefer n8n or Make? Ask in the Issues tab and we will share a ready-made workflow for it.

FAQ

Can I sync a website to a knowledge base without login?

Yes. It never logs in, so you need no website account, password or cookies; only a free Apify account to run it.

Is there a website to knowledge base API?

You can use this Actor as one. Start it with Apify's REST API or the Python and JavaScript clients and read the results as JSON (see the code above), or call it from an AI agent through MCP.

How much does it cost to sync a website to a knowledge base?

About $0.16 for 1,000 changed chunks on Apify's free plan and $0.14 on paid plans, start fee and page fees included. The free $5 monthly credit covers about 33,000 changed chunks.

Can I export website chunks to Excel or Google Sheets?

Yes. Download the results as Excel, CSV, JSON, XML or HTML from the run's Storage tab, or send them to Google Sheets with Apify's Google Sheets integration or the API.

Is it legal to sync a website to a knowledge base?

It reads only public data any logged-out visitor can see: the website you choose (sync only sites you have the right to crawl). It goes at a polite pace, respects robots.txt, never logs in and never solves CAPTCHAs. It returns each page's main content as text chunks with title and headings, as the site published it; copyright in it stays with its owner. To have a record removed, open an issue on this Actor's Issues tab; if a site owner formally asks us to stop, this Actor will stop. You are responsible for using the data lawfully, incl. anti-spam and privacy laws; this is not legal advice.

Is it cheaper than the leading tools?

For keeping an index current, yes: you pay $0.50 per 1,000 pages checked plus $0.10 per 1,000 changed chunks (about $0.12 to $0.14 per 1,000 chunks on the first runs we measured, page checks included), instead of paying to re-crawl and re-embed the whole site. A quiet day costs only the page checks and the start fee.

Can I use it without a vector database?

Yes. By default the changes are only in the dataset, one row per chunk, ready for Google Sheets, n8n, Make, Zapier or your own code; the vector database push is optional.

Can an AI agent use it?

Yes. Through Apify's MCP server an agent can sync a docs site and read only the changed chunks with their citations; see 'Use it from AI agents' below.

How are changes detected?

Each page's main content is converted to Markdown and split into chunks by heading, paragraph and sentence. Each chunk's ID comes from the page URL, its heading path and a hash of its normalized text. A run compares the page's chunk IDs with the ones it remembered from the previous run: new IDs become 'upsert' rows, missing IDs become 'delete' rows, and a page whose IDs are all the same returns nothing. Whitespace changes and moved menus are not changes. Pages that redirect or point by rel=canonical to another page with the same content are merged into it, and the old URL's chunks are deleted; a URL with and without a trailing slash counts as one page.

Why do unchanged chunks keep their IDs?

Chunk boundaries are cut at content-defined points (heading boundaries and paragraph or sentence breaks chosen by their own hash), not at fixed character offsets, so editing a paragraph re-cuts only the chunk it sits in and, rarely, the next one. With overlap on, the chunk after an edited one also gets a new ID, because its text includes the overlap.

Where is the memory kept, and how do I start over?

In a named key-value store 'aurelix-rag-site-sync' in your own Apify account: one record per sync (sync ID plus chunk size and overlap), read once and written once per run. Re-running the same input reuses it. To start over, use a new sync ID or delete the record. Changing chunk size or overlap starts a new sync, because the IDs change. Pages not checked for 180 days are forgotten.

How do I connect Pinecone or Qdrant?

Choose the database in 'Push changes to a vector database'. Pinecone: your API key and the index host from the Pinecone console; with an OpenAI key the chunks are embedded and written as vectors with the text and citation as metadata, without one the index's integrated embedding is used (set the text field your index embeds). Qdrant: your cluster URL, API key, collection and an OpenAI key; the collection is created with cosine distance if it does not exist. Keys are stored encrypted by Apify and never logged or returned.

What happens if my vector database or OpenAI key fails?

The keys are checked before any page is fetched; a wrong key or host ends the run at once with the reason, and only the start fee is charged. If the database refuses a write during the run, the run stops, that page is not charged and not remembered, and it is sent again on the next run, so the database never silently misses a change.

Which IDs do I use in my own vector database?

Use 'chunk_id' as the vector or point ID (a UUID, accepted by Pinecone, Qdrant, Weaviate, Chroma and pgvector) and store 'url', 'doc_id' and 'citation' as metadata. For each 'upsert' row embed 'text' and upsert by chunk_id; for each 'delete' row delete that chunk_id.

Does it respect robots.txt?

Always, and it cannot be switched off. Disallowed URLs are never requested, Crawl-delay is honoured up to 6 seconds (sites asking for more are skipped and not charged), and links on nofollow pages are not followed.

Limits (stated honestly)

• No JavaScript rendering: pages that are an empty app shell are skipped (and keep their chunks if already known).
• Only HTML pages; PDFs, images and other files are skipped. HTML pages over 2 MB are skipped.
• Max pages caps the pages fetched per run (up to 5,000), including pages that could not be read. Known pages not reached keep their chunks and are checked in a later run; set Max pages above the size of the site.
• Pages are found by following links from the start URLs. A first run can miss pages no crawled page links to; a later run returns them as new when it finds them. Turn on 'Also use the sites' sitemaps' for complete coverage.
• Sitemaps (and robots.txt) are read within 30 seconds at the start. On a slow site with a big sitemap index, the sitemap is skipped and pages are found by following links from the start URLs; a site that asks for a Crawl-delay over 6 seconds is not crawled, even when robots.txt answers late.
• At most two new requests per second per site (slower with a Crawl-delay up to 6 s; sites asking for more are skipped). The run stops early, with everything saved so far kept, when 10 pages in a row cannot be read (blocked, errors, no answer), when no page can be read for 45 s, or when the site answers so slowly that the run takes more than 12 s per page.
• One sync holds up to 300,000 chunks (memory). A run that would go past it stops with a message; use a larger chunk size or split the site into several syncs.
• Changing chunk size or overlap, or the start URLs without a sync ID, starts a new sync with new IDs. Known pages that your include/exclude patterns now leave out are deleted from the index (delete rows) and no longer fetched.
• A page that shrinks by more than 90% (from at least 5 chunks) is treated as broken and keeps its chunks; if the shrink is real, start a new sync.
• Vector database sync covers Pinecone and Qdrant; embeddings use OpenAI only. For other databases, apply the dataset rows yourself by chunk_id.
• Pages behind a login, paywall or bot challenge are not accessible.

Works great with

• Before this: Website to Markdown API: one Markdown file per page of the same site, for a one-off copy or an LLM prompt
• Before this: PDF Table Extraction & PDF to Markdown OCR: PDF and Word files on the same site as Markdown chunks, which this Actor skips

Related Aurelix tools

Public Aurelix Actors, grouped by the data they collect. All are pay-per-result and need no login.

AI, content and SEO

• Google Maps Rank Tracker: Extract Google Maps local rankings on a geo-grid at $1.00 per 1,000 grid points on the free plan: your rank...
• Google Trends by Region: Google Trends interest over time, by region and rising queries for any number of keywords, plus trending searches.
• Google Autocomplete API: Expands seed keywords into hundreds of real Google and YouTube autocomplete phrases, tagged by search intent.
• PDF Table Extraction & PDF to Markdown OCR: Turns linked PDF and Word files into Markdown per page, with tables as JSON rows, RAG chunks and OCR.
• Broken Link Crawler & SEO Audit: Audits a whole site: one row per page with SEO checks and broken links, plus a ranked issues summary.
• Bulk TikTok Transcripts Scraper: Returns TikTok video transcripts as text, timed segments and SRT, from video links or @profiles' latest videos.
• Website Change Tracker: Re-checks a site and returns only added, changed and deleted pages, with chunks to upsert and IDs to delete.
• Bulk Website Screenshot Tool: Full-page screenshots or PDFs of a list of URLs, cookie banners hidden, lazy images loaded, a public link per image.
• Website to Markdown API: Crawls a site and returns one row of clean main-content Markdown per page, with title, headings and links.
• YouTube Transcript Scraper: Returns YouTube video transcripts as text, timed segments and AI chunks, from videos, channels, playlists or searches.
• YouTube Video Scraper API: Full details for every YouTube video in a search, channel or playlist: views, likes, comments, tags, publish time.

Jobs and hiring data

• Job Bank Canada Scraper API: Official Job Bank Canada postings by month: NOC and NAICS codes, wages, hours and location.
• Greenhouse Jobs Scraper: Live jobs from a company's Greenhouse, Lever, Ashby or SmartRecruiters board, with salary and Canadian province tags.
• Career Page Monitor: New, removed and reposted roles plus hiring velocity per company, from seven job-board systems including Workday.
• LinkedIn Jobs Country Filter Scraper: Public LinkedIn job postings by keyword and country: title, company, location, seniority, job type, salary text, description.
• Multiple Job Boards Scraper: LinkedIn, about 50 company career sites, Remote OK and We Work Remotely jobs in one deduplicated list.
• Remote Job Board Scraper API: Remote OK and We Work Remotely jobs in one deduplicated list with salary, regions and tags.
• Wellfound Jobs API & Scraper: Wellfound startup jobs with salary and equity ranges, company stage and size, $0.50 per 1,000.
• Workday Job Postings Scraper: Jobs from any Workday career site by link or company name, with full descriptions and Canadian province tags.

E-commerce and product data

• AliExpress Scraper API: AliExpress search results by keyword: price, discount, rating, units sold, ship-from country and sponsored flag.
• Amazon Best Sellers & Amazon New Releases: Amazon Best Sellers and New Releases top 100 on 10 marketplaces: rank, ASIN, price, rating, rank change.
• Best Buy Canada Scraper API: Best Buy Canada search results: CAD price, sale savings, rating, reviews and seller, plus optional UPC and specs.
• Shopify App Store Scraper: Shopify App Store apps with pricing plans, ratings, category ranks and reviews; new-app and new-review watch.
• Shopify Product Export Scraper: Every public product of any Shopify store with variants, SKUs, prices, barcodes, stock counts and collections.
• Shopify Best Sellers Tracker: Competitor Shopify stores' best-sellers, new products, price changes, stock-outs and restocks since your last run, plus catalogue summary.

Leads and company data

• Companies House Scraper: Newly incorporated UK companies from Companies House: SIC codes, region, registered office, directors with roles.
• Contact Page Scraper: Published company emails, phones, addresses, social pages and team members from a list of company websites.
• Decision Maker Finder: Founders, CEOs, VPs and department heads from company team pages, with title, seniority and function.
• FMCSA API: FMCSA carriers by DOT, MC or name, plus daily new-authority leads: contacts, fleet, cargo, insurance, out of service.
• Newly Funded Companies Hiring: Companies that just raised money (SEC Form D), with their open roles and business contacts, as a new-only monitor.
• Google Maps Phone Scraper: Local business lists from Google Maps: phones, websites, ratings, hours, coordinates; optional website emails and socials.
• New Business Licenses Scraper: Newly licensed businesses from 11 US and Canadian city open-data portals in one schema, with category, issue date and address.
• Recently Funded Companies API: US companies that just filed an SEC Form D: amount raised, industry, state, executives, confirmed website.
• Tech Stack API: CMS, e-commerce platform, analytics, ad pixels, chat, CDN and email provider for each domain you enter.
• Yellow Pages Canada Scraper: YellowPages.ca businesses by search term and city: phone, website, address, postal code, hours and rating.

Ad intelligence

• Competitor Watch: One report per competitor with only what is new: Meta and TikTok ads, page changes, Shopify products and prices, jobs.
• Facebook Ad Library Search & API: Facebook and Instagram business ads from the Meta Ad Library: copy, creatives, landing pages, dates, optional EU reach.
• TikTok Ads Library API: TikTok Ads Library ads for Europe, UK, Switzerland and Turkey with targeting, reach, ad text and landing pages.

Canada and government data

• Canada Building Permits: Building permits and development applications from 7 Canadian cities: address, type, status, dates, value, applicant, contractor and Toronto planner contact.
• Government Tenders Scraper: Open tenders from SAM.gov, CanadaBuys and EU TED in one table: buyer, codes, value, deadline, contact and link.
• New York Building Permits Scraper: Building permits from 12 US city open-data portals in one schema, with value, status, contractor, owner and applicant.
• USPTO Trademark API: US trademark search and watch from USPTO: mark, owner or number lookups, new filings, owner, attorney, goods.

More data tools

• Agoda API (Unofficial): Scrape Agoda guest reviews at $0.50 per 1,000 with this unofficial Agoda API, cheaper than the leading tools: Agoda and...
• Merge CSV Files & Datasets: Remove Duplicates, Excel, JSON: Merges datasets, CSV, JSON and Excel files and removes duplicates with careful URL, email and number matching.
• Polymarket Odds Tracker: Polymarket outcomes with implied probability, bid/ask, volume and results, plus price history and price-move alerts.
• Tennis Betting Odds Scraper: Tennis results and upcoming matches with opening and closing odds per bookmaker and implied probabilities.

Recent changes

• 0.1: Redirected, canonical and trailing-slash duplicate pages are merged and their old chunks deleted; excluded pages are deleted; a site-wide 404 must repeat on the next run before chunks are deleted; runs on hanging or blocking sites stop within about a minute; one sync holds up to 300,000 chunks.

The full history is on the Changelog tab.

Tested daily

An automated health check runs this Actor every day on a fixed input and checks that the run succeeds and every changed chunk it returns has op, chunk_id, doc_id, url, sync_id, scraped_at filled in; a failed check alerts the maintainer straight away.

Was this useful?

• Something went wrong? Open an issue and tell us what you ran: we answer fast and aim to reply within a day.
• It worked? Leave a review: it takes a minute and helps other people find this tool.