Website Change Tracker | Page Change Detection, RAG Sync

What is Website Change Tracker?

Keep a RAG index in sync: only pages added, changed or deleted since your last run, as Markdown chunks with stable IDs, for $1.50 per 1,000 changed pages, no login.

For AI teams keeping a knowledge base current. Each checked page costs $0.50 per 1,000 on top; menus, footers, cookie banners and 'last updated' lines never count as changes. Your first run with the example input costs about $0.043 (20 changed pages in our test run).

Start from	Start URLs or sitemap URLs, plus a corpus ID
Returns	Only added, changed and deleted pages, with chunks to upsert and chunk IDs to delete
IDs	Stable document ID (canonical URL) and content-anchored chunk IDs, valid UUIDs
Automate	Schedules, webhook push of the delta, Zapier, Make, n8n, MCP for AI agents
Price	$1.50 per 1,000 changed pages ($1.05 per 1,000 changed pages on paid Apify plans)
Login needed	No

Why Website Change Tracker users pick this one

• Re-embed only what changed. Each run returns just the pages that changed.
• Stable chunk IDs. Anchored on heading path and text, so edits elsewhere don't move them.
• Explicit deletions. Chunk IDs to delete for edited and removed pages.
• Webhook delta. Chunks to upsert and IDs to delete, pushed in batches.
• No false changes. Broken or blocked pages are flagged, never mistaken for deletions.

Get started in 3 steps

1. Paste start URLs (or a sitemap URL), choose a corpus ID such as 'acme-docs' and set Max pages above the size of the site.
2. Run it once: the baseline returns every page as 'added' with all its chunks (or switch 'Return the baseline' off to record it silently).
3. Schedule the same input (daily or weekly). Each run returns only added, changed and deleted pages; upsert 'upsert_chunks' and delete 'delete_chunk_ids' in your vector database, by hand, through the API, a webhook or n8n (see the FAQ below).

▶ Try it with the example input

The form opens with an example input already filled in: that run costs about $0.043 on the free plan.

How much does it cost to track website changes?

Pay per event: you pay only for what the run returns, plus a small start fee. Bronze, Silver, Gold and higher Apify plans pay less per result.

Changed pages	Free plan	Paid plans (Bronze, Silver, Gold)
100	$0.15	$0.11
1,000	$1.50	$1.05
10,000	$15.00	$10.50

Totals include the $0.003 start fee per run, rounded to the cent. Apify's free plan includes $5 of credit a month, enough for about 3,300 changed pages (or, with checks at $0.50 per 1,000, a 250-page docs site checked daily for a month).

Every returned page was also checked, so 1,000 changed pages cost $2.00 in all on the free plan ($1.50 + $0.50), and a run with no changes costs only the checks and the start fee. Not charged: extraction failures (a known page that came back empty, blocked, JavaScript-only or broken; flagged as 'extraction_failed' for free and its previous version kept), pages blocked by robots.txt, fetch errors, non-HTML responses and duplicates. A page check is charged only when its change record also fits under the maximum charge you set, so nothing is paid for a change you did not receive; changes not returned because of the maximum charge come in the next run.

Billing events

Event	Free plan	Paid plans (Bronze, Silver, Gold)
page-change	$1.50 per 1,000 changed pages	$1.05 per 1,000 changed pages
page-checked	$0.50 per 1,000 pages checked	$0.50 per 1,000 pages checked
apify-actor-start	$0.003 per run	$0.003 per run

Website Change Tracker vs alternatives

	This Actor	ryanclinton/website-change-monitor	muhammad-bilal/web-drift-detector
Price per 1,000, free plan	$0.50 per 1,000 checked + $1.50 per 1,000 changed pages	$100 per 1,000 sites monitored	$0.20 per 1,000 checks + $0.60 per 1,000 results
Price per 1,000, paid plans	$0.50 per 1,000 checked + $1.05 per 1,000 changed	$47 per 1,000 sites monitored	$0.20 per 1,000 checks + $0.60 per 1,000 results
Start fee per run	$0.003	$0.0002 (at 4 GB)	$0.0002 (at 4 GB)
Platform usage billed on top	No	No	No
Markdown chunks to upsert, stable chunk IDs	✓	✗ (line diffs)	✗ (section diffs)
Chunk IDs to delete for edited and removed pages	✓	✗	not stated
Blocked or empty pages never reported as deleted	✓	not stated	not stated
Delta pushed to a webhook	✓ chunks and delete IDs	Change alerts	not stated
Severity scores for human alerts	✗	✓	✓

Prices checked regularly on each Actor's Store page.

Benchmarks: how we compare

Test	Input	Result	Run time
Baseline	docs.apify.com/academy, Max pages 20	20 pages recorded as 284 chunks	22 s
Baseline	docs.python.org/3/tutorial/, Max pages 100	17 pages recorded as 300 chunks	20 s
Re-check with no edits	the same Python tutorial corpus, run again	17 pages checked, 0 added, 0 changed, 0 deleted	16 s
Controlled edit and deletion	an Apify Academy corpus with one stored chunk made stale and one stored page that now returns 404	exactly the expected result: 1 changed page (1 chunk to upsert, the stale ID to delete) and 1 deleted page; 18 unchanged and 6 new pages	24 s
Busy page	Wikipedia's Current events portal (83 chunks), re-checked 3 times in a row	unchanged each time: no false changes from layout or menus	3 to 5 s
Slow site	a site asking for a 6-second Crawl-delay, Max pages 8	8 pages recorded, delay honoured	62 s
Nothing readable	4 silent hosts, an unknown host and a site asking for a 30-second Crawl-delay	nothing recorded and no page charges; only the start fee	28 to 32 s

Apify cloud runs at the default 256 MB with plain HTTP and no proxy, one run per row. The controlled test edited the stored corpus record before the run and compared the run's output with the expected chunk IDs. Pages are fetched at most two a second per site, so a run takes at least half a second per page.

What data you get

Website Change Tracker keeps a website's content in sync with your RAG index: first run = baseline (every page 'added'); later runs only what changed. The first run with a corpus ID records every page as Markdown chunks; every later run re-checks the site and returns only the pages that were added, changed or deleted, each with the exact chunks to upsert and the chunk IDs to delete. Chunk IDs are anchored on content (page URL, heading path and a hash of the normalized text), so an unchanged section keeps its ID when the sections around it change, and you only re-embed what really changed. Navigation, footers, cookie banners and 'last updated' lines are ignored, and a page that comes back empty, blocked or broken is flagged, never reported as deleted.

• change: added, changed, deleted, or extraction_failed (free; the page could not be read this run and its previous version is kept)
• doc_id, url (canonical), fetched_url, title, reason (why deleted or unreadable)
• upsert_chunks: [{chunk_id, position, heading_path, text, char_count, url, citation}] to embed and upsert
• delete_chunk_ids: chunk IDs to remove from your index; upsert_count, delete_count, unchanged_chunk_count, chunk_count
• markdown (whole main content of added and changed pages), word_count, language, last_modified, http_status
• first_seen_at, previous_check_at, scraped_at (ISO 8601 UTC), corpus_id; plus a SUMMARY record per run with all counts

change	url	title	upsert_count	delete_count	unchanged_chunk_count
added	https://docs.apify.com/academy	Apify Academy | Academy | Apify Documentation	9	0	0
added	https://docs.apify.com/academy/advanced-web-scraping	Advanced web scraping | Academy | Apify Documentation	3	0	0
added	https://docs.apify.com/academy/tutorials	What's this section? | Academy | Apify Documentation	1	0	0
added	https://docs.apify.com/academy/scraping-with-ai	Web scraping with AI and Apify | Academy | Apify Docume...	9	0	0
added	https://docs.apify.com/academy/scraping-basics-javascript	Web scraping basics for JavaScript devs | Academy | Api...	22	0	0

One full changed page record, exactly as a real run returned it (long text and lists shortened here):

{
  "corpus_id": "apify-academy-docs",
  "change": "added",
  "doc_id": "d67cabce-5772-5274-acc5-2b5cf3bf4520",
  "url": "https://docs.apify.com/academy",
  "fetched_url": "https://docs.apify.com/academy",
  "title": "Apify Academy | Academy | Apify Documentation",
  "reason": null,
  "upsert_count": 9,
  "delete_count": 0,
  "unchanged_chunk_count": 0,
  "chunk_count": 9,
  "upsert_chunks": [
    {
      "chunk_id": "2f2b178a-262e-52a2-8e63-40fe11712104",
      "position": 0,
      "heading_path": [
        "Apify Academy"
      ],
      "text": "Learn everything about web scraping and automation with our free courses that will turn you into an expert scraper devel... [5 more characters]",
      "char_count": 125,
      "url": "https://docs.apify.com/academy",
      "citation": "Apify Academy (https://docs.apify.com/academy)"
    },
    {
      "chunk_id": "e7b0e55b-909b-5bdf-824a-d81f49cdfe9f",
      "position": 1,
      "heading_path": [
        "Apify Academy",
        "Beginner courses",
        "Web scraping with AI"
      ],
      "text": "Learn how to use AI to extract information from websites in this practical course, starting from the absolute basics.\n\n[... [54 more characters]",
      "char_count": 174,
      "url": "https://docs.apify.com/academy",
      "citation": "Apify Academy > Beginner courses > Web scraping with AI (https://docs.apify.com/academy)"
    },
    {
      "chunk_id": "3fa16237-c166-577f-9709-514f29e55a77",
      "position": 2,
      "heading_path": [
        "Apify Academy",
        "Beginner courses",
        "Web scraping basics with JS"
      ],
      "text": "Learn how to use JavaScript to extract information from websites in this practical course, starting from the absolute ba... [72 more characters]",
      "char_count": 192,
      "url": "https://docs.apify.com/academy",
      "citation": "Apify Academy > Beginner courses > Web scraping basics with JS (https://docs.apify.com/academy)"
    },
    "... [6 more]"
  ],
  "delete_chunk_ids": [],
  "markdown": "# Apify Academy\n\nLearn everything about web scraping and automation with our free courses that will turn you into an exp... [2378 more characters]",
  "word_count": 377,
  "language": "en",
  "last_modified": "2026-11-04T21:51:26Z",
  "http_status": 200,
  "first_seen_at": "2026-11-04T21:57:10Z",
  "previous_check_at": null,
  "source_url": "https://docs.apify.com/academy",
  "scraped_at": "2026-11-04T21:57:10Z"
}

Use cases

• Keeping a RAG chatbot or support assistant in sync with a help center or documentation site
• Updating only changed embeddings in Pinecone, Qdrant, Weaviate, Chroma or pgvector on a schedule
• Feeding AI agents a daily 'what changed in these docs' digest with source citations
• Monitoring competitor docs, policy or pricing pages for real content changes, without boilerplate noise

Use-case recipes

Keep a docs RAG index in sync

For AI developers. Run once with a new corpusId to fill the index, then schedule the same input daily. For each record, embed and upsert every item of upsert_chunks with chunk_id as the vector ID and citation as metadata, delete every ID in delete_chunk_ids, and skip extraction_failed records.

{
  "startUrls": [
    "https://docs.apify.com/academy"
  ],
  "corpusId": "apify-academy",
  "maxPages": 50,
  "chunkSize": 1500
}

Start tracking after a full copy

For teams that already loaded the site with Website to Markdown. Turn off returnBaseline so the first run only records the corpus and you pay per page checked, not per page returned. From the second run on, you get only the changes.

{
  "startUrls": [
    "https://docs.python.org/3/tutorial/"
  ],
  "corpusId": "py-tutorial",
  "maxPages": 50,
  "returnBaseline": false
}

Push changes to n8n or your own endpoint

For automation builders. Set webhookUrl to an n8n, Make or Zapier webhook. Each POST holds up to 25 change records; when final is true, summary has the run's counts, so the workflow knows the sync is complete.

{
  "startUrls": [
    "https://docs.apify.com/academy"
  ],
  "corpusId": "apify-academy-hook",
  "maxPages": 30,
  "webhookUrl": "https://example.com/hooks/rag-sync"
}

Track a whole help center, including orphan pages

For support teams. Use scope = domain and turn on useSitemaps, so pages no menu links to are tracked too. Exclude listings that change on every visit, such as search and tag pages, and set maxPages above the size of the site.

{
  "startUrls": [
    "https://example.com/help/"
  ],
  "corpusId": "help-center",
  "maxPages": 100,
  "scope": "domain",
  "useSitemaps": true,
  "excludeGlobs": [
    "**/search**",
    "**/tags/**"
  ]
}

Input

Field	Type	What it does	Default
startUrls (Start URLs or sitemaps)	array	Pages to start from, e.g. ['https://docs.apify.com/academy'], or sitemap URLs ending in .xml, e.g. ['https://example.com/sitemap.xml'].	["https://docs.apify.com/academy"]
corpusId (Corpus ID)	string	Name of the tracked corpus, e.g. 'apify-academy'.	"default"
maxPages (Max pages checked per run)	integer	Stop after this many pages are checked (and charged), e.g. 500.	500
scope (Crawl scope)	string	'path' = only URLs under the start URL's folder (start https://example.com/docs/ tracks /docs/...), 'domain' = the whole site of each...	"path"
includeGlobs (Include URL globs)	array	Track only links whose full URL matches one of these patterns, e.g. ['https://docs.apify.com/academy/**'].
excludeGlobs (Exclude URL globs)	array	Never follow links whose full URL matches one of these patterns, e.g. ['/changelog/', '**?page=*'].

All 10 fields, with examples: see the Input tab.

Switching from ryanclinton/website-change-monitor

Already running ryanclinton/website-change-monitor? Map its input fields to this Actor's like this:

ryanclinton/website-change-monitor input	This Actor's input	Notes
urls	startUrls	Links are followed within the chosen scope, so one start URL tracks a whole docs section. To watch only the pages you list, add them to includeGlobs too.
kvStoreName	corpusId	State is always kept in the named store aurelix-website-change-tracker in your account, one record per corpus ID; named stores do not expire.
webhookUrl	webhookUrl	Ours POSTs the delta itself (chunks to upsert, chunk IDs to delete) in batches of 25, ending with a final batch that holds the run summary.
notifyOnlyChanges	not needed	After the baseline only changed pages are returned. returnBaseline decides whether the first run returns every page or records it silently.
mode / cssSelector	no equivalent	Always compares the page's main content as Markdown chunks; a single CSS selector cannot be watched.
ignoreNoise / ignorePatterns / ignoreSelectors	not needed	Navigation, footers, cookie banners and volatile lines such as Last updated are removed automatically; custom patterns are not supported.
discoverFromSitemap / maxDiscoveredPages	useSitemaps + maxPages	Sitemap URLs in scope seed the crawl; maxPages (up to 20,000) caps the pages checked per run.
baselineMode	no equivalent	Each run compares against the corpus as last recorded, like a rolling baseline; a locked baseline is not offered.
watchKeywords / minSeverity	no equivalent	No severity scores or keyword escalation; this tool feeds a vector index, not human alerts.
slackWebhookUrl	no equivalent	Use webhookUrl with a Zapier, Make or n8n step if you want a Slack message.

• Output changes from one alert per page to one record per added, changed or deleted page, with upsert_chunks and delete_chunk_ids ready for a vector database.
• Pricing changes from $100 per 1,000 sites monitored to $0.50 per 1,000 pages checked plus $1.50 per 1,000 changed pages, and a $0.003 start fee per run (comparison table above).
• A page that comes back blocked, empty or broken is a free extraction_failed record and its previous version is kept; only HTTP 404/410 or a redirect away counts as deleted.
• Your first run with a corpus ID is the baseline, so expect every page as added (or none returned with returnBaseline off).

Use with AI agents (Apify MCP server, Claude, ChatGPT, n8n AI agent)

Website Change Tracker is a ready-made tool for AI agents. Through the Apify MCP server an agent reads this Actor's input schema, fills in the input from your request, starts the run and reads the changed pages back. You pay the same per-result price as a run started by hand, from your own Apify account.

Example prompts

• "Track https://docs.apify.com/academy as corpus apify-academy, up to 50 pages, and tell me which pages were added, changed or deleted since the last run"
• "Re-check the Python tutorial at https://docs.python.org/3/tutorial/ for corpus py-tutorial and give me the chunk IDs I need to delete from my vector database"
• "Record a silent baseline of https://example.com/help/ as corpus help-center without returning the pages, so later runs only send me changes"

Claude (claude.ai and Claude Desktop)

Open Settings > Connectors, choose Add custom connector, paste https://mcp.apify.com/?tools=aurelix/website-change-monitor and sign in to Apify when asked. Then ask Claude one of the prompts above.

Claude Code, Cursor, VS Code and other MCP clients

$claude mcp add --transport http apify "https://mcp.apify.com/?tools=aurelix/website-change-monitor"

Clients configured with JSON take the same URL:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com/?tools=aurelix/website-change-monitor"
    }
  }
}

A client without Apify sign-in sends the header Authorization: Bearer <YOUR_APIFY_TOKEN>; to run the server locally instead, use npx -y @apify/actors-mcp-server --actors aurelix/website-change-monitor with APIFY_TOKEN set.

ChatGPT

Where your ChatGPT plan offers custom connectors (developer mode), add https://mcp.apify.com/?tools=aurelix/website-change-monitor as a connector, sign in to Apify, and ask for the data in plain words.

n8n AI agent

Give an AI Agent node an MCP Client Tool with the endpoint https://mcp.apify.com/?tools=aurelix/website-change-monitor and a Bearer credential holding your Apify token. The agent can then call Website Change Tracker whenever a workflow step needs changed pages; for a fixed, non-agent workflow use Apify's n8n node with Run Actor instead.

Use it from AI agents (MCP), the API or no-code

Copy this to your AI assistant

Use the Apify Actor aurelix/website-change-monitor (Website Change Tracker) through Apify's MCP server or API. Main input: startUrls (Start URLs or sitemaps); corpusId (Corpus ID). Example input: {"startUrls": ["https://docs.apify.com/academy"], "corpusId": "apify-academy", "maxPages": 300, "webhookUrl": "https://example.com/hooks/rag-sync"}. Each result is one changed page with fields such as corpus_id, change, doc_id, url, fetched_url, title, reason, upsert_count. It costs $1.50 per 1,000 changed pages on Apify's free plan plus a $0.003 start fee per run, and needs no login.

MCP server URL for this Actor: https://mcp.apify.com/?tools=aurelix/website-change-monitor (setup for each client in the section above).

API

Start a run and read the dataset with Apify's REST API or the Python and JavaScript clients:

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("aurelix/website-change-monitor").call(run_input={'startUrls': ['https://docs.apify.com/academy'], 'corpusId': 'apify-academy', 'maxPages': 300, 'webhookUrl': 'https://example.com/hooks/rag-sync'})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

No-code: schedule it in 3 clicks

Send the results to Google Sheets, Zapier, Make, n8n, Slack or any webhook URL.

1. Fill in the input, then click Save as a new task.
2. In the task, click Schedule and pick daily or weekly (Apify Schedules).
3. Paste a webhook URL into Webhook URL for the delta to get what changed after every run (each run already reports only what is new).

Prefer n8n or Make? Ask in the Issues tab and we will share a ready-made workflow for it.

FAQ

Can I track website changes without login?

Yes. It never logs in, so you need no website account, password or cookies; only a free Apify account to run it.

Is there a website change detection API?

You can use this Actor as one. Start it with Apify's REST API or the Python and JavaScript clients and read the results as JSON (see the code above), or call it from an AI agent through MCP.

How much does it cost to track website changes?

About $1.50 for 1,000 changed pages on Apify's free plan and $1.05 on paid plans, start fee included. The free $5 monthly credit covers about 3,300 changed pages (or, with checks at $0.50 per 1,000, a 250-page docs site checked daily for a month).

Can I export website changes to Excel or Google Sheets?

Yes. Download the results as Excel, CSV, JSON, XML or HTML from the run's Storage tab, or send them to Google Sheets with Apify's Google Sheets integration or the API.

Is it legal to monitor websites for changes?

It reads only public data any logged-out visitor can see: the website you choose (track only sites you have the right to crawl). It goes at a polite pace, respects robots.txt, never logs in and never solves CAPTCHAs. It returns each page's main content as Markdown chunks with title and headings; the content is returned as the site published it; copyright in it stays with its owner. To have a record removed, open an issue on this Actor's Issues tab; if a site owner formally asks us to stop, this Actor will stop. You are responsible for using the data lawfully, incl. anti-spam and privacy laws; this is not legal advice.

How are changes detected?

Each page's main content is converted to Markdown, split into chunks by heading and paragraph, and each chunk gets an ID from the page's canonical URL, its heading path and a hash of its normalized text. A run compares the page's chunk IDs with the ones stored for your corpus: new IDs are upserts, missing IDs are deletions, and a page whose set of IDs is unchanged is not returned. Moving sections around or changing whitespace is not a change.

Why do unchanged chunks keep their IDs when the text around them changes?

Chunk boundaries are cut at content-defined points (heading boundaries and paragraph or sentence breaks chosen by their own hash), not at fixed character offsets, so inserting or editing a paragraph re-cuts only the chunk it sits in and, rarely, the next one. In our tests an edited paragraph changed 1 of 7 chunk IDs on its page, and a newly inserted section added 1 chunk and kept all 7 existing IDs.

What is ignored as boilerplate?

Everything outside the page's main content: navigation, headers, footers, sidebars, cookie and consent banners, share buttons, forms and scripts. Inside the content, lines such as 'Last updated: ...', copyright lines, 'x min read' and 'Was this page helpful?' are dropped before comparison. In our tests, pages whose only differences were a new menu, cookie banner, footer and 'last updated' date were not returned, and re-checking 218 real documentation pages (Python, FastAPI and Apify docs) returned 0 false changes.

How is a deleted page different from an extraction failure?

A page is deleted only when the site says so: HTTP 404 or 410, or a redirect to another page or out of scope. A known page that returns an error (403, 429, 5xx), a bot challenge, an empty JavaScript shell, or main content that shrank by more than 90% is returned as 'extraction_failed' with the reason, free of charge, and its previous version stays in the corpus, so a temporary outage never wipes your index.

Where is the state kept?

In a named key-value store 'aurelix-website-change-tracker' in your own Apify account, one record per corpus ID ('corpus-

What does the webhook send?

If you set 'webhookUrl', the run POSTs JSON in batches of 25 records: {corpus_id, run_id, batch, final, changes: [...]}; every run ends with a batch where final is true and 'summary' holds the run's counts. A webhook error is retried 3 times and never fails the run; the delta is always in the dataset too.

Which IDs do I use in my vector database?

Use 'chunk_id' as the point or vector ID (a UUID, accepted as-is by Qdrant, Weaviate, Pinecone, Chroma and pgvector) and store 'doc_id', 'url' and 'citation' as metadata. Upsert every item of 'upsert_chunks' and delete every ID in 'delete_chunk_ids'; skip 'extraction_failed' records. In n8n: Schedule Trigger, then an HTTP Request to this Actor's run-sync-get-dataset-items endpoint, a Code node that splits the records into upserts and one delete list, an embeddings call, and the vector database's upsert and delete endpoints. We share a ready import file for OpenAI embeddings and Qdrant on request through the Issues tab.

Does it respect robots.txt?

Always, and it cannot be switched off. Disallowed URLs are never requested, Crawl-delay is honoured up to 6 seconds (sites asking for more are skipped and not charged), redirects into disallowed paths are not followed, and links on nofollow pages are not followed.

What does a run with no changes cost?

Only the page checks ($0.50 per 1,000) and the $0.003 start fee; nothing is returned, so no change is charged. A 17-page tutorial re-checked with no edits costs 17 checks. Blocked, broken and unreadable pages, robots.txt blocks, fetch errors and duplicates are not charged at all.

What happens on the first run?

The first run with a corpus ID is the baseline. With Return the baseline on (the default), every page comes back as added with all its chunks and is charged as a change. Turn it off to record the baseline silently and pay only per page checked.

What if Max pages is lower than the number of pages on the site?

Pages already in the corpus but not reached in this run are kept, not deleted, and the status message says how many were not reached. New pages found later come back as added. Set Max pages above the size of the site so every known page is re-checked on each run.

What happens when a run hits my maximum charge or its timeout?

Changes that could not be returned are not recorded as seen, so they come back in the next run. A page check is charged only when its change record also fits under the maximum charge. Progress is saved every 200 pages and at the end, and the run stops before its timeout with its results and state saved.

Can I track several sites?

Yes. Each corpus ID is a separate state, so use one corpus ID per site or per index. One corpus can also hold several start URLs. Runs with different corpus IDs never affect each other.

Can I change the chunk size later?

Not for an existing corpus: the chunk size is fixed at its first run so chunk IDs stay stable. Use a new corpus ID to re-chunk; it starts a new baseline.

Where do I find the run's counts?

In the run's status message, in the SUMMARY record of the run's key-value store, and in the final webhook batch when a webhook URL is set: pages checked, added, changed, deleted, unchanged and failed.

Limits (stated honestly)

• No JavaScript rendering: pages that are an empty app shell are skipped (and flagged if already in the corpus).
• Only HTML pages are tracked; PDFs, images and other files are skipped. HTML pages over 2 MB are skipped.
• Max pages caps the pages checked per run (up to 20,000). Known pages not reached in a run are kept, not deleted, and checked in a later run; set Max pages above the size of the site.
• At most two new requests per second per site (slower with a Crawl-delay up to 6 s; sites asking for more are skipped). Conversion is CPU-bound at 256 MB: about 1,000 large documentation pages an hour; the run stops before its timeout with its results and state saved.
• Chunk size is fixed per corpus at its first run; to re-chunk, use a new corpus ID (which starts a new baseline).
• A page that shrinks by more than 90% (from at least 200 words) is treated as broken, not as a change; if the shrink is real, delete the corpus record or use a new corpus ID.
• rel=canonical is followed only within the same site and scope; a site whose canonical tags all point to one page will collapse to that page.
• If the site stops responding (15 errors in a row), the run stops early and the corpus is kept as it was.
• Pages behind a login, paywall or bot challenge are not accessible.

Works great with

• Before this: Website to Markdown API: one full Markdown copy of the same startUrls to start the corpus

Related Aurelix tools

Public Aurelix Actors, grouped by the data they collect. All are pay-per-result and need no login.

AI, content and SEO

• Google Trends by Region: Google Trends interest over time, by region and rising queries for any number of keywords, plus trending searches.
• Google Autocomplete API: Expands seed keywords into hundreds of real Google and YouTube autocomplete phrases, tagged by search intent.
• PDF to Markdown OCR & Table Extraction: Turns linked PDF and Word files into Markdown per page, with tables as JSON rows, RAG chunks and OCR.
• Broken Link Crawler & SEO Audit: Audits a whole site: one row per page with SEO checks and broken links, plus a ranked issues summary.
• Website to Markdown API: Crawls a site and returns one row of clean main-content Markdown per page, with title, headings and links.

Jobs and hiring data

• Greenhouse Jobs Scraper: Live jobs from a company's Greenhouse, Lever, Ashby or SmartRecruiters board, with salary and Canadian province tags.
• LinkedIn Jobs Country Filter Scraper: Public LinkedIn job postings by keyword and country: title, company, location, seniority, job type, salary text, description.
• Multiple Job Boards Scraper: LinkedIn, about 50 company career sites, Remote OK and We Work Remotely jobs in one deduplicated list.

E-commerce and product data

• AliExpress Scraper API: AliExpress search results by keyword: price, discount, rating, units sold, ship-from country and sponsored flag.
• Amazon New Releases & Best Sellers: Amazon Best Sellers and New Releases top 100 on 10 marketplaces: rank, ASIN, price, rating, rank change.
• Shopify Product Export Scraper: Every public product of any Shopify store with variants, SKUs, prices, barcodes, stock counts and collections.

Leads and company data

• Newly Funded Companies Hiring: Companies that just raised money (SEC Form D), with their open roles and business contacts, as a new-only monitor.
• Google Maps Phone Scraper: Local business lists from Google Maps: phones, websites, ratings, hours, coordinates; optional website emails and socials.
• Recently Funded Companies API: US companies that just filed an SEC Form D: amount raised, industry, state, executives, confirmed website.
• Tech Stack API: CMS, e-commerce platform, analytics, ad pixels, chat, CDN and email provider for each domain you enter.

Ad intelligence

• Competitor Watch: One report per competitor with only what is new: Meta and TikTok ads, page changes, Shopify products and prices, jobs.
• Facebook Ad Library Search & API: Facebook and Instagram business ads from the Meta Ad Library: copy, creatives, landing pages, dates, optional EU reach.

Canada and government data

• Canada Building Permits: Building permits and development applications from 7 Canadian cities: address, type, status, dates, value, applicant, contractor and Toronto planner contact.

More data tools

• Tennis Betting Odds Scraper: Tennis results and upcoming matches with opening and closing odds per bookmaker and implied probabilities.

Recent changes

• 0.1: Store page corrected: there is no ready-made n8n template; the FAQ describes the workflow and we share an import file on request.

The full history is on the Changelog tab.

Tested daily

An automated health check runs this Actor every day on a fixed input and checks that at least 0 changed pages come back with corpus_id, change, doc_id, url, scraped_at filled in; a failed check alerts the maintainer straight away.

Your feedback

Missing a field, found a bug or need another filter? Open an issue on the Issues tab; we aim to answer within a day. If this Actor saved you time, a short review helps other people find it.