Pricing

Pay per event

Try for free

Go to Apify Store

RAG Incremental Reindexer - Only Changed Pages & Chunks

Try for free

Tells you which of your RAG sources actually changed since the last run, and returns embedding-ready chunks only for the delta. Saves embedding costs by skipping unchanged pages.

Pricing

Pay per event

Rating

0.0

(0)

Developer

Aleksandr Dvorzhitckii

Actor stats

Bookmarked

Total users

Monthly active users

3 days ago

Last modified

Categories

Developer tools

URLs to check

startUrls

Required

One or more pages to compare against your stored baseline. Only server-rendered / static HTML is supported - JavaScript-only pages cannot be read without a browser.

Type:array

Max pages

maxPages

Optional

Safety limit. The Actor stops after checking this many pages, which prevents accidental runaway costs on large URL lists.

Type:integer

Minimum:1

Default:500

Fetch engine

crawlerType

Optional

Cheerio is a plain HTTP client: cheapest and fastest, but returns nothing for sites that render text with JavaScript. jsdom can execute some scripts in-process but is noticeably slower and less reliable. Sites built as SPAs will need a browser-based Actor instead of this one.

Type:string

Default:cheerio

Options:

cheeriojsdom

Proxy configuration

proxyConfiguration

Optional

Optional. Enable if the sites you check block requests from datacenter IPs.

Type:object

Default:

{
  "useApifyProxy": false
}

State store name

stateStoreName

Optional

Name of the named Key-Value Store that persists your baseline between runs. This store is retained indefinitely, so scheduled runs always have something to compare against. Use a different store name for each separate project or corpus.

Type:string

Default:rag-reindexer-baseline

Reset baseline (re-index everything)

resetBaseline

Optional

When enabled, the Actor discards the stored baseline and treats every page as changed. Use it on the very first run if you want a full corpus, or after a major content migration.

Type:boolean

Default:false

What to compare

compareMode

Optional

Comparing visible text instead of raw HTML is what makes the detection useful: raw HTML 'changes' on every page view because of ads, counters and nonces. Use 'text-and-metadata' if a title or description change should also trigger re-indexing.

Type:string

Default:text

Options:

texttext-and-metadata

Mask volatile content

maskVolatile

Optional

Patterns that are replaced with placeholders before hashing, so they never count as a real change.

Type:object

Default:

{
  "dates": true,
  "dateFormats": [
    "iso",
    "slash",
    "dots",
    "long"
  ],
  "viewCounters": true
}

Minimum change threshold (%)

changeThresholdPercent

Optional

A page is only reported as changed when at least this percentage of the visible text differs. Set to 0 to report any difference. Raising it (for example to 2) filters out trivial wording tweaks that are not worth re-embedding.

Type:number

Minimum:0

Maximum:100

Default:0

Emit embedding-ready chunks

emitChunks

Optional

Include a list of text chunks (with character offsets) for every changed page, ready to be embedded and upserted. Charges one 'chunk-exported' event per chunk.

Type:boolean

Default:true

Target chunk size (characters)

chunkSize

Optional

Approximate number of characters per chunk. Chunking tries to end on a paragraph or sentence boundary rather than cutting mid-word. Set the value your embedding model expects (around 2000 characters is comfortable for most models).

Type:integer

Minimum:200

Maximum:8000

Default:2000

Chunk overlap (characters)

chunkOverlap

Optional

How many characters each chunk repeats from the previous one. A little overlap helps retrieval quality by keeping sentences that straddle a boundary intact.

Type:integer

Minimum:0

Maximum:1000

Default:200

Include clean Markdown

emitCleanMarkdown

Optional

Also return the extracted article body as Markdown for changed pages. Useful when you rebuild your index from Markdown rather than from raw text.

Type:boolean

Default:false

Which pages to output

outputMode

Optional

By default the dataset contains only changed pages, which keeps the output small and directly usable as a re-indexing queue. Switch to 'all' when you want a record of every page checked.

Type:string

Default:changedOnly

Options:

changedOnlyall

Completion webhook URL

webhookUrl

Optional

Optional. When set, the Actor sends a small POST request here after the run finishes with the number of changed pages and the dataset ID, so your pipeline can trigger re-embedding automatically.

Type:string

Default:

Instagram Scraper

apify/instagram-scraper

Extract Instagram posts, reels, profiles, places, hashtags, carousels, and comments. Get data from Instagram using one or more Instagram URLs or search queries: content, context, metrics, metadata. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Apify

412K

4.7

(601)

Google Maps Scraper

compass/crawler-google-places

Extract data from thousands of Google Maps locations and businesses, including reviews, reviewer details, images, contact info, including full name, email, and job title, opening hours, prices & more. Export data, run via API, schedule and monitor runs, or integrate with other tools.

Compass

627K

4.7

(1,816)

Instagram Profile Scraper

apify/instagram-profile-scraper

Scrape all Instagram profile info. Just add Instagram usernames, IDs or URLs and extract name, join date, number of followers, location, bio, website, related profiles, video&post count, latest posts. Export scraped data, schedule scraper via API, and integrate with other tools or AI workflows.

Apify

232K

4.7

(175)

TikTok Scraper

clockworks/tiktok-scraper

Extract data from TikTok videos, hashtags, and users. Use URLs or search queries to scrape TikTok profiles, hashtags, posts, URLs, shares, followers, hearts, names, video, and music-related data. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Clockworks

297K

4.8

(370)

LinkedIn Profile Scraper + Email ✅ No Cookies

harvestapi/linkedin-profile-scraper

Extract detailed information from LinkedIn profiles in bulk, including complete work experience, education history, skills and more. No cookies or account required.

HarvestAPI

70K

4.6

(92)

Facebook Posts Scraper

apify/facebook-posts-scraper

Extract posts, videos, and engagement metrics from Facebook pages. Get text captions, reactions, video transcripts, images, external links, collaborators, and more from Facebook pages and profiles. Export ad data, schedule runs via API, and integrate with other tools or AI workflows.

Apify

116K

4.6

(234)

Google Search Results Scraper

apify/google-search-scraper

Scrape Google Search Engine Results Pages (SERPs). Select the country or language and extract organic and paid results, AI Mode, AI overviews, ads, queries, People Also Ask, prices, reviews, like a Google SERP API. Export data, run the scraper via API, schedule runs, or integrate with other tools.

Apify

194K

4.6

(218)

Instagram Reel Scraper

apify/instagram-reel-scraper

Scrape or download Instagram reels. Just add Instagram profile or reel URL, and get caption, timestamp, transcript, hashtags, mentions, tagged users, comments, likes, shares, views, duration, and downloaded video. Export scraped data, schedule via API, and integrate with other tools or AI workflows.

Apify

154K

4.4

(103)

Contact Details Scraper

vdrmota/contact-info-scraper

Email extractor and lead scraper to extract and download emails, phone numbers, Facebook, Twitter, LinkedIn, Instagram, Threads, Snapchat, and Telegram profiles from any website. Extract contact information at scale from lists of URLs and download the data as Excel, CSV, JSON, HTML, and XML.