# Google Custom Search API Alternative (Site Search) (`cleandata-labs/site-search`) Actor Google is closing the Custom Search JSON API for existing customers on 1 Jan 2027. This is a drop-in replacement for searching your own sites: same /customsearch/v1 response format, so you only change the base URL. Pay per use (~$1.50 per 1,000 searches). Not affiliated with Google. - **URL**: https://apify.com/cleandata-labs/site-search.md - **Developed by:** [CleanData Labs](https://apify.com/cleandata-labs) (community) - **Categories:** Developer tools, AI, Integrations - **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $1.00 / 1,000 indexed page (new or changed)s This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## Google Custom Search API Alternative (Site Search) **A replacement for Google's Custom Search JSON API (`customsearch/v1`, used with a Programmable Search Engine) when you search your own websites.** Google is closing that API for existing customers on **1 January 2027** ([Google's notice](https://developers.google.com/custom-search/v1/overview)); it is already closed to new customers. **Migration:** change the base URL from `https://www.googleapis.com` to `https://cleandata-labs--site-search.apify.actor`, send your Apify token instead of `key`, and use your index name as `cx`. Requests, response JSON, pagination and errors keep the same format. *Not affiliated with, endorsed by or sponsored by Google. "Google", "Custom Search" and "Programmable Search Engine" are trademarks of Google LLC and are used here only to describe API compatibility.* ### What does it do? This Actor **indexes the websites you own** (docs, blog, help center, product pages) and gives you a **search HTTP endpoint** that answers in the **same JSON format as the Custom Search JSON API**: `items[].title / link / snippet / displayLink`, `searchInformation.totalResults`, `queries.nextPage`, and so on. If your site search page, CMS plugin, internal tool or AI agent calls `customsearch/v1` to search **your own site**, you keep that code. - **Same request:** `GET /customsearch/v1?q=...&cx=...&start=...&num=...` (also `/customsearch/v1/siterestrict`) - **Same response keys and types:** totals are strings, `items` is missing when nothing matched, `nextPage` disappears after result 100, and errors come back as `{"error": {"code": 400, ...}}` - **Your pages only:** built from your own site's pages (sitemap + links), with PDFs included. It never calls Google and never shows Google results. - **Good results for real queries:** BM25 ranking with stemming (`plugins` finds `plugin`), title / heading / URL boosts, typo tolerance (`midleware` finds `middleware`), and pages that match only some of your words when no page matches all of them. Phrase search, `-exclude`, `OR`, `site:` and language filters work too. ~2–10 ms server time; ~0.1–0.5 s round trip. - **Pay per use:** $1.50 per 1,000 searches and $1 per 1,000 indexed pages. No monthly minimum. #### Who is it for? | You use the Custom Search JSON API to... | Is this Actor for you? | |---|---| | search **your own** site(s) from code (site search page, CMS, app, chatbot/RAG) | **Yes** | | show a search box widget (Programmable Search Element) on ≤50 of your domains | Not needed. Google keeps that widget free | | search the **whole web** | **No.** Use a web search API instead (e.g. [Web Search](https://apify.com/apify/web-search) or [RAG Web Browser](https://apify.com/apify/rag-web-browser)) | | search **other people's** websites | **No.** You may only index sites you own or are authorised to index | ### How it works 1. **Index** (a normal Actor run): give it your site URL, a section (`https://docs.example.com/guide/`) or a `sitemap.xml`. The Actor crawls your pages with our [URL to Markdown](https://apify.com/cleandata-labs/url-to-markdown) engine, extracts the main text, title, headings, language and dates, and saves a compact search index to a key-value store in **your** Apify account (`site-search-`). 2. **Search** (Standby HTTP endpoint, always available): call `https://cleandata-labs--site-search.apify.actor/customsearch/v1?q=...&cx=` with your Apify token. No run to start; the server wakes up on the first request. 3. **Refresh**: schedule the index run (e.g. daily or weekly). Re-runs are **incremental**: only new or changed pages are re-indexed at the full price; unchanged pages (same sitemap `lastmod`, HTTP 304 via ETag / Last-Modified, or identical text) are carried over at 1/10 of the price, and deleted pages (404/410) are dropped. The new index replaces the old one atomically, and searches keep working during the refresh. ### Quick start **1. Build the index** (Apify Console → Input, or API): ```json { "mode": "index", "confirmOwnership": true, "startUrls": [{ "url": "https://docs.example.com/" }], "indexName": "docs", "maxPages": 500 } ``` **2. Search:** ```bash curl -H "Authorization: Bearer $APIFY_TOKEN" \ "https://cleandata-labs--site-search.apify.actor/customsearch/v1?cx=docs&q=reset+password&num=5" ``` ```json { "kind": "customsearch#search", "queries": { "request": [{ "searchTerms": "reset password", "count": 5, "startIndex": 1, "totalResults": "12", "cx": "docs", "...": "..." }], "nextPage": [{ "startIndex": 6, "count": 5, "...": "..." }] }, "context": { "title": "docs.example.com" }, "searchInformation": { "searchTime": 0.0021, "formattedSearchTime": "0.00", "totalResults": "12", "formattedTotalResults": "12" }, "items": [ { "kind": "customsearch#result", "title": "Reset your password", "htmlTitle": "Reset your password", "link": "https://docs.example.com/account/reset-password/", "displayLink": "docs.example.com", "snippet": "To reset your password, open Settings → Security and click ...", "htmlSnippet": "To reset your password, open Settings → Security and click ...", "formattedUrl": "https://docs.example.com/account/reset-password/", "htmlFormattedUrl": "https://docs.example.com/account/reset-password/", "pagemap": { "metatags": [{ "og:title": "Reset your password", "og:description": "..." }] } } ] } ``` ### Migration guide from the Custom Search JSON API What changes: **(1) the base URL, (2) authentication (your Apify token instead of `key`), (3) `cx` = your index name.** Your parsing, pagination and error handling stay the same. #### Plain HTTP (JavaScript `fetch`) ```diff - const url = new URL("https://www.googleapis.com/customsearch/v1"); - url.searchParams.set("key", process.env.GOOGLE_API_KEY); - url.searchParams.set("cx", "017576662512468239146:omuauf_lfve"); + const url = new URL("https://cleandata-labs--site-search.apify.actor/customsearch/v1"); + url.searchParams.set("cx", "docs"); // your indexName url.searchParams.set("q", query); url.searchParams.set("start", String(start)); - const res = await fetch(url); + const res = await fetch(url, { headers: { Authorization: `Bearer ${process.env.APIFY_TOKEN}` } }); const data = await res.json(); const results = (data.items ?? []).map(i => ({ title: i.title, url: i.link, text: i.snippet })); const next = data.queries.nextPage?.[0]?.startIndex; // unchanged ``` #### Python, official `google-api-python-client` (tested) ```diff from googleapiclient.discovery import build + from google.oauth2.credentials import Credentials - service = build("customsearch", "v1", developerKey=GOOGLE_API_KEY) + service = build("customsearch", "v1", + credentials=Credentials(token=APIFY_TOKEN), # sent as "Authorization: Bearer ..." + client_options={"api_endpoint": "https://cleandata-labs--site-search.apify.actor"}) - res = service.cse().list(q="timeout", cx="017576662512468239146:omuauf_lfve", num=10).execute() + res = service.cse().list(q="timeout", cx="docs", num=10).execute() for item in res.get("items", []): print(item["title"], item["link"]) ``` `service.cse().siterestrict().list(...)` works too. #### Python `requests` ```diff - r = requests.get("https://www.googleapis.com/customsearch/v1", - params={"key": KEY, "cx": CX, "q": q, "start": start}) + r = requests.get("https://cleandata-labs--site-search.apify.actor/customsearch/v1", + params={"cx": "docs", "q": q, "start": start}, + headers={"Authorization": f"Bearer {APIFY_TOKEN}"}) ``` #### Libraries with a configurable endpoint Some tools already let you point the Google search tool at a compatible endpoint, for example LangChain's `GoogleSearchAPIWrapper` (`GOOGLE_CSE_API_ENDPOINT`, [proposed](https://github.com/langchain-ai/langchain-google/pull/2027)) or apps exposing a `GOOGLE_CSE_URL` setting. Set it to `https://cleandata-labs--site-search.apify.actor` and use your index name as the engine ID. If the library can only send `key=` (not a header), append `token=` to the endpoint URL instead. That works, but tokens in URLs can end up in logs, so prefer a [scoped token](https://docs.apify.com/platform/integrations/api#limited-permissions) that can only run this Actor. > **Never put your Apify token in browser JavaScript.** Like the old `key`, call the endpoint from your server (or a tiny proxy / edge function), not directly from the visitor's browser. ### Supported parameters | Parameter | Supported | Notes | |---|---|---| | `q` | ✅ | words (pages with all words rank first; pages with some of them follow; small typos are corrected), `"exact phrase"` (always required), `-exclude`, `a OR b`, `site:host/path`, `-site:` | | `cx` | ✅ | your **index name** (default `default`) | | `num` | ✅ | 1–10 (default 10) | | `start` | ✅ | 1-based; `start + num` ≤ 101, like the original | | `siteSearch`, `siteSearchFilter` (`i`/`e`) | ✅ | host or host/path prefix | | `exactTerms`, `excludeTerms`, `orTerms`, `hq` | ✅ | | | `lr` | ✅ | `lang_en`, `lang_de`... (from the page's `lang` attribute) | | `sort=date` | ✅ | by the page's published date when present | | `key` | ignored | use your Apify token instead | | `safe`, `gl`, `hl`, `cr`, `filter`, `dateRestrict`, `fields` | accepted, ignored | echoed in `queries.request` where the original does | | `searchType=image`, image params, `fileType`, `rights`, spelling, promotions, facets | ❌ | not in this version | ### Input (index mode) | Field | Default | What it does | |---|---|---| | `confirmOwnership` | `false` | **Required.** Confirms you own the sites or are authorised to index them. Nothing is crawled without it | | `startUrls` | – | Home page, a section, and/or `sitemap.xml` URLs. The crawl stays on these domains, and a section URL keeps it inside that section | | `indexName` | `default` | Your `cx`. Re-indexing with the same name updates the index | | `incremental` | true | Re-index only new or changed pages; unchanged pages cost $0.0001 instead of $0.001. Turn off to force a full rebuild | | `maxPages` | 200 | Max pages to index (up to 50,000) | | `useSitemap` / `maxCrawlDepth` | true / 5 | How pages are found | | `includeUrlGlobs` / `excludeUrlGlobs` | – | e.g. `https://example.com/docs/**`, `**/tag/**` | | `additionalDomains` | – | other domains you own to follow (e.g. `blog.example.com`) | | `keepElementsCssSelector` / `removeElementsCssSelector` | auto | index only `main`, drop `.sidebar`... | | `indexPdfs` | true | index PDFs linked from your site | | `respectRobotsTxt` | true | keep it on (see below) | | `maxConcurrency` | 6 | lower it for small servers | | `customHttpHeaders` | – | e.g. a header your firewall uses to allow the crawler | `mode: "search"` with `q` / `queries` runs test searches as a normal run (results in the dataset and in `OUTPUT`). This **batch search mode** is also the cheapest way to run many queries at once (e.g. an agent or a nightly job): you pay per query plus one short run, with no Standby server time. A run **without** `confirmOwnership` (for example the default input) is a free demo: it crawls nothing, answers 3 sample queries from a built-in 6-page sample index, and tells you how to index your own site. ### Responsible crawling (please read) - **Only index websites you own or are explicitly authorised to index.** You confirm this with `confirmOwnership`. This Actor is a site search tool, not a web scraper or a web search engine. - **robots.txt is respected** by default (user agent `SiteSearchIndexBot`). Pages with `` or an `X-Robots-Tag: noindex` header are never indexed; they are skipped for free. To exclude pages from site search only, use ``. - The crawler identifies itself honestly and uses moderate concurrency. If your firewall blocks it (HTTP 403/429), allowlist the user agent or send a secret header via `customHttpHeaders`. - The index stays in **your** Apify account (named key-value store `site-search-`). Delete the store to delete the index. ### How much does it cost? Pay-per-event pricing: | Event | Price | When | |---|---|---| | Indexed page | **$0.001** ($1 per 1,000 pages) | Each new or changed page or PDF added to the index. Skipped pages (noindex, robots.txt, errors, duplicates) are free | | Unchanged page | **$0.0001** ($0.10 per 1,000 pages) | On re-index runs: a page that did not change since the last run and is carried over | | Search query | **$0.0015** ($1.50 per 1,000 searches) | Each successful search response (also when it has 0 results). Errors (400/404) are free | | Actor start | $0.00005 | Apify's standard start event | **Standby server time is billed by Apify to the caller**, on top of the event prices (this is how Apify bills every pay-per-event Actor in Standby mode). The server runs with 256 MB (0.25 compute units per hour) and stops after 60 seconds without requests. One wake-up with a few searches and the idle tail ≈ 0.005 CU ≈ $0.001–0.0015. If searches arrive non-stop all day, the server never sleeps: ≈ 6 CU/day ≈ $1.25–1.80/day depending on your Apify plan. For bulk or nightly searches, use batch search mode (`mode: "search"`) instead: no idle server time. | Example | Our price | |---|---| | 300-page docs site, re-indexed weekly (~5% of pages change) | first run $0.30, then ~$0.05 per weekly run | | 50,000-page site, re-indexed weekly (~2% change) | first run $50, then ~$5.90 per weekly run (~$26/month) | | 10,000 searches/month | $15 + Standby server time (Google CSE was $5 per 1,000 after 100/day free; Vertex AI site search is $4 per 1,000) | ### Limits - Keyword (BM25) ranking with stemming, field boosts, light typo correction and partial matching. No semantic search and no synonyms yet. Long changelog / release-notes pages sometimes rank above the topic page; exclude them with `excludeUrlGlobs` if that bothers you. - JavaScript-only pages (content rendered in the browser, empty HTML) are not indexed. Use server-rendered pages or your sitemap. - Index size: tested up to ~160 pages per index (index ≈ 4 KB per page, compressed); designed for up to ~50,000 pages per index. - Incremental re-index needs the previous index from v0.2 or later; the first run after upgrading is a full re-index. - Speed (measured on Apify, 1 GB): ~3 pages/s on light docs sites, ~1 page/s on heavy pages (first index). Re-index runs where pages did not change are ~10x faster (100 unchanged pages in 12 s). For more than ~5,000 pages, raise the run timeout (default 1 hour) and memory (more memory = more CPU). - Very large sites: the index is loaded into the Standby server's memory (≈ 4 KB per page compressed). Above ~20,000 pages, give the Standby server more memory (via a task) and expect a slower first request. - First request after an idle period takes ~5 s (server wake-up + index load); then ~2–10 ms per search inside the server. - New pages appear after the next index run (schedule it). ### FAQ **Does it return Google results?** No. It searches only the pages it indexed from your sites. It never calls Google. **Do I need a Google API key?** No. Only an Apify account and token. **Can I keep my old `cx`?** Use your index name as `cx`. Google's `cx` values contain `:`, which index names can't, so you change the value once (usually an environment variable). **Is my token safe?** Send it in the `Authorization` header from your backend. Apify strips it before the request reaches the Actor. **Can I search several sites?** Yes: put several start URLs (domains you own) in one index, then filter with `siteSearch` or `site:`. # Actor input Schema ## `mode` (type: `string`): index = crawl your site and build or refresh the search index. search = run test queries against an existing index. For live search from your app, use the Standby HTTP endpoint (see README); you don't need to start runs for that. ## `confirmOwnership` (type: `boolean`): Required for indexing. This Actor is for searching YOUR OWN sites (or sites you have written permission to index), like a site search box. It is not a web search engine and does not return Google results. Without this box ticked, the run is a free demo on built-in sample pages (nothing is crawled). ## `startUrls` (type: `array`): Your site's home page or a section (e.g. https://docs.example.com/guide/), and/or a sitemap URL (https://example.com/sitemap.xml). Crawling stays on these domains, and a section URL keeps the crawl inside that section. ## `indexName` (type: `string`): Name of the index. Use it as cx in search requests. Letters, digits and dashes. Re-running index mode with the same name updates the index; searches keep working during the refresh. The index is stored in your account in a key-value store named site-search-. ## `indexTitle` (type: `string`): Optional. Returned as context.title in search responses. ## `maxPages` (type: `integer`): Stop after this many indexed pages. You pay per indexed page; skipped pages (noindex, robots.txt, errors, duplicates) are free. ## `incremental` (type: `boolean`): When an index with this name already exists, only new or changed pages are re-indexed (detected with the sitemap's lastmod, ETag / Last-Modified and a content hash). Unchanged pages are carried over at the lower 'unchanged page' price, and pages that now return 404/410 are removed. Turn off to force a full rebuild. ## `useSitemap` (type: `boolean`): Also read the sitemaps listed in robots.txt (or /sitemap.xml) to find pages. ## `maxCrawlDepth` (type: `integer`): How many clicks away from the start URLs to follow links. 0 = only start URLs and sitemap pages. ## `includeUrlGlobs` (type: `array`): Optional glob patterns, e.g. https://example.com/docs/\*\*. When set, they replace the default 'stay inside the start URL's section' rule. ## `excludeUrlGlobs` (type: `array`): Optional glob patterns, e.g. **/tag/** or \*\*?page=\*. ## `additionalDomains` (type: `array`): Optional extra hostnames to follow links into (e.g. blog.example.com). Only add domains you own or may index. ## `keepElementsCssSelector` (type: `string`): Optional, e.g. main, article or .docs-content. Default: automatic main-content detection. ## `removeElementsCssSelector` (type: `string`): Optional, e.g. .cookie-banner, .sidebar ## `indexPdfs` (type: `boolean`): PDF text is indexed like a page (first 100 pages of each PDF). ## `respectRobotsTxt` (type: `boolean`): Keep this on. Turn it off only for your own site when robots.txt blocks crawlers you want to allow. Pages with a noindex meta tag are always skipped. ## `maxConcurrency` (type: `integer`): How many pages to fetch at the same time. Lower it if your server is small. ## `requestTimeoutSecs` (type: `integer`): Per-page timeout. ## `customHttpHeaders` (type: `object`): Optional headers sent with every request to your site, e.g. a header your firewall uses to allow this crawler. ## `q` (type: `string`): Search mode only. Same syntax as before: words, "exact phrase", -exclude, OR, site:docs.example.com/api ## `queries` (type: `array`): Search mode only. Each query is charged as one search. ## `num` (type: `integer`): Search mode only. ## `start` (type: `integer`): Search mode only. 1-based, like the original API. ## `siteSearch` (type: `string`): Search mode only, e.g. docs.example.com/api ## `lr` (type: `string`): Search mode only, e.g. lang_en (uses the page's lang attribute). ## Actor input object example ```json { "mode": "index", "confirmOwnership": false, "startUrls": [ { "url": "https://example.com/" } ], "indexName": "default", "maxPages": 200, "incremental": true, "useSitemap": true, "maxCrawlDepth": 5, "indexPdfs": true, "respectRobotsTxt": true, "maxConcurrency": 6, "requestTimeoutSecs": 30, "num": 10, "start": 1 } ``` # Actor output Schema ## `pages` (type: `string`): No description ## `summary` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ { "url": "https://example.com/" } ] }; // Run the Actor and wait for it to finish const run = await client.actor("cleandata-labs/site-search").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": [{ "url": "https://example.com/" }] } # Run the Actor and wait for it to finish run = client.actor("cleandata-labs/site-search").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ { "url": "https://example.com/" } ] }' | apify call cleandata-labs/site-search --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,cleandata-labs/site-search" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/nH87z67ZQUZVYywTp/builds/1acYp1asrqUxh1pNN/openapi.json