Google Custom Search API Alternative (Site Search)

A replacement for Google's Custom Search JSON API (customsearch/v1, used with a Programmable Search Engine) when you search your own websites. Google is closing that API for existing customers on 1 January 2027 (Google's notice); it is already closed to new customers.

Migration: change the base URL from https://www.googleapis.com to https://cleandata-labs--site-search.apify.actor, send your Apify token instead of key, and use your index name as cx. Requests, response JSON, pagination and errors keep the same format.

Not affiliated with, endorsed by or sponsored by Google. "Google", "Custom Search" and "Programmable Search Engine" are trademarks of Google LLC and are used here only to describe API compatibility.

What does it do?

This Actor indexes the websites you own (docs, blog, help center, product pages) and gives you a search HTTP endpoint that answers in the same JSON format as the Custom Search JSON API: items[].title / link / snippet / displayLink, searchInformation.totalResults, queries.nextPage, and so on. If your site search page, CMS plugin, internal tool or AI agent calls customsearch/v1 to search your own site, you keep that code.

• Same request: GET /customsearch/v1?q=...&cx=...&start=...&num=... (also /customsearch/v1/siterestrict)
• Same response keys and types: totals are strings, items is missing when nothing matched, nextPage disappears after result 100, and errors come back as {"error": {"code": 400, ...}}
• Your pages only: built from your own site's pages (sitemap + links), with PDFs included. It never calls Google and never shows Google results.
• Good results for real queries: BM25 ranking with stemming (plugins finds plugin), title / heading / URL boosts, typo tolerance (midleware finds middleware), and pages that match only some of your words when no page matches all of them. Phrase search, -exclude, OR, site: and language filters work too. ~2-10 ms server time; ~0.1-0.5 s round trip.
• Freshness you can see: every result says when the page was last crawled (lastCrawled) and, when known, last modified (lastModified); every response says when the index was last updated and how many pages it has. See Freshness fields.
• Relevance tests: give a list of test queries with the pages you expect, and get the rank of each expected page plus hit@1, hit@3, hit@10 and MRR. Run it before and after migrating, and after each re-index. See Relevance test mode.
• Pay per use: $1.50 per 1,000 searches and $1 per 1,000 indexed pages. No monthly minimum.

Who is it for?

You use the Custom Search JSON API to...	Is this Actor for you?
search your own site(s) from code (site search page, CMS, app, chatbot/RAG)	Yes
show a search box widget (Programmable Search Element) on <=50 of your domains	Not needed. Google keeps that widget free
search the whole web	No. Use a web search API instead (e.g. Web Search or RAG Web Browser)
search other people's websites	No. You may only index sites you own or are authorised to index

How it works

1. Index (a normal Actor run): give it your site URL, a section (https://docs.example.com/guide/) or a sitemap.xml. The Actor crawls your pages with our URL to Markdown engine, extracts the main text, title, headings, language and dates, and saves a compact search index to a key-value store in your Apify account (site-search-<indexName>).
2. Search (Standby HTTP endpoint, always available): call https://cleandata-labs--site-search.apify.actor/customsearch/v1?q=...&cx=<indexName> with your Apify token. No run to start; the server wakes up on the first request.
3. Refresh: schedule the index run (e.g. daily or weekly). Re-runs are incremental: only new or changed pages are re-indexed at the full price; unchanged pages (same sitemap lastmod, HTTP 304 via ETag / Last-Modified, or identical text) are carried over at 1/10 of the price, and deleted pages (404/410) are dropped. The new index replaces the old one atomically, and searches keep working during the refresh.

Quick start

1. Build the index (Apify Console -> Input, or API):

{
  "mode": "index",
  "confirmOwnership": true,
  "startUrls": [{ "url": "https://docs.example.com/" }],
  "indexName": "docs",
  "maxPages": 500
}

2. Search:

curl -H "Authorization: Bearer $APIFY_TOKEN" \
  "https://cleandata-labs--site-search.apify.actor/customsearch/v1?cx=docs&q=reset+password&num=5"

{
  "kind": "customsearch#search",
  "queries": {
    "request": [{ "searchTerms": "reset password", "count": 5, "startIndex": 1, "totalResults": "12", "cx": "docs", "...": "..." }],
    "nextPage": [{ "startIndex": 6, "count": 5, "...": "..." }]
  },
  "context": { "title": "docs.example.com" },
  "searchInformation": { "searchTime": 0.0021, "formattedSearchTime": "0.00", "totalResults": "12", "formattedTotalResults": "12" },
  "items": [
    {
      "kind": "customsearch#result",
      "title": "Reset your password",
      "htmlTitle": "<b>Reset</b> your <b>password</b>",
      "link": "https://docs.example.com/account/reset-password/",
      "displayLink": "docs.example.com",
      "snippet": "To reset your password, open Settings -> Security and click ...",
      "htmlSnippet": "To <b>reset</b> your <b>password</b>, open Settings -> Security and click ...",
      "formattedUrl": "https://docs.example.com/account/reset-password/",
      "htmlFormattedUrl": "https://docs.example.com/account/reset-password/",
      "pagemap": {
        "metatags": [{ "og:title": "Reset your password", "og:description": "..." }],
        "sitesearch": [{ "lastCrawled": "2026-10-31T05:23:07+00:00", "lastModified": "2026-09-11", "lastModifiedSource": "sitemap" }]
      }
    }
  ],
  "siteSearchIndex": { "name": "docs", "title": "docs.example.com", "pages": 480, "updatedAt": "2026-10-31T06:57:37+00:00", "version": "20261031065737", "oldestPageCrawledAt": "2026-10-24T06:50:12+00:00", "newestPageCrawledAt": "2026-10-31T06:57:30+00:00" }
}

Migration guide from the Custom Search JSON API

What changes: (1) the base URL, (2) authentication (your Apify token instead of key), (3) cx = your index name. Your parsing, pagination and error handling stay the same.

Plain HTTP (JavaScript fetch)

- const url = new URL("https://www.googleapis.com/customsearch/v1");
- url.searchParams.set("key", process.env.GOOGLE_API_KEY);
- url.searchParams.set("cx", "017576662512468239146:omuauf_lfve");
+ const url = new URL("https://cleandata-labs--site-search.apify.actor/customsearch/v1");
+ url.searchParams.set("cx", "docs");                       // your indexName
  url.searchParams.set("q", query);
  url.searchParams.set("start", String(start));
- const res = await fetch(url);
+ const res = await fetch(url, { headers: { Authorization: `Bearer ${process.env.APIFY_TOKEN}` } });
  const data = await res.json();
  const results = (data.items ?? []).map(i => ({ title: i.title, url: i.link, text: i.snippet }));
  const next = data.queries.nextPage?.[0]?.startIndex;      // unchanged

Python, official google-api-python-client (tested)

from googleapiclient.discovery import build
+ from google.oauth2.credentials import Credentials

- service = build("customsearch", "v1", developerKey=GOOGLE_API_KEY)
+ service = build("customsearch", "v1",
+                 credentials=Credentials(token=APIFY_TOKEN),   # sent as "Authorization: Bearer ..."
+                 client_options={"api_endpoint": "https://cleandata-labs--site-search.apify.actor"})
- res = service.cse().list(q="timeout", cx="017576662512468239146:omuauf_lfve", num=10).execute()
+ res = service.cse().list(q="timeout", cx="docs", num=10).execute()
  for item in res.get("items", []):
      print(item["title"], item["link"])

service.cse().siterestrict().list(...) works too.

Python requests

- r = requests.get("https://www.googleapis.com/customsearch/v1",
-                  params={"key": KEY, "cx": CX, "q": q, "start": start})
+ r = requests.get("https://cleandata-labs--site-search.apify.actor/customsearch/v1",
+                  params={"cx": "docs", "q": q, "start": start},
+                  headers={"Authorization": f"Bearer {APIFY_TOKEN}"})

Libraries with a configurable endpoint

Some tools already let you point the Google search tool at a compatible endpoint, for example LangChain's GoogleSearchAPIWrapper (GOOGLE_CSE_API_ENDPOINT, proposed) or apps exposing a GOOGLE_CSE_URL setting. Set it to https://cleandata-labs--site-search.apify.actor and use your index name as the engine ID. If the library can only send key= (not a header), append token=<APIFY_TOKEN> to the endpoint URL instead. That works, but tokens in URLs can end up in logs, so prefer a scoped token that can only run this Actor.

Never put your Apify token in browser JavaScript. Like the old key, call the endpoint from your server (or a tiny proxy / edge function), not directly from the visitor's browser.

Supported parameters

Parameter	Supported	Notes
q	Yes	words (pages with all words rank first; pages with some of them follow; small typos are corrected), "exact phrase" (always required), -exclude, a OR b, site:host/path, -site:
cx	Yes	your index name (default default)
num	Yes	1-10 (default 10)
start	Yes	1-based; start + num <= 101, like the original
siteSearch, siteSearchFilter (i/e)	Yes	host or host/path prefix
exactTerms, excludeTerms, orTerms, hq	Yes
lr	Yes	lang_en, lang_de... (from the page's lang attribute)
sort=date	Yes	by the page's published date when present
key	ignored	use your Apify token instead
safe, gl, hl, cr, filter, dateRestrict, fields	accepted, ignored	echoed in queries.request where the original does
searchType=image, image params, fileType, rights, spelling, promotions, facets	No	not in this version

Freshness fields

Two kinds of extra data are added to every search response (Standby endpoint and batch mode: "search"). All Google-format keys stay exactly as before; these are additions only.

Per result: items[].pagemap.sitesearch[0]

Field	Meaning
lastCrawled	When our crawler last downloaded this page, or your server confirmed it unchanged (HTTP 304). ISO 8601, UTC. Pages skipped on a re-index because their sitemap lastmod did not change keep their earlier date
lastModified	When the page last changed, if known: the sitemap's <lastmod>, otherwise the page's HTTP Last-Modified header. Missing when neither exists
lastModifiedSource	sitemap or http-header

Per response: top-level siteSearchIndex

Field	Meaning
name, title	Index name (your cx) and title
pages	Number of pages in the index
updatedAt	When the last index run finished and published this index version
version	Index version id (changes on every index run)
oldestPageCrawledAt, newestPageCrawledAt	Oldest and newest lastCrawled among all pages

Why these places: in the Custom Search JSON API schema, pagemap is a free-form object, so an extra sitesearch entry is valid there, and JSON clients (including Google's official client libraries) ignore unknown top-level keys such as siteSearchIndex. Code that only reads the Google fields keeps working unchanged. In batch search mode, the dataset rows also get lastCrawled and lastModified columns.

Use them to monitor your re-index schedule, e.g. alert when updatedAt is older than your schedule interval. Indexes built before v0.2 have no lastModified data; re-index once to get it.

Relevance test mode

Check that real queries still find the right pages, for example before and after moving from the Custom Search JSON API, and after each re-index. Run the Actor with:

{
  "mode": "relevance",
  "indexName": "docs",
  "relevanceTests": [
    { "query": "reset password", "expected": ["https://docs.example.com/account/reset-password/"] },
    { "query": "api rate limits", "expected": ["https://docs.example.com/api/limits/", "https://docs.example.com/faq/"] },
    { "query": "webhooks", "expected": ["/guides/webhooks/"] }
  ]
}

• expected is one or more URLs or URL prefixes. A result matches when its URL starts with one of them (http/https, www. and #fragments are ignored; a value that starts with / matches that path on any host). The test passes at the rank of the first matching result.
• Each test query is searched once (top 10 results, same ranking as the live endpoint). siteSearch and lr from the input apply to all tests. Up to 1,000 tests per run.
• Dataset: one row per query with rank (1-10, or empty when not found), found, hitAt1, hitAt3, hitAt10, matchedUrl, expected and the top 3 result URLs (view "Relevance test results").
• Summary: the RELEVANCE_REPORT record in the run's key-value store: hitAt1, hitAt3, hitAt10 (share of queries whose expected page is in the top 1 / 3 / 10), mrr (mean reciprocal rank over the top 10; 0 for not found), failedQueries (expected page not in the top 10, with what came up instead), notInTop3Queries, invalid tests, and the index version and update time the test ran against, so you can compare runs.
• Cost: each test query is charged as one normal search ($0.0015). Invalid tests are skipped for free. An empty test list or an unknown index stops the run with a clear message and nothing charged.

Tip: keep the same test list in a saved task and run it after every scheduled re-index; a drop in hit@3 tells you a page moved or lost its content.

Input (index mode)

Field	Default	What it does
confirmOwnership	false	Required. Confirms you own the sites or are authorised to index them. Nothing is crawled without it
startUrls	-	Home page, a section, and/or sitemap.xml URLs. The crawl stays on these domains, and a section URL keeps it inside that section
indexName	default	Your cx. Re-indexing with the same name updates the index
incremental	true	Re-index only new or changed pages; unchanged pages cost $0.0001 instead of $0.001. Turn off to force a full rebuild
maxPages	200	Max pages to index (up to 50,000)
useSitemap / maxCrawlDepth	true / 5	How pages are found
includeUrlGlobs / excludeUrlGlobs	-	e.g. https://example.com/docs/**, **/tag/**
additionalDomains	-	other domains you own to follow (e.g. blog.example.com)
keepElementsCssSelector / removeElementsCssSelector	auto	index only main, drop .sidebar...
indexPdfs	true	index PDFs linked from your site
respectRobotsTxt	true	keep it on (see below)
maxConcurrency	6	lower it for small servers
customHttpHeaders	-	e.g. a header your firewall uses to allow the crawler

mode: "search" with q / queries runs test searches as a normal run (results in the dataset and in OUTPUT). mode: "relevance" runs known-answer tests (see Relevance test mode). This batch search mode is also the cheapest way to run many queries at once (e.g. an agent or a nightly job): you pay per query plus one short run, with no Standby server time.

A run without confirmOwnership (for example the default input) is a free demo: it crawls nothing, answers 3 sample queries from a built-in 6-page sample index, and tells you how to index your own site.

Responsible crawling (please read)

• Only index websites you own or are explicitly authorised to index. You confirm this with confirmOwnership. This Actor is a site search tool, not a web scraper or a web search engine.
• robots.txt is respected by default (user agent SiteSearchIndexBot). Pages with <meta name="robots" content="noindex"> or an X-Robots-Tag: noindex header are never indexed; they are skipped for free. To exclude pages from site search only, use <meta name="SiteSearchIndexBot" content="noindex">.
• The crawler identifies itself honestly and uses moderate concurrency. If your firewall blocks it (HTTP 403/429), allowlist the user agent or send a secret header via customHttpHeaders.
• The index stays in your Apify account (named key-value store site-search-<indexName>). Delete the store to delete the index.

How much does it cost?

Pay-per-event pricing:

Event	Price	When
Indexed page	$0.001 ($1 per 1,000 pages)	Each new or changed page or PDF added to the index. Skipped pages (noindex, robots.txt, errors, duplicates) are free
Unchanged page	$0.0001 ($0.10 per 1,000 pages)	On re-index runs: a page that did not change since the last run and is carried over
Search query	$0.0015 ($1.50 per 1,000 searches)	Each successful search response (also when it has 0 results). Errors (400/404) are free
Actor start	$0.00005	Apify's standard start event

Standby server time is billed by Apify to the caller, on top of the event prices (this is how Apify bills every pay-per-event Actor in Standby mode). The server runs with 256 MB (0.25 compute units per hour) and stops after 60 seconds without requests. One wake-up with a few searches and the idle tail ~ 0.005 CU ~ $0.001-0.0015. If searches arrive non-stop all day, the server never sleeps: ~ 6 CU/day ~ $1.25-1.80/day depending on your Apify plan. For bulk or nightly searches, use batch search mode (mode: "search") instead: no idle server time.

Example	Our price
300-page docs site, re-indexed weekly (~5% of pages change)	first run $0.30, then ~$0.05 per weekly run
50,000-page site, re-indexed weekly (~2% change)	first run $50, then $5.90 per weekly run ($26/month)
10,000 searches/month	$15 + Standby server time (Google CSE was $5 per 1,000 after 100/day free; Vertex AI site search is $4 per 1,000)

Limits

• Keyword (BM25) ranking with stemming, field boosts, light typo correction and partial matching. No semantic search and no synonyms yet. Long changelog / release-notes pages sometimes rank above the topic page; exclude them with excludeUrlGlobs if that bothers you.
• JavaScript-only pages (content rendered in the browser, empty HTML) are not indexed. Use server-rendered pages or your sitemap.
• Index size: tested up to ~160 pages per index (index ~ 4 KB per page, compressed); designed for up to ~50,000 pages per index.
• Incremental re-index needs the previous index from v0.2 or later; the first run after upgrading is a full re-index.
• Speed (measured on Apify, 1 GB): ~3 pages/s on light docs sites, ~1 page/s on heavy pages (first index). Re-index runs where pages did not change are ~10x faster (100 unchanged pages in 12 s). For more than ~5,000 pages, raise the run timeout (default 1 hour) and memory (more memory = more CPU).
• Very large sites: the index is loaded into the Standby server's memory (~ 4 KB per page compressed). Above ~20,000 pages, give the Standby server more memory (via a task) and expect a slower first request.
• First request after an idle period takes ~5 s (server wake-up + index load); then ~2-10 ms per search inside the server.
• New pages appear after the next index run (schedule it).

FAQ

Does it return Google results? No. It searches only the pages it indexed from your sites. It never calls Google. Do I need a Google API key? No. Only an Apify account and token. Can I keep my old cx? Use your index name as cx. Google's cx values contain :, which index names can't, so you change the value once (usually an environment variable). Is my token safe? Send it in the Authorization header from your backend. Apify strips it before the request reaches the Actor. Can I search several sites? Yes: put several start URLs (domains you own) in one index, then filter with siteSearch or site:.