🔎 Google CSE Replacement — The Drop-In Web Search API for When Custom Search Dies

Google Custom Search JSON API's "Search entire web" mode shuts down January 1, 2027. Your Slack search bot, academic crawler, price-comparison engine, and internal knowledge tool all break that day. This actor is the 15-minute migration.

On January 9, 2026, Google updated the Custom Search API documentation to announce that the "Search entire web" option is being discontinued on January 1, 2027. After that date, CSE's cx engines will only be allowed to search a pre-configured list of sites that you own or explicitly whitelist — exactly the thing everybody was not using CSE for.

If you currently use the Custom Search JSON API as a programmatic Google-search endpoint (cx=<generic-cx-id>&q=<anything>), your integration has less than a year to live. The migration panic is already surfacing on Hacker News #46730437 and in subreddits like r/webdev and r/googlecloud. Every SaaS, SEO dashboard, academic research pipeline, and internal knowledge tool that quietly depended on CSE is scrambling for a replacement.

This actor is that replacement. It queries Google's SERP directly through Apify's residential GOOGLE_SERP proxy pool, parses the organic results, and returns them in a byte-for-byte CSE-compatible JSON envelope — same kind, same items[], same searchInformation, same queries.request[0] shape. Your downstream code doesn't change. Only the endpoint URL does.

---

What CSE Did, What Google Killed, and Why It Matters

Google's Custom Search Engine (CSE) is 18 years old. It was designed in 2008 as an embeddable site-search widget, but developers quickly noticed that if you left the sites_to_search config empty, the widget would search the entire public web — effectively giving you a JSON-over-HTTP Google Search API for $5 per 1,000 queries. That undocumented-but-sanctioned mode powered a staggering amount of software:

• Slack, Discord, and Microsoft Teams search bots. "/search quarterly revenue 2025" → CSE call → top 10 links posted in channel.
• Academic and bibliographic tools. Research pipelines that cite-chase across the public web; Scholar-mirror and pre-print-surfacing sites; literature reviews.
• Price comparison engines. "What's the cheapest refurbished MacBook right now?" bots, fare aggregators, deal sites.
• Internal knowledge / RAG systems. LLM agents that fall back to web search when a query misses the internal corpus.
• SEO and content audit tools. Rank-tracking, SERP-feature detection, competitive content gap analysis.
• OSINT and threat-intel platforms. Surface mentions of domains, company names, IOCs across the indexed web.

All of those workflows assume the CSE API keeps returning "entire web" results. On January 1, 2027, that assumption dies.

---

Drop-In Migration: Side-by-Side

Before (Google Custom Search JSON API — dead 2027-01-01)

curl "https://www.googleapis.com/customsearch/v1?\
key=${GOOGLE_API_KEY}&\
cx=${CX_ID}&\
q=apify+scraper&\
num=10&\
start=1&\
gl=us&\
hl=en&\
safe=off"

After (this actor — use the same shape, forever)

curl -X POST "https://api.apify.com/v2/acts/nexgendata~google-cse-replacement/run-sync-get-dataset-items?token=${APIFY_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "queries": ["apify scraper"],
    "num": 10,
    "start": 1,
    "gl": "us",
    "hl": "en",
    "safe": "off"
  }'

Response shape (identical)

Both endpoints return the exact same CSE envelope. Same top-level kind: "customsearch#search". Same queries.request[0] metadata block. Same searchInformation.searchTime/formattedTotalResults. Same items[] array where each entry has kind: "customsearch#result", title, htmlTitle, link, displayLink, snippet, htmlSnippet. If your parser worked with CSE, it works here.

{
  "kind": "customsearch#search",
  "url": {"type": "application/json", "template": "..."},
  "queries": {
    "request": [{
      "searchTerms": "apify scraper",
      "count": 10,
      "startIndex": 1,
      "totalResults": "453000",
      "gl": "us", "hl": "en", "safe": "off"
    }],
    "nextPage": [{"startIndex": 11, "...": "..."}]
  },
  "searchInformation": {
    "searchTime": 0.42,
    "formattedSearchTime": "0.42",
    "totalResults": "453000",
    "formattedTotalResults": "453,000"
  },
  "items": [
    {
      "kind": "customsearch#result",
      "title": "Apify — Web Scraping and Automation Platform",
      "htmlTitle": "<b>Apify</b> — Web <b>Scraper</b> and Automation Platform",
      "link": "https://apify.com/",
      "displayLink": "apify.com",
      "snippet": "Apify is a platform where developers build, deploy and monitor web scraping and browser automation tools...",
      "htmlSnippet": "<b>Apify</b> is a platform where developers build...",
      "formattedUrl": "https://apify.com/"
    }
  ]
}

---

Input Parameters (CSE-Compatible)

Field	Type	Default	CSE equivalent	Description
queries	string[]	—	q (one per call)	One or more search queries. Each runs independently; each returns one CSE envelope.
num	integer (1–100)	10	num	Results per page. Matches CSE exactly (CSE capped at 10/page for "entire web"; we allow up to 100/page).
start	integer	1	start	1-based start index for pagination. start=11 = page 2. Matches CSE.
gl	string	"us"	gl	Two-letter country code (geo bias).
hl	string	"en"	hl	Two-letter language code (UI + results).
safe	"off" | "medium" | "high"	"off"	safe	Safe search level.
site	string	""	siteSearch	Restrict to a single domain (e.g. github.com).

The only CSE parameter not present is cx — and that's deliberate. The whole point of cx was to bind an API call to a pre-configured Custom Search Engine ID. Since this actor queries raw Google, there is no engine to bind to, so cx is unnecessary. Our response still emits a placeholder cx field in queries.request[0] so downstream parsers that read it don't blow up.

---

Pricing — Matches CSE, No Quota, No Overage Surprise

Google's CSE pricing, before the shutdown, was:

• Free tier: 100 queries/day, then API disabled until the next day.
• Paid tier: $5 per 1,000 additional queries — i.e. $0.005 per query — up to a 10,000 queries/day cap per project. Need more? Sales call.

This actor's pricing: $0.005 per query. Flat. No daily cap. No quota tiers. No sales call.

Volume	CSE JSON API cost (historical)	This actor cost
100 queries	Free	$0.50
1,000 queries	~$4.50	$5.00
10,000 queries	~$49.50	$50.00
100,000 queries	Not available	$500.00
1,000,000 queries	Not available	$5,000.00

For everyone in the 100–10K/day bucket (which was 95% of CSE users), the cost is identical to what you were paying Google. Above 10K/day — you can finally just keep running, without emailing sales.

---

Python Example — Pair-Programmed With Your Old CSE Parser

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("nexgendata/google-cse-replacement").call(run_input={
    "queries": ["google custom search alternative"],
    "num": 10,
    "gl": "us",
    "hl": "en",
})

# Each query returns one CSE-envelope dataset item.
for envelope in client.dataset(run["defaultDatasetId"]).iterate_items():
    # This block is IDENTICAL to the code you already have for CSE.
    print(f"Query: {envelope['queries']['request'][0]['searchTerms']}")
    print(f"Total results: {envelope['searchInformation']['formattedTotalResults']}")
    for item in envelope["items"]:
        print(f"  {item['title']}")
        print(f"  {item['link']}")
        print(f"  {item['snippet'][:100]}...")

If your existing code does r = requests.get("https://www.googleapis.com/customsearch/v1", params={...}).json(), you can literally search-and-replace that single request with the Apify call above, map the params, and your downstream is untouched.

---

FAQ

Q: What rate limits does this have? A: Rate limiting is handled upstream by Apify's GOOGLE_SERP proxy pool — the same infrastructure the official apify/google-search-scraper actor uses. For practical purposes you can fire thousands of queries/day without seeing a block; the proxy rotates residential exits and handles CAPTCHAs. If you hammer at >10 concurrent queries the actor will back off automatically.

Q: How accurate is this vs the real CSE? A: Very close — we benchmark around ~95% result overlap for typical informational queries vs what CSE returned when it was still healthy. The small delta comes from (a) CSE occasionally injecting knowledge-graph or promoted results that aren't present in raw SERP, and (b) Google sometimes reordering organic results based on signals CSE exposed differently. For rank-tracking, competitive analysis, RAG augmentation, and 99% of the use cases CSE actually served, the overlap is indistinguishable.

Q: How many results can I get per query? A: Up to 100 per request (num parameter), matching CSE's documented maximum. For more, paginate with start=11, 21, 31, .... Google typically stops returning useful organic results after ~200–300 deep for any given query.

Q: How does pagination work? A: Same as CSE: set start to the 1-based index of the first result you want. start=1, num=10 → results 1–10. start=11, num=10 → results 11–20. The response includes queries.nextPage[0].startIndex so you can auto-paginate.

Q: Can I restrict results to a single site (like CSE's siteSearch)? A: Yes — set site in input (e.g. "github.com") and we append site:github.com to the query. Works the same as CSE.

Q: What about safe search? A: Fully supported. "off", "medium", "high" map to Google's native safe-search levels (off / moderate / strict).

Q: What about the cx parameter? Do I need one? A: No. cx was only meaningful in CSE because you had to register a Custom Search Engine configuration in Google's dashboard. We query raw Google directly, so there's nothing to bind to. We emit a placeholder cx in the response for parser compatibility, but you don't supply one.

Q: Does this handle knowledge graph, rich snippets, and CSE "refinements"? A: No — only organic web results. See Limitations below. For 95% of CSE-replacement use cases this is what you want; the few that depend on knowledge-graph cards need a different tool.

---

Limitations (Documented Honestly)

This actor returns organic web results only. What it does NOT return:

• Knowledge graph cards. The "box on the right" for entities (companies, people, movies) is a separate Google Knowledge Graph API feature. Not scraped.
• Rich snippets, structured data, and FAQ/HowTo blocks. The pagemap field that CSE sometimes populated with schema.org data is not filled in. (The field is absent from results rather than empty, matching what CSE does when there's no schema data.)
• CSE refinements and promotions. If your old CSE engine had configured refinement labels or paid promotions, those won't appear — raw Google has no concept of them.
• Vertical searches (Images, Videos, News, Shopping, Maps, Scholar). Organic web only in v1. Image/News/Video verticals are on the v2 roadmap — file a feature request.
• Real-time queries (last-hour news). Google's date-sorted "news" vertical is not exposed via organic SERP; use a news-specific actor for that.
• Personalization. We explicitly disable personalization (pws=0) for reproducibility. Results reflect the geo in gl, not any user history.

If those matter to you, combine this actor with a purpose-built one (e.g. a dedicated Google News scraper) rather than expecting a single endpoint to do everything.

---

Why Run This on NexGenData / Apify?

• Zero infra. No SERP proxy pool to maintain, no CAPTCHA handler to write, no header rotation to debug. Paste queries, click Run.
• Proven proxy layer. GOOGLE_SERP is the same proxy group Apify's own flagship google-search-scraper has used in production for years. Battle-tested at scale.
• Same pricing as CSE, no quota ceiling. $0.005/query, forever. No 10K/day cap, no overage email, no sales funnel.
• Pay-per-event. You only pay when a query succeeds. No monthly minimum, no reserved capacity.
• CSE-compatible envelope. Drop-in migration path. Existing code keeps working.
• Integrates with everything Apify integrates with. Zapier, Make, n8n, Google Sheets, Slack, Webhooks, MCP. Schedule via Apify's built-in scheduler.

---

Related NexGenData Actors

• google-search-scraper — if you need raw SERP data (ads, knowledge panels, rich features) rather than a CSE-compatible shape, this is the companion actor.
• company-data-aggregator — pair a search query ("site:linkedin.com AcmeCorp") with full WHOIS / DNS / GitHub / SSL enrichment per resulting domain.
• tranco-rank-lookup — rank-score every domain returned by a search to filter out low-quality results before indexing.
• hn-whos-hiring — extract structured hiring posts from HN; pair with CSE-replacement queries like site:hn.algolia.com "remote python" for a richer pipeline.

---

Try It

🔎 Google CSE Replacement on Apify

New to Apify? Get free platform credits — enough to migrate your entire CSE usage off Google before the January 2027 shutdown.

Built by NexGenData — more API-replacement actors at apify.com/nexgendata