Google Forums Scraper

Search Google's dedicated Forums vertical for public discussions from Reddit, Quora, Stack Overflow, Stack Exchange, GitHub Discussions, Hacker News, Discourse communities, and niche forums across the web.

The Actor supports one query or fairly allocated batches, pagination, platform and date filters, localization, SafeSearch, and source-backed thread metadata. The verified cloud path uses Apify's GOOGLE_SERP group as HTTP transport and normally completes in seconds. When that transport does not expose cards for udm=18, the Actor transparently uses a Google web search restricted to established forum domains and labels every such record with searchMode: "web_forum_fallback".

What it extracts

Every result includes:

• Stable SHA-256 ID and canonical public thread URL
• Thread title, domain, host, platform, forum/community, and source name
• Query position, result position, page, original query, and effective filtered query
• Google search URL, country, language, market, filter context, and scrape time

When Google displays or the public URL proves the values, records also include:

• Snippet/description and displayed URL breadcrumbs
• Platform thread ID
• Forum/community URL
• Author and verified public author URL pattern for Reddit
• Reply/comment, view, and vote counts with matching display text
• Displayed publication date, normalized timestamp, and relative time
• Favicon and thumbnail

Optional values are omitted instead of returned as null, blank, or fabricated fields. The Actor does not open or scrape the destination threads, so it does not cross logins, private communities, paywalls, or target-site access controls.

Single query

{
  "mode": "single",
  "query": "best mechanical keyboard",
  "maxItems": 10,
  "maxPages": 2,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["GOOGLE_SERP"]
  }
}

Multiple queries

{
  "mode": "batch",
  "queries": [
    "best mechanical keyboard",
    "standing desk recommendations"
  ],
  "maxItems": 20,
  "maxPages": 2,
  "maxConcurrency": 2,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["GOOGLE_SERP"]
  }
}

The global limit is shared fairly between unique queries and final output is interleaved, preventing the first query from consuming the entire dataset.

Filters

• platform: all forums, Reddit, Quora, Stack Overflow, Stack Exchange, GitHub Discussions, Hacker News, or Discourse
• exactPhrase: require an exact phrase in addition to the query
• excludedTerms: add Google exclusion terms
• timeRange: past hour, day, week, month, year, or any time
• safeSearch: on or off
• gl and hl: Google country and language localization

Platform filters use transparent site: clauses in effectiveQuery; the original user query remains available separately.

Performance and reliability controls

Input	Default	Range
maxItems	50	1–500
maxPages	3	1–5 per query
maxConcurrency	2	1–10
maxRequestRetries	1	0–3
navigationTimeoutSecs	45	10–120
requestHandlerTimeoutSecs	90	30–180
resultsWaitSecs	12	3–60

Proxy and CAPTCHA behavior

GOOGLE_SERP is appropriate here because the target is a public google.com/search result page. The Actor sends HTTP requests through that group; it never tunnels Playwright HTTPS traffic through the HTTP-only SERP transport. Google's August 2026 opaque /goto links are resolved as normal public clicks with redirects disabled, and only a validated HTTP(S) Location target is accepted.

Direct, Residential, and custom proxy settings use the browser fallback with consistent Chrome fingerprints, stable per-session cookies, bounded retries, and country-aligned locales. Explicit CAPTCHA or unusual-traffic pages fail closed without retrying the same challenge and are never stored in the dataset. The Actor does not solve or bypass CAPTCHAs.

Output example

{
  "id": "f2f0...",
  "type": "forum",
  "resultType": "discussion",
  "position": 1,
  "title": "What is the best quiet switch?",
  "url": "https://www.reddit.com/r/MechanicalKeyboards/comments/...",
  "domain": "reddit.com",
  "platform": "Reddit",
  "forum": "r/MechanicalKeyboards",
  "replies": 245,
  "repliesText": "245 comments",
  "query": "quiet mechanical keyboard switches",
  "searchMode": "web_forum_fallback",
  "source": "Google Search (forum-filtered fallback)"
}

Counts and result content change over time; the example illustrates field types only.

Data-quality guarantees

• Correct Google Forums vertical (udm=18) with an explicitly labeled forum-filtered web fallback
• Stable canonical-URL IDs and global deduplication
• Typed integer engagement counts and ISO timestamps where derivable
• Status-code and content-type checks before parsing
• Explicit challenge and no-results classification
• No CAPTCHA, consent, navigation, or error pages stored as discussions
• No raw response, cookies, tokens, proxy credentials, or opaque Google state in output

Use reasonable limits and comply with Google and destination-community terms and applicable laws.