ApifyGoogleForums – Google Forums Search Scraper

ApifyGoogleForums is an Apify Actor that searches the Google Search “Forums” tab via a structured API and returns forum results (threads, Q&A, discussions) as clean JSON, one page per dataset item. It is designed for pay‑per‑event pricing (setup + page processed) and production‑grade reliability.

---

What this Actor does

• Searches Google Forums for a given query (q).
• Supports localization via:
  • location (human‑readable location string),
  • gl (country code),
  • hl (language code).
• Supports advanced filters from the Google Forums API:
  • nfpr – exclude auto‑corrected queries,
  • filter – enable/disable duplicate filtering.
• Allows choosing the device type used for the search (desktop, mobile, tablet).
• Handles pagination across multiple pages of forums results.
• Writes one dataset item per page, and optionally persists the raw combined results to a JSON file.

Each dataset item contains:

• search_parameters: effective parameters used for the query.
• search_metadata: counts, pagination info, and pages processed.
• search_information: search‑level metadata returned by the API.
• search_timestamp: ISO timestamp of the page extraction.
• page_number: page index (1‑based).
• forum_results: array of forum entries for that page (position, title, link, displayed meta, source, snippet, date, etc.).

---

Input

The full input schema is defined in input_schema.json and rendered automatically in the Apify UI. Key fields:

• q (string, required)
  Search query for forums, for example:

  • "what is the best programming language",
  • "python vs java",
  • "best coffee brewing methods forum".

• location (string, optional)
  Geographic location for localized results, e.g.:

  • "Austin, TX, Texas, United States",
  • "New York, NY".

• gl (string, optional)
  Country code for localization (ISO 3166‑1 alpha‑2, lowercase), e.g. us, uk, de.
  See the Apify input UI for the complete enumerated list.

• hl (string, optional)
  Language code for localization (ISO 639‑1, lowercase).
  The actor normalizes regional variants (such as en-gb) to base codes (such as en).

• lr (string, optional)
  Language restriction for search results, for example lang_en, lang_es.

• nfpr (\"0\" | \"1\", optional)
  Controls whether to exclude auto‑corrected results:

  • "0" – include auto‑corrected results (default),
  • "1" – exclude auto‑corrected results; only the exact query is used.

• filter (\"0\" | \"1\", optional)
  Controls duplicate filtering:

  • "0" – include duplicates (default),
  • "1" – filter duplicates to keep unique content.

• device (\"desktop\" | \"mobile\" | \"tablet\", optional)
  Device type to emulate for the Google Forums search. Defaults to "desktop".

• max_pages (integer, optional, default 1)
  Maximum number of pages of Google Forums results to fetch:

  • 1 – only the first page (default),
  • N > 1 – up to N pages,
  • 0 – no limit; fetch all available pages.

  Each processed page is charged as one page_processed event in the pay‑per‑event model.

• output_file (string, optional)
  Optional filename to save the full combined results JSON into the default key‑value store. If omitted, a filename like:

  google_forums_search_<query>_<timestamp>.json

  is generated automatically.

---

Example inputs

1. Basic forums search

Search for discussions about the best programming language, one page:

{
  "q": "what is the best programming language",
  "max_pages": 1
}

2. Localized forums search

Forums discussions about coffee brewing, localized to Austin, US, in English:

{
  "q": "best coffee brewing methods forum",
  "location": "Austin, TX, Texas, United States",
  "gl": "us",
  "hl": "en",
  "max_pages": 1
}

3. Forums search with filters and pagination

Two pages of technology discussions, excluding auto‑corrected results and filtering duplicates:

{
  "q": "technology trends 2026",
  "gl": "us",
  "hl": "en",
  "nfpr": "1",
  "filter": "1",
  "device": "desktop",
  "max_pages": 2
}

4. All available pages

Fetch all available forum pages for a given query:

{
  "q": "python vs java web development",
  "gl": "us",
  "hl": "en",
  "max_pages": 0
}

---

Output structure

Each page of results becomes one dataset item, for example:

{
  "search_parameters": {
    "q": "what is the best programming language",
    "location": null,
    "gl": "us",
    "hl": "en",
    "lr": null,
    "nfpr": 1,
    "filter": 1,
    "device": "desktop",
    "max_pages": 2
  },
  "search_metadata": {
    "total_results": 42,
    "forums_count": 20,
    "pages_processed": 2,
    "max_pages_set": 2,
    "pagination_limit_reached": true
  },
  "search_information": {
    "query_displayed": "what is the best programming language",
    "organic_results_state": "Results for exact spelling"
  },
  "search_timestamp": "2026-03-16T10:30:00.123456",
  "page_number": 1,
  "forum_results": [
    {
      "position": 1,
      "title": "For you what is the best programming language?",
      "link": "https://www.reddit.com/r/learnprogramming/comments/14o4uqz/for_you_what_is_the_best_programming_language_w/",
      "displayed_meta": "70+ comments · 2 years ago",
      "source": "Reddit · r/learnprogramming",
      "snippet": "C++ It's the biggest language in the world until now...",
      "date": "2 years ago"
    }
    // ...
  ]
}

See dataset_schema.json for the full, strongly typed output schema.

---

Pricing and charging model

This Actor uses pay‑per‑event pricing:

• setup – one‑time setup fee per run (default $0.02).
• page_processed – one event per successfully pushed forums page (default $0.02 per page).

At the beginning of the run in production mode, the Actor:

1. Uses Apify’s ChargingManager to read the effective pricing and maximum charge limit.
2. Estimates the maximum possible cost as setup + page_price * pages_to_check.
3. If the projected cost would exceed the remaining budget, it:
   • logs a clear warning,
   • pushes a structured InsufficientFunds error item to the dataset,
   • exits gracefully without charging.
4. If there is sufficient budget, it charges the setup event.
5. After each page is pushed successfully to the dataset, it charges one page_processed event.

When the PRODUCTION environment variable is not set to true, the Actor runs in local testing mode: it logs what would be charged, but does not perform any real charges.

---

Running locally

From the project root:

cd ApifyGoogleForums
apify run --input '{
  "q": "what is the best programming language",
  "gl": "us",
  "hl": "en",
  "max_pages": 1
}'

You can then inspect:

• The default dataset for page‑by‑page forum_results.
• The generated JSON file (if output_file is set) in the default key‑value store.

For production runs on Apify, set PRODUCTION=true in your Actor environment to enable pay‑per‑event charging.

---

Last updated: 2026‑03‑16