Gemini Scraper — Apify Actor

Scrapes text responses from Google Gemini by replaying authenticated browser requests. Accepts a list of queries, sends each to Gemini, and saves the full AI-generated response to an Apify dataset.

⚠️ This actor uses Gemini's internal browser API, not the official Gemini REST API. Session cookies are hardcoded in the source and must be refreshed in the code when they expire (~24 hours).

---

Features

• Send multiple queries in a single run
• Only input required: a list of queries
• Full response text extracted from Gemini's streaming protocol
• Per-query error isolation — one failure does not stop the rest
• Dataset output with structured fields: query, response, error, scrapedAt

---

Input

Only one field is required.

Field	Type	Required	Description
queries	string[]	✅	One or more queries to send to Gemini

Example Input JSON

{
  "queries": [
    "best smartphones 2026",
    "python vs javascript for web development",
    "how does quantum computing work"
  ]
}

---

Output

Each query produces one record in the default dataset.

Success Record

{
  "query":     "best smartphones 2026",
  "response":  "The top smartphones in 2026 include...",
  "error":     null,
  "scrapedAt": "2026-06-19T10:30:00.000Z"
}

Failure Record

{
  "query":     "best smartphones 2026",
  "response":  null,
  "error":     "HTTPError: 403 Forbidden",
  "scrapedAt": "2026-06-19T10:30:00.000Z"
}

---

When Cookies Expire

All session values (cookies, headers, params) are hardcoded at the top of src/main.py under COOKIES, HEADERS, PARAMS, and SESSION_ID.

When requests start returning 401 or 403, refresh the values:

Step-by-step

1. Open https://gemini.google.com in Chrome while signed in.
2. Press F12 → Network tab.
3. Send any message in Gemini.
4. Filter by StreamGenerate → click the request.
5. Go to Headers → Request Headers → copy the Cookie header.
6. Extract __Secure-STRP, NID, and COMPASS values.
7. Update COOKIES in src/main.py.
8. From the same request's Query String Parameters, copy bl and f.sid.
9. Update PARAMS in src/main.py.
10. Redeploy the actor.

---

Project Structure

gemini-actor/
├── src/
│   └── main.py              # Core actor logic + hardcoded session values
├── .actor/
│   ├── actor.json           # Apify actor manifest
│   └── INPUT_SCHEMA.json    # Input schema (queries only)
├── output_schema.json       # Dataset view columns
├── dataset_schema.json      # Per-record field types
├── requirements.txt         # Python dependencies
├── Dockerfile               # Container build
├── Agent.md                 # Internal developer/agent guide
├── claude.md                # AI assistant context file
└── README.md                # This file

---

Local Development

# Install dependencies
pip install apify requests

# Run locally (requires Apify CLI)
apify run

---

Dependencies

apify>=2.0.0
requests>=2.31.0

---

Limitations

• Session cookies expire in ~24 hours and require a manual code update + redeploy
• Subject to Gemini's internal rate limits
• Response format may break if Google updates the StreamGenerate protocol
• Not suitable for high-volume production use without cookie rotation

---

License

MIT