# g2 scraper(it works!) (`coder_luuffy/g2-scraper`) Actor

- **URL**: https://apify.com/coder\_luuffy/g2-scraper.md
- **Developed by:** [coder\_luuffy](https://apify.com/coder_luuffy) (community)
- **Categories:** Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / actor start

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## G2.com Category Listings (MongoDB)

Returns stored G2.com software listings for one or more category URLs.

Input and output match [`coder_luffy/g2-scraper`](https://console.apify.com/actors/k5kcptc7CJLhQVC3q/input)
field for field, so this is a drop-in replacement for it — except that records are read from
MongoDB instead of scraped from G2.com.

### Input

| Field | Type | Default | Description |
|---|---|---|---|
| `urls` | array of strings | required | One or more G2.com category page URLs. A bare slug such as `crm` also works. |
| `maxItems` | integer | `25` | Maximum listings to return across all URLs. Max `50000`. |
| `requestTimeoutSecs` | integer | `30` | Per-query timeout in seconds, `5` to `120`. |
| `enrichFromProductPage` | boolean | `false` | Accepted for input compatibility, but ignored — see below. |

```json
{
  "urls": [
    "https://www.g2.com/categories/aca-compliance",
    "https://www.g2.com/categories/it-financial-management-itfm"
  ],
  "maxItems": 25
}
```

`maxItems` is a total across all URLs, not a per-URL cap. Categories are read in the order given
until the total is reached; the rest are skipped. Duplicate URLs pointing at the same category are
read once.

All of these resolve to the same category:

```
https://www.g2.com/categories/data-labeling/
https://www.g2.com/categories/data-labeling
http://g2.com/categories/data-labeling?page=3
www.g2.com/categories/data-labeling
data-labeling
```

An entry that resolves to neither a URL nor a slug is skipped with a warning; the run continues
with the rest. If nothing resolves, the run fails.

`enrichFromProductPage` exists so that input copied from `coder_luffy/g2-scraper` is accepted
verbatim. There are no product pages to fetch when reading from MongoDB, so setting it to `true`
logs a warning and changes nothing.

### Output

One dataset item per matching document in `g2list.catalog_listings`, carrying the same 25 fields in
the same order as `coder_luffy/g2-scraper`:

```json
{
  "productId": 103210,
  "productUuid": null,
  "productName": "NavigateHCR",
  "productSlug": "navigatehcr",
  "productUrl": "https://www.g2.com/products/navigatehcr/reviews",
  "logoUrl": "https://images.g2crowd.com/uploads/product/image/large_detail/...",
  "vendor": "NavigateHCR",
  "vendorUrl": "https://www.g2.com/sellers/navigatehcr",
  "ratingAvg": 5,
  "numReviews": 1,
  "entryLevelPrice": null,
  "productDescription": null,
  "prosHighlights": [],
  "consHighlights": [],
  "userSentiment": null,
  "typicalUsers": null,
  "industries": null,
  "marketSegment": null,
  "consultingServicesUrl": null,
  "category": "ACA Compliance",
  "categoryId": null,
  "pageNumber": 1,
  "sourceUrl": "https://www.g2.com/categories/aca-compliance",
  "scrapedAt": "2026-08-12T19:40:38.144Z",
  "error": null
}
```

Records come back in `pageNumber` then `_id` order, which is stable across runs.

An unknown category is not an error — it contributes no items and logs a warning. A run whose
categories are all unknown succeeds with an empty dataset.

#### How fields are filled

A document's own value always wins. When a field is absent, it is derived if possible, and
otherwise `null` (or `[]` for the two list fields). So as richer records land in MongoDB, they
pass straight through and the nulls fill themselves in — no code change needed.

| Output field | Source |
|---|---|
| `productId` `productName` `productUrl` `logoUrl` `vendor` `vendorUrl` `ratingAvg` `numReviews` `prosHighlights` `consHighlights` `userSentiment` `pageNumber` | Stored directly |
| `productSlug` | Stored, else parsed from `productUrl` |
| `category` | Stored, else `categoryName` |
| `sourceUrl` | Stored, else built from the category slug |
| `scrapedAt` | Stored, else `ingestedAt`, as an ISO string |
| `consultingServicesUrl` | Stored, else `/products/<slug>/imp` built from `productUrl` |
| `productUuid` `entryLevelPrice` `productDescription` `typicalUsers` `industries` `marketSegment` `categoryId` `error` | Stored, else `null` |

Against `catalog_listings` as it stands, 12 of the 25 columns are filled on every record,
`consultingServicesUrl` `productSlug` `category` `sourceUrl` `scrapedAt` are derived, and the
remaining 8 are `null` because that data is not in the collection.

Fields present in `catalog_listings` but not in the reference format — `_id`, `isAiVerified`,
`solutionType`, `prosDetail`, `consDetail` — are dropped, so the output stays field-for-field
identical to the reference actor.

### Environment variables

The connection string is read from the environment rather than from the actor input, so the
password never lands in the run's input record or the Console UI.

| Variable | Required | Default | Description |
|---|---|---|---|
| `MONGODB_URL` | yes | — | MongoDB connection string |
| `MONGODB_DB` | no | `g2list` | Database name |
| `MONGODB_COLLECTION` | no | `catalog_listings` | Collection holding the listings |

On the Apify platform, add `MONGODB_URL` under **Actor → Settings → Environment variables** and
tick **Secret**. Locally it is read from `.env` in the project root:

```
MONGODB_URL=mongodb+srv://user:password@host/?appName=g2list
```

### Running locally

```bash
npm install
npm test
echo '{ "urls": ["aca-compliance"], "maxItems": 25 }' > storage/key_value_stores/default/INPUT.json
npm start
```

Results are written to `storage/datasets/default/`.

`npm test` covers the field mapping — that the output carries the reference actor's 25 fields in
its order, that sparse documents are derived correctly, and that full scraper documents pass
through untouched.

### Deploying

```bash
npm install -g apify-cli
apify login
apify push
```

`.env` is excluded by `.dockerignore`, so set `MONGODB_URL` on the actor before the first run.

### Scaling note

`catalog_listings` holds ~242,000 listings across ~1,737 categories. It is indexed on
`categorySlug`, which is the field this actor queries.

Measured on the largest category, `emerging-ai-software` (6,000 docs): the query examines exactly
the 6,000 matching documents via the index and returns in ~13 ms, with the sort done in memory
well inside MongoDB's 32 MB limit. Results stream through a cursor and are pushed in batches of
500, so memory stays flat regardless of category size.

If categories grow much beyond this, a compound index turns the sort into an index scan:

```js
db.catalog_listings.createIndex({ categorySlug: 1, pageNumber: 1, _id: 1 })
```

# Actor input Schema

## `urls` (type: `array`):

One or more G2.com category page URLs. A bare category slug such as "crm" is also accepted.

## `maxItems` (type: `integer`):

Maximum listings to return across all URLs.

## `requestTimeoutSecs` (type: `integer`):

Per-query timeout in seconds. Applied to MongoDB server selection, the socket, and each query.

## `enrichFromProductPage` (type: `boolean`):

Accepted for compatibility with the G2.com Scraper input, but ignored: this Actor reads stored records from MongoDB and never fetches product pages.

## Actor input object example

```json
{
  "urls": [
    "https://www.g2.com/categories/crm"
  ],
  "maxItems": 25,
  "requestTimeoutSecs": 30,
  "enrichFromProductPage": false
}
```

# Actor output Schema

## `listings` (type: `string`):

One item per software listing, in the same format as the G2.com Scraper.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://www.g2.com/categories/crm"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("coder_luuffy/g2-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": ["https://www.g2.com/categories/crm"] }

# Run the Actor and wait for it to finish
run = client.actor("coder_luuffy/g2-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://www.g2.com/categories/crm"
  ]
}' |
apify call coder_luuffy/g2-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,coder_luuffy/g2-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/BCs2ayQQShMcHf4hx/builds/f3AKpQHhgh3Otwsac/openapi.json