# B2B Agency Leads: Clutch, Sortlist & More (Merged) (`artsiom_k/agency-directory-scraper`) Actor

Find agencies across Clutch, Sortlist, GoodFirms and TechBehemoths in one run — the same agency listed on several sites is merged into a single lead with a combined rating, review total and website.

- **URL**: https://apify.com/artsiom\_k/agency-directory-scraper.md
- **Developed by:** [Artsiom Kunitsyn](https://apify.com/artsiom_k) (community)
- **Categories:** Lead generation, Business
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $5.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## B2B Agency Leads: Clutch, Sortlist, GoodFirms & TechBehemoths — merged

Collect agency leads from **four B2B agency directories in one run** — [Clutch](https://clutch.co),
[Sortlist](https://www.sortlist.com), [GoodFirms](https://www.goodfirms.co) and
[TechBehemoths](https://techbehemoths.com) — and get **one row per real agency**, not four
overlapping lists. An agency that appears on several directories is merged into a single lead with
its website, a review-weighted combined rating, the total review count, and every directory's own
profile link. No login required.

### Contents

- [Key features](#key-features)
- [Output](#output)
- [Input](#input)
- [Input examples](#input-examples)
- [Incremental (delta) mode](#incremental-delta-mode)
- [FAQ](#faq)

### 🔑 Key features

- **Cross-directory de-duplication.** The same agency is matched by website domain first, then by
  normalized name plus a compatible country — never on name alone when the websites differ. You get
  `listed_on_count` (how many directories list it — a quick signal of how established an agency is),
  `combined_rating`, `total_reviews` and a `sources` array with each directory's own numbers.
- **Four directories, one schema.** Rating (0–5), hourly rate range, minimum project size, team
  size, founding year, locations, services and website are normalized across all four sites.
- **Pick services and country.** Digital marketing, SEO, web development, web design, software
  development, mobile app development — worldwide or narrowed to one of 45 countries.
- **Websites for Clutch agencies, too.** Clutch's listing pages don't show a website; this actor
  can open each Clutch profile to get it (on by default), which is what makes Clutch agencies merge
  reliably with the other directories.
- **Delta mode built in.** Every uncapped run classifies each agency as `new`, `changed`,
  `unchanged` or `delisted` against a persisted baseline — pull only what changed on a schedule.
- **Reliable by design.** Failed pages are retried on a fresh connection; a page that still fails is
  skipped and reported (never silently truncating a crawl); a directory that is down is skipped and
  the run still delivers the others, with the problem listed in the run's `SUMMARY`.
- **`maxItems` defaults to 50** — a fast, cheap preview that also keeps an unconfigured run inside
  Apify's automated 5-minute check. Clear it (`null`) for a full crawl.

### 📋 Output

One dataset item per real-world agency. Agencies found on more than one directory come first.

**Example record (real data, trimmed) — one agency found on all four directories:**

```json
{
  "external_id": "19e7d7aec375c4cc",
  "name": "Ngital",
  "website": "https://ngital.com",
  "domain": "ngital.com",
  "country": "BD",
  "city": "Dhaka",
  "locations": ["Dhaka, Bangladesh"],
  "hourly_rate_min": 25,
  "hourly_rate_max": 49,
  "hourly_rate_currency": "USD",
  "min_project_size": "$1,000+",
  "team_size": "50 - 249",
  "founded_year": 2020,
  "services": [
    { "name": "Social Media Marketing", "percent": 40 },
    { "name": "Digital Strategy", "percent": 30 }
  ],
  "combined_rating": 5,
  "total_reviews": 76,
  "listed_on_count": 4,
  "listed_on": ["clutch", "goodfirms", "sortlist", "techbehemoths"],
  "sources": [
    { "source": "clutch", "url": "https://clutch.co/profile/ngital-0", "rating": 5, "review_count": 7 },
    { "source": "goodfirms", "url": "https://www.goodfirms.co/company/ngital", "rating": 5, "review_count": 7 },
    { "source": "techbehemoths", "url": "https://techbehemoths.com/company/ngital", "rating": 5, "review_count": 24 },
    { "source": "sortlist", "url": "https://www.sortlist.com/agency/ngital", "rating": 5, "review_count": 38 }
  ],
  "change_type": "new"
}
```

Where directories disagree on a single-value field (hourly rate, team size, founding year…), the
value comes from the most detailed directory (Clutch, then GoodFirms, then TechBehemoths, then
Sortlist); each directory's own values are always in `sources`.

### 🔧 Input

| Field | Type | Default | Description |
|---|---|---|---|
| `services` | array | `["digital-marketing"]` | Categories to collect: digital marketing, SEO, web development, web design, software development, mobile app development. |
| `country` | string | worldwide | Only agencies based in this country (45 supported). |
| `sources` | array | all four | Which directories to read: `clutch`, `goodfirms`, `sortlist`, `techbehemoths`. |
| `maxItems` | integer | `50` | Stop after this many agencies (and read at most this many records from each directory). `null` = full crawl. |
| `enrichClutchProfiles` | boolean | `true` | Open each Clutch profile for its website, phone and founding year. Off = faster and cheaper, Clutch agencies then merge by name only. |
| `mode` | string | `auto` | `auto` / `full` / `incremental` — see [Incremental mode](#incremental-delta-mode). |
| `concurrency` | integer | `8` | Parallel profile fetches per directory. |
| `proxyConfiguration` | object | **RESIDENTIAL** | Proxy used for Clutch only — see FAQ. |

### 📥 Input examples

**Default preview (50 agencies from each directory, best-matched first):**

```json
{}
```

**All SEO and web-design agencies in Germany, full crawl:**

```json
{ "services": ["seo", "web-design"], "country": "DE", "maxItems": null }
```

**Only the three directories that need no proxy:**

```json
{ "sources": ["sortlist", "goodfirms", "techbehemoths"], "maxItems": 1000 }
```

### 🔁 Incremental (delta) mode

`auto` does a full scan the first time an uncapped run happens for a given scope (services +
country + directories), then only pushes new or changed agencies on later runs and reports
agencies that disappeared as `delisted`. A run with **Max results** set always returns rows and never
updates the baseline; a run in which a directory failed or pages were skipped doesn't either, so a
partial crawl can never wrongly mark agencies as delisted.

### ❓ FAQ

**How does the de-duplication decide two listings are the same agency?**
Same website domain — always. Otherwise the same normalized name (case, accents, punctuation and
suffixes like "LLC"/"GmbH" ignored) **and** a compatible country, and only if the two don't have
*different* websites. Two agencies with the same name but different websites, or in different
countries, stay separate. A wrong merge is treated as worse than a missed one.

**Why do I see few merged agencies in a 50-item preview?**
The directories list tens of thousands of agencies each, and a preview takes the top of each ranking.
The overlap between directories grows with volume — a run of a few hundred agencies per directory
typically shows dozens of agencies listed on two or three sites.

**Does it need a proxy?**
Only for Clutch. Clutch returns HTTP 403 to nearly all requests from datacenter/cloud IPs; Apify's
Residential proxy (the default here) gets through, with occasional per-attempt failures that are
retried on a fresh IP. The other three directories are fetched directly. If you don't select Clutch,
no proxy is used at all.

**What is the `enrichClutchProfiles` option for, and what does it cost?**
Clutch's listing pages carry rating, reviews, rates and team size but not the agency's website. With
the option on, each Clutch agency's profile page is fetched too (through the Residential proxy),
which is what gives it a website and lets it merge with the other directories by domain. Turning it
off skips those extra requests.

**Are there fields some directories don't provide?**
Yes — each directory shows a different subset: Sortlist has no hourly rate or founding year in its
listing data, Clutch's service-mix percentages are unique to it, TechBehemoths' hourly rate is a band,
and so on. Missing values are `null`. Ratings are normalized to a 0–5 scale; TechBehemoths ratings are
overwhelmingly 5.0 on that site itself.

**Terms of use and personal data.**
Each directory has its own terms of use, and they differ. You are responsible for how you use the
collected data — review the terms of the directories you select and applicable regulations
(including GDPR/CCPA for any personal data such as phone numbers) before using it commercially.

# Actor input Schema

## `services` (type: `array`):

Which agency categories to collect. Each is translated to the matching category on every selected directory (a directory with no equivalent page is skipped for that service). Defaults to Digital marketing.

## `country` (type: `string`):

Optional: only agencies based in this country. Leave on Worldwide for no filter. Sortlist, TechBehemoths and Clutch are narrowed by URL; GoodFirms is filtered by each agency's listed location.

## `sources` (type: `array`):

Which directories to read. Agencies found on more than one selected directory are merged into a single row (the more directories you pick, the more merges).

## `maxItems` (type: `integer`):

Stop after this many agencies (and read at most this many records from each directory). Defaults to 50 — a fast, cheap preview, and what keeps an unconfigured run within Apify's automated 5-minute QA check. Agencies listed on several directories are pushed first. Raise it or clear it (set to null) for a full crawl — worldwide categories hold tens of thousands of agencies.

## `enrichClutchProfiles` (type: `boolean`):

Clutch's listing pages don't show an agency's website. When on (recommended), each Clutch agency's profile page is fetched to get its website, phone and founding year — the website is what lets Clutch agencies merge reliably with the other directories. Turn off for a faster, cheaper run that matches Clutch agencies by name only.

## `mode` (type: `string`):

"auto" (recommended): full scan on the first uncapped run for a given scope, incremental (new/changed only) afterwards; a run with Max results set always returns rows. "full": always push every agency and refresh the baseline. "incremental": always push only new/changed agencies. Only uncapped runs update the baseline or detect delisted agencies.

## `concurrency` (type: `integer`):

How many profile pages to fetch in parallel per directory (TechBehemoths and Clutch need one profile fetch per agency).

## `proxyConfiguration` (type: `object`):

Apify Proxy used for Clutch only — the other directories are fetched directly. Leave on the RESIDENTIAL group: Clutch returns HTTP 403 to nearly all requests from Apify's datacenter/cloud IPs (confirmed on real infra: 1 of 18 pages with no proxy, 2 of 18 through datacenter proxy). Residential works for roughly 70% of attempts and each failed attempt is retried on a fresh IP.

## Actor input object example

```json
{
  "services": [
    "digital-marketing"
  ],
  "country": "",
  "sources": [
    "clutch",
    "goodfirms",
    "sortlist",
    "techbehemoths"
  ],
  "maxItems": 50,
  "enrichClutchProfiles": true,
  "mode": "auto",
  "concurrency": 8,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL"
    ]
  }
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("artsiom_k/agency-directory-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("artsiom_k/agency-directory-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call artsiom_k/agency-directory-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,artsiom_k/agency-directory-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/7NkzirRyDRwOpUjtc/builds/qv7aCZeBQLxt9CEiN/openapi.json