Clutch.co Scraper + Sortlist, GoodFirms & TechBehemoths: Merged Agency Leads

Collect agency leads from Clutch.co and three more B2B agency directories in one run — Clutch, Sortlist, GoodFirms and TechBehemoths — and get one row per real agency, not four overlapping lists. An agency that appears on several directories is merged into a single lead with its website, phone number, a review-weighted combined rating, the total review count, and every directory's own profile link. No login required.

Contents

• Key features
• Output
• Input
• Input examples
• Incremental (delta) mode
• FAQ

🔑 Key features

• Cross-directory de-duplication. The same agency is matched by website domain first, then by normalized name plus a compatible country — never on name alone when the websites differ. You get listed_on_count (how many directories list it — a quick signal of how established an agency is), combined_rating, total_reviews and a sources array with each directory's own numbers.
• Four directories, one schema. Rating (0–5), hourly rate range, minimum project size, team size, founding year, locations, services and website are normalized across all four sites.
• Pick services and country. Digital marketing, SEO, web development, web design, software development, mobile app development — worldwide or narrowed to one of 45 countries.
• Websites for Clutch agencies, too. Clutch's listing pages don't show a website; this actor can open each Clutch profile to get it (on by default), which is what makes Clutch agencies merge reliably with the other directories.
• Phone numbers. Taken from Clutch and TechBehemoths; for agencies only on Sortlist or GoodFirms (which don't publish phones), the agency's own homepage is checked for its phone. phone_source says where each number came from.
• Delta mode built in. Every uncapped run classifies each agency as new, changed, unchanged or delisted against a persisted baseline — pull only what changed on a schedule.
• Reliable by design. Failed pages are retried on a fresh connection; a page that still fails is skipped and reported (never silently truncating a crawl); a directory that is down is skipped and the run still delivers the others, with the problem listed in the run's SUMMARY.
• maxItems defaults to 50 — a fast, cheap preview that also keeps an unconfigured run inside Apify's automated 5-minute check. Clear it (null) for a full crawl.

📋 Output

One dataset item per real-world agency. Agencies found on more than one directory come first.

Example record (real data, trimmed) — one agency found on all four directories:

{
  "external_id": "19e7d7aec375c4cc",
  "name": "Ngital",
  "website": "https://ngital.com",
  "domain": "ngital.com",
  "country": "BD",
  "city": "Dhaka",
  "locations": ["Dhaka, Bangladesh"],
  "hourly_rate_min": 25,
  "hourly_rate_max": 49,
  "hourly_rate_currency": "USD",
  "min_project_size": "$1,000+",
  "team_size": "50 - 249",
  "founded_year": 2020,
  "services": [
    { "name": "Social Media Marketing", "percent": 40 },
    { "name": "Digital Strategy", "percent": 30 }
  ],
  "combined_rating": 5,
  "total_reviews": 76,
  "listed_on_count": 4,
  "listed_on": ["clutch", "goodfirms", "sortlist", "techbehemoths"],
  "sources": [
    { "source": "clutch", "url": "https://clutch.co/profile/ngital-0", "rating": 5, "review_count": 7 },
    { "source": "goodfirms", "url": "https://www.goodfirms.co/company/ngital", "rating": 5, "review_count": 7 },
    { "source": "techbehemoths", "url": "https://techbehemoths.com/company/ngital", "rating": 5, "review_count": 24 },
    { "source": "sortlist", "url": "https://www.sortlist.com/agency/ngital", "rating": 5, "review_count": 38 }
  ],
  "change_type": "new"
}

Where directories disagree on a single-value field (hourly rate, team size, founding year…), the value comes from the most detailed directory (Clutch, then GoodFirms, then TechBehemoths, then Sortlist); each directory's own values are always in sources.

🔧 Input

Field	Type	Default	Description
services	array	["digital-marketing"]	Categories to collect: digital marketing, SEO, web development, web design, software development, mobile app development.
country	string	worldwide	Only agencies based in this country (45 supported).
sources	array	all four	Which directories to read: clutch, goodfirms, sortlist, techbehemoths.
maxItems	integer	50	Stop after this many agencies (and read at most this many records from each directory). null = full crawl.
enrichClutchProfiles	boolean	true	Open each Clutch profile for its website, phone and founding year. Off = faster and cheaper, Clutch agencies then merge by name only.
findPhonesOnWebsites	boolean	true	For agencies with no directory phone, open the agency's homepage once and take its phone.
mode	string	auto	auto / full / incremental — see Incremental mode.
concurrency	integer	8	Parallel profile fetches per directory.
proxyConfiguration	object	RESIDENTIAL	Proxy used for Clutch only — see FAQ.

📥 Input examples

Default preview (50 agencies from each directory, best-matched first):

{}

All SEO and web-design agencies in Germany, full crawl:

{ "services": ["seo", "web-design"], "country": "DE", "maxItems": null }

Only the three directories that need no proxy:

{ "sources": ["sortlist", "goodfirms", "techbehemoths"], "maxItems": 1000 }

🔁 Incremental (delta) mode

auto does a full scan the first time an uncapped run happens for a given scope (services + country + directories), then only pushes new or changed agencies on later runs and reports agencies that disappeared as delisted. A run with Max results set always returns rows and never updates the baseline; a run in which a directory failed or pages were skipped doesn't either, so a partial crawl can never wrongly mark agencies as delisted.

❓ FAQ

How does the de-duplication decide two listings are the same agency? Same website domain — always. Otherwise the same normalized name (case, accents, punctuation and suffixes like "LLC"/"GmbH" ignored) and a compatible country, and only if the two don't have different websites. Two agencies with the same name but different websites, or in different countries, stay separate. A wrong merge is treated as worse than a missed one.

Why do I see few merged agencies in a 50-item preview? The directories list tens of thousands of agencies each, and a preview takes the top of each ranking. The overlap between directories grows with volume — a run of a few hundred agencies per directory typically shows dozens of agencies listed on two or three sites.

Does it need a proxy? Only for Clutch. Clutch returns HTTP 403 to nearly all requests from datacenter/cloud IPs; Apify's Residential proxy (the default here) gets through, with occasional per-attempt failures that are retried on a fresh IP. The other three directories are fetched directly. If you don't select Clutch, no proxy is used at all.

What is the enrichClutchProfiles option for, and what does it cost? Clutch's listing pages carry rating, reviews, rates and team size but not the agency's website. With the option on, each Clutch agency's profile page is fetched too (through the Residential proxy), which is what gives it a website and lets it merge with the other directories by domain. Turning it off skips those extra requests.

Are there fields some directories don't provide? Yes — each directory shows a different subset: Sortlist has no hourly rate or founding year in its listing data, Clutch's service-mix percentages are unique to it, TechBehemoths' hourly rate is a band, and so on. Missing values are null. Ratings are normalized to a 0–5 scale; TechBehemoths ratings are overwhelmingly 5.0 on that site itself.

Terms of use and personal data. Each directory has its own terms of use, and they differ. You are responsible for how you use the collected data — review the terms of the directories you select and applicable regulations (including GDPR/CCPA for any personal data such as phone numbers) before using it commercially.