Clutch B2B Agency Directory Scraper

Scrapes B2B service-provider rankings from Clutch.co — the leading directory for IT/web/ marketing/design agencies, ranked per category with verified client reviews. A genuinely new data-type for this portfolio: agency/vendor intelligence, not consumer listings, jobs, or product reviews.

Public data only. No login, no cookies, no browser.

The one thing you need to know before using this

Search results are Schema.org microdata (itemprop/itemscope), not JSON — every field lives on a <meta content="...">, <img src="..."> or <a href="..."> element. Detail pages, by contrast, carry real LocalBusiness JSON-LD with the full company description. See CRAWLING_METHOD.md for the full trail, including three dead-end targets tried and abandoned before landing here.

What you get

Three record types share one dataset, told apart by recordType.

PROVIDER — one row per agency/company

List cards already carry name, profile URL, phone, aggregate rating (value/count/best/worst), price range and full postal address. Turn on Fetch provider detail pages to also attach providerDetail, which adds the full company description and a named list of service packages — neither present on the list page.

SEARCH_SUMMARY — one row per category query

Pages fetched, providers returned, and segmentResolved — whether the category actually matched something upstream.

ERROR — one row per input that failed

So every entry in Categories maps to at least one output row.

Input

Field	What it does
Categories	Clutch category slugs (it-services, web-developers, digital-marketing) — one search per entry
Fetch provider detail pages	adds full description + service packages (off by default — one extra request per provider)
Max providers / max pages per category	pagination caps — see the dedup note below

Example

{
  "categories": ["it-services", "digital-marketing"],
  "includeProviderDetails": true,
  "maxItems": 50
}

Notes on reliability

• A category page mixes several sections — a "Leaders Matrix" ranking widget, a "Directory" list, "Sponsor"/"featured" slots — and some of them repeat across ?page=N (verified live: ~45% id overlap between page 1 and page 2). This actor dedupes by provider id and stops once a page carries no new providers, so you always get a clean, duplicate-free set regardless of that overlap.
• A bogus category answers a clean HTTP 404 — reported as segmentResolved: false, not silently widened to some default/broader listing.
• A dead/renamed profile slug answers a clean HTTP 404 — the search row is still emitted, with providerDetail: null.
• No hard WAF was observed — a per-TLS-profile split, not a blanket block (some profiles clean, some 403); the client rotates through a wide profile pool automatically.

Output envelope

Every record carries _input, _source and _scrapedAt. Since search results are microdata rather than a single upstream JSON object, each PROVIDER row's provider field is this actor's own explicit field extraction (not a renamed copy of a JSON blob, because none exists on that page) — the providerDetail field, when requested, IS raw JSON-LD passed through verbatim.

See CRAWLING_METHOD.md for the full reverse-engineering trail.