Clutch.co Agency Scraper

Scrapes B2B agency/company profiles from Clutch.co directory listing pages — the default /agencies directory or any category page (/developers, /web-design, /seo-firms, etc.).

What it does

Given one or more Clutch directory URLs, the actor:

1. Fetches each listing page and parses every agency card on it.
2. Follows pagination via the ?page={n} query parameter until either a page returns no cards, maxItems is reached, or maxPagesPerStartUrl is hit.
3. Pushes one dataset item per agency as it's scraped (streamed, not batched at the end).

It uses Crawlee's BeautifulSoupCrawler — Clutch's listing pages are server-rendered, so no headless browser is needed.

Input

Field	Type	Default	Description
startUrls	array	[{"url": "https://clutch.co/agencies"}]	Clutch directory/category URLs to crawl.
maxItems	integer	50	Max agency profiles to scrape across all start URLs.
maxPagesPerStartUrl	integer	50	Safety cap on paginated pages followed per start URL.
minRequestDelaySecs / maxRequestDelaySecs	number	1.5 / 3	Randomized delay range applied before each request.
proxyConfiguration	object	Apify Proxy, RESIDENTIAL group	Proxy config. Residential is recommended — Clutch rate-limits/blocks datacenter IPs.

Example input:

{
  "startUrls": [
    { "url": "https://clutch.co/agencies" },
    { "url": "https://clutch.co/web-design" }
  ],
  "maxItems": 100,
  "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"] }
}

Output

One dataset item per agency:

{
  "company_name": "Power Digital",
  "website": "https://powerdigitalmarketing.com/growthplan/?utm_source__c=thirdpartydirectory&...",
  "clutch_profile_url": "https://clutch.co/profile/power-digital",
  "tagline": "Power Digital is a comprehensive advertising agency specializing in...",
  "min_project_size": "$5,000+",
  "hourly_rate": "$100 - $149 / hr",
  "employee_count": "250 - 999",
  "location": "San Diego, CA",
  "rating": 4.8,
  "review_count": 66,
  "services_focus": ["15% Advertising", "25% Social Media Marketing", "..."],
  "verified_status": true
}

Fields Clutch doesn't display for a given card (e.g. an undisclosed hourly rate) come back as null rather than a guessed value.

Note: website is the agency's real external URL — Clutch wraps outbound links in an r.clutch.co/redirect?... tracking redirect, which is unwrapped automatically. tagline is the card's description blurb (Clutch doesn't expose a separate short tagline on listing pages), so it can run a sentence or two.

Local development

pip install -r requirements-dev.txt
pytest

Parser tests run against real captured Clutch markup in tests/fixtures/ — no network access required.

To run the actor itself locally against the live site:

mkdir -p storage/key_value_stores/default
cat > storage/key_value_stores/default/INPUT.json <<'EOF'
{
  "startUrls": [{ "url": "https://clutch.co/agencies" }],
  "maxItems": 5,
  "proxyConfiguration": { "useApifyProxy": false }
}
EOF
python -m src

Results land in storage/datasets/default/.

Deploying

$apify push