DesignRush Scraper

Scrapy-based Apify Actor for DesignRush: the agency directory, individual agency profiles, and the work inquiries posted to the DesignRush Marketplace.

Getting past Cloudflare

DesignRush is behind Cloudflare's managed challenge, which fingerprints the client at the transport layer — the TLS ClientHello (cipher and extension ordering, GREASE, ALPN), the HTTP/2 SETTINGS frame, and header ordering. Python's default TLS stack produces a fingerprint no browser emits, so plain Scrapy is challenged regardless of the User-Agent it sends.

Two things fix that, and both are needed:

1. TLS fingerprint normalisation. Every request is downloaded through impit, a Rust HTTP client that reproduces the complete fingerprint of a real browser build. See my_actor/downloaders.py. Scrapy's own client headers are stripped so they cannot contradict the ones impit emits, and the impersonated browser is rotated per request.

   HTTP/3 is deliberately off (IMPIT_HTTP3). Cloudflare challenges DesignRush requests over HTTP/3 far more aggressively than the same requests over HTTP/2 -- measured against the directory, HTTP/2 answered 8 of 8 while HTTP/3 was challenged on 5 of 8, and on the Apify platform HTTP/3 was challenged every time. Browsers do reach this site over HTTP/3, but impit's QUIC fingerprint evidently does not pass for one.

2. Residential exit IPs, rotated per identity. my_actor/proxies.py assigns every request a named Apify proxy session, and the download handler picks the proxy up from request.meta['proxy']. Sessions matter: Apify's own ApifyHttpProxyMiddleware hands out one identical proxy URL for every request, and since that URL is also the key connections are pooled under, a retry reuses the warm connection and goes back out through the very IP Cloudflare just flagged. First attempts rotate over a small pool of sessions so connections are still reused; each retry gets a session, and therefore an exit IP, of its own.

   Outside the Apify platform, set the PROXY_URLS setting to a list of proxy URLs and the same rotation applies to them.

The challenge is intermittent rather than absolute: the same URL that is blocked on one (fingerprint, IP) pair usually succeeds on the next. CloudflareChallengeMiddleware detects the interstitial — by the cf-mitigated header or the challenge markers in the body, since it is served with an ordinary 403 — and retries with a different impersonated browser and a fresh proxy session. In practice a handful of retries per few dozen requests is normal and the crawl completes.

Modes

The mode input selects the spider.

agency — directory and profiles

Start URLs are routed by path, so listings and profiles can be mixed freely:

Start URL	Yields
/agency	the directory index -- holds no cards itself, so all 66 top-level categories are crawled
/agency/<category>	one item per agency card, 50 per page, walked one page at a time
/agency/profile/<slug>	one full profile item

Set scrapeProfiles to follow every card in a listing through to its profile, so a single run emits both item shapes. Popular categories run past 300 pages, so cap the crawl with maxPages.

Listing items carry the card data: name, slogan, description, website, logo, badges, top services, location, employee band, portfolio count and the featured review. Minimal budget and average hourly rate are included where the card publishes them -- on one sampled page that was 32 and 41 cards out of 50. Profile items add the contact details, overview stats (employees, minimal budget, average hourly rate, year founded), rating and review count, services, industries, client types, awards, headquarters and other locations, the portfolio entries, and the full review list. Most of the profile's core data comes from the page's schema.org/Organization JSON-LD, which is richer and more stable than the rendered markup.

marketplace-projects — work inquiries

Scrapes the project feed behind the Marketplace's "View more Marketplace Projects" button, which re-requests the page with a page parameter and an XHR header and is answered with JSON.

This mode always starts from /marketplace and ignores startUrls. That page returns the whole feed and paginates via a hasMore flag, whereas a service page such as /marketplace/web-design answers with a short, unpaginated batch filtered to that service -- starting there would quietly cap the output at a handful of projects. The spider still understands both shapes if pointed at one directly with scrapy crawl.

Each inquiry carries its project ID, type, industry, description, budget, service and timestamps.

Note on personal data. The feed also exposes contact details for whoever filed each inquiry (contactName, contactEmail, contactPhone, company), and these are included in the output. In a 60-row sample, 50 rows held a DesignRush staff account rather than the client — but 10 rows carried genuine submitter contacts, including personal mailbox addresses and real company domains. Handle the output accordingly.

Input

See .actor/input_schema.json. Summary:

Field	Meaning
mode	agency or marketplace-projects
startUrls	pages to start from, agency mode only; ignored in marketplace mode
maxPages	cap on result pages per start URL; empty means no cap
scrapeProfiles	in agency mode, follow listing cards to their profiles and merge both into one row
proxyConfiguration	Apify Proxy settings; RESIDENTIAL is strongly recommended

Running it

On the platform, or locally with the Apify CLI:

$apify run --purge          # reads storage/key_value_stores/default/INPUT.json

The project is also a plain Scrapy project, which is the quickest way to iterate on selectors — though without Apify Proxy you will see more challenges:

scrapy list
scrapy crawl agency -a start_urls=https://www.designrush.com/agency/logo-branding -a max_pages=2 -O out.json
scrapy crawl agency -a start_urls=https://www.designrush.com/agency/profile/clay -O profile.json
scrapy crawl marketplace_projects -a max_pages=3 -O projects.json

Layout

my_actor/
├── main.py                        # Actor input handling and spider dispatch
├── downloaders.py                 # impit-backed download handler (TLS fingerprinting)
├── proxies.py                     # proxy assignment with per-retry session rotation
├── middlewares.py                 # Cloudflare challenge detection and identity rotation
├── items.py                       # AgencyItem, MarketplaceProjectItem
├── pipelines.py                   # drops empty fields from exported rows
├── settings.py                    # Scrapy settings
└── spiders/
    ├── _base.py                   # shared extraction and argument helpers
    ├── agency.py                  # directory listings and agency profiles
    └── marketplace_projects.py    # marketplace work inquiries