# DesignRush Scraper (`rl1987/design-rush-scraper`) Actor

Scrapes the DesignRush agency directory, agency profiles and Marketplace work inquiries.

- **URL**: https://apify.com/rl1987/design-rush-scraper.md
- **Developed by:** [R.L.](https://apify.com/rl1987) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.50 / 1,000 agency rows

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## DesignRush Scraper

Scrapy-based Apify Actor for [DesignRush](https://www.designrush.com/): the agency
directory, individual agency profiles, and the work inquiries posted to the
DesignRush Marketplace.

### Getting past Cloudflare

DesignRush is behind Cloudflare's managed challenge, which fingerprints the client
at the transport layer — the TLS ClientHello (cipher and extension ordering,
GREASE, ALPN), the HTTP/2 SETTINGS frame, and header ordering. Python's default
TLS stack produces a fingerprint no browser emits, so plain Scrapy is challenged
regardless of the `User-Agent` it sends.

Two things fix that, and both are needed:

1. **TLS fingerprint normalisation.** Every request is downloaded through
   [`impit`](https://github.com/apify/impit), a Rust HTTP client that reproduces
   the complete fingerprint of a real browser build. See
   `my_actor/downloaders.py`. Scrapy's own client headers are stripped so they
   cannot contradict the ones impit emits, and the impersonated browser is
   rotated per request.

   HTTP/3 is deliberately off (`IMPIT_HTTP3`). Cloudflare challenges DesignRush
   requests over HTTP/3 far more aggressively than the same requests over HTTP/2 --
   measured against the directory, HTTP/2 answered 8 of 8 while HTTP/3 was
   challenged on 5 of 8, and on the Apify platform HTTP/3 was challenged every
   time. Browsers do reach this site over HTTP/3, but impit's QUIC fingerprint
   evidently does not pass for one.
2. **Residential exit IPs, rotated per identity.** `my_actor/proxies.py` assigns
   every request a named Apify proxy session, and the download handler picks the
   proxy up from `request.meta['proxy']`. Sessions matter: Apify's own
   `ApifyHttpProxyMiddleware` hands out one identical proxy URL for every request,
   and since that URL is also the key connections are pooled under, a retry reuses
   the warm connection and goes back out through the very IP Cloudflare just
   flagged. First attempts rotate over a small pool of sessions so connections are
   still reused; each retry gets a session, and therefore an exit IP, of its own.

   Outside the Apify platform, set the `PROXY_URLS` setting to a list of proxy
   URLs and the same rotation applies to them.

The challenge is intermittent rather than absolute: the same URL that is blocked
on one (fingerprint, IP) pair usually succeeds on the next. `CloudflareChallengeMiddleware`
detects the interstitial — by the `cf-mitigated` header or the challenge markers in
the body, since it is served with an ordinary 403 — and retries with a different
impersonated browser and a fresh proxy session. In practice a handful of retries per
few dozen requests is normal and the crawl completes.

### Modes

The `mode` input selects the spider.

#### `agency` — directory and profiles

Start URLs are routed by path, so listings and profiles can be mixed freely:

| Start URL | Yields |
| --- | --- |
| `/agency` | the directory index -- holds no cards itself, so all 66 top-level categories are crawled |
| `/agency/<category>` | one item per agency card, 50 per page, walked one page at a time |
| `/agency/profile/<slug>` | one full profile item |

Set `scrapeProfiles` to follow every card in a listing through to its profile, so a
single run emits both item shapes. Popular categories run past 300 pages, so cap
the crawl with `maxPages`.

Listing items carry the card data: name, slogan, description, website, logo,
badges, top services, location, employee band, portfolio count and the featured
review. Minimal budget and average hourly rate are included where the card
publishes them -- on one sampled page that was 32 and 41 cards out of 50. Profile items add the contact details, overview stats (employees, minimal
budget, average hourly rate, year founded), rating and review count, services,
industries, client types, awards, headquarters and other locations, the portfolio
entries, and the full review list. Most of the profile's core data comes from the
page's `schema.org/Organization` JSON-LD, which is richer and more stable than the
rendered markup.

#### `marketplace-projects` — work inquiries

Scrapes the project feed behind the Marketplace's "View more Marketplace Projects"
button, which re-requests the page with a `page` parameter and an XHR header and is
answered with JSON.

This mode always starts from `/marketplace` and ignores `startUrls`. That page
returns the whole feed and paginates via a `hasMore` flag, whereas a service page
such as `/marketplace/web-design` answers with a short, unpaginated batch filtered
to that service -- starting there would quietly cap the output at a handful of
projects. The spider still understands both shapes if pointed at one directly with
`scrapy crawl`.

Each inquiry carries its project ID, type, industry, description, budget, service
and timestamps.

> **Note on personal data.** The feed also exposes contact details for whoever filed
> each inquiry (`contactName`, `contactEmail`, `contactPhone`, `company`), and these
> are included in the output. In a 60-row sample, 50 rows held a DesignRush staff
> account rather than the client — but 10 rows carried genuine submitter contacts,
> including personal mailbox addresses and real company domains. Handle the output
> accordingly.

### Input

See `.actor/input_schema.json`. Summary:

| Field | Meaning |
| --- | --- |
| `mode` | `agency` or `marketplace-projects` |
| `startUrls` | pages to start from, agency mode only; ignored in marketplace mode |
| `maxPages` | cap on result pages per start URL; empty means no cap |
| `scrapeProfiles` | in `agency` mode, follow listing cards to their profiles and merge both into one row |
| `proxyConfiguration` | Apify Proxy settings; `RESIDENTIAL` is strongly recommended |

### Running it

On the platform, or locally with the Apify CLI:

```bash
apify run --purge          # reads storage/key_value_stores/default/INPUT.json
```

The project is also a plain Scrapy project, which is the quickest way to iterate on
selectors — though without Apify Proxy you will see more challenges:

```bash
scrapy list
scrapy crawl agency -a start_urls=https://www.designrush.com/agency/logo-branding -a max_pages=2 -O out.json
scrapy crawl agency -a start_urls=https://www.designrush.com/agency/profile/clay -O profile.json
scrapy crawl marketplace_projects -a max_pages=3 -O projects.json
```

### Layout

```
my_actor/
├── main.py                        # Actor input handling and spider dispatch
├── downloaders.py                 # impit-backed download handler (TLS fingerprinting)
├── proxies.py                     # proxy assignment with per-retry session rotation
├── middlewares.py                 # Cloudflare challenge detection and identity rotation
├── items.py                       # AgencyItem, MarketplaceProjectItem
├── pipelines.py                   # drops empty fields from exported rows
├── settings.py                    # Scrapy settings
└── spiders/
    ├── _base.py                   # shared extraction and argument helpers
    ├── agency.py                  # directory listings and agency profiles
    └── marketplace_projects.py    # marketplace work inquiries
```

# Actor input Schema

## `mode` (type: `string`):

Which part of DesignRush to scrape. Each mode produces its own item shape.

## `startUrls` (type: `array`):

Pages to start from, in agency mode only. These can be directory listings such as https://www.designrush.com/agency/logo-branding and/or individual profiles such as https://www.designrush.com/agency/profile/clay - each URL is routed by its path. Leave empty to crawl the whole directory from https://www.designrush.com/agency. Ignored in marketplace mode, which always starts from https://www.designrush.com/marketplace, the only marketplace page whose project feed paginates.

## `maxPages` (type: `integer`):

Stop after this many result pages per start URL. Leave empty to crawl every page - popular directory categories run to 300+ pages of 50 agencies each.

## `proxyConfiguration` (type: `object`):

DesignRush is behind Cloudflare. The scraper normalises its TLS fingerprint to look like a real browser, but it also needs a residential exit IP — datacenter addresses get challenged far more often.

## `scrapeProfiles` (type: `boolean`):

Follow every card found in a directory listing to its profile page and merge the two into a single row, rather than emitting the card alone. Much slower and far more requests, but each agency arrives with its contact details, overview stats, portfolio and reviews. Profile URLs passed directly in "Start URLs" are always scraped regardless of this setting.

## Actor input object example

```json
{
  "mode": "agency",
  "startUrls": [
    {
      "url": "https://www.designrush.com/agency/logo-branding"
    }
  ],
  "maxPages": 5,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL"
    ]
  },
  "scrapeProfiles": false
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://www.designrush.com/agency/logo-branding"
        }
    ],
    "maxPages": 5,
    "proxyConfiguration": {
        "useApifyProxy": true,
        "apifyProxyGroups": [
            "RESIDENTIAL"
        ]
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("rl1987/design-rush-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://www.designrush.com/agency/logo-branding" }],
    "maxPages": 5,
    "proxyConfiguration": {
        "useApifyProxy": True,
        "apifyProxyGroups": ["RESIDENTIAL"],
    },
}

# Run the Actor and wait for it to finish
run = client.actor("rl1987/design-rush-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://www.designrush.com/agency/logo-branding"
    }
  ],
  "maxPages": 5,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL"
    ]
  }
}' |
apify call rl1987/design-rush-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,rl1987/design-rush-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/U99uJBPXHgCYGd5f6/builds/SveaeXcFscvaUJ21r/openapi.json