# Scrappey Web Scraper – Managed Browser & Proxy API (`dormic/apify-scrappey`) Actor

Scrape data from modern, JavaScript-heavy web pages using the Scrappey.com API integrated with an Apify Actor. A robust solution for complex scraping scenarios — managed browser sessions, proxy rotation, and full browser automation.

- **URL**: https://apify.com/dormic/apify-scrappey.md
- **Developed by:** [Pim](https://apify.com/dormic) (community)
- **Categories:** Automation, Developer tools, Open source
- **Stats:** 172 total users, 1 monthly users, 100.0% runs succeeded, 12 bookmarks
- **User rating**: 5.00 out of 5 stars

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Scrappey Web Scraper for Apify

A web scraping actor for reliable access to modern, JavaScript-heavy websites. Handles automated verification flows, managed browser sessions, proxy rotation, and full browser automation. Powered by the [Scrappey.com](https://scrappey.com) API.

### Features

- **Managed Web Access** - Reliably retrieve content from dynamic sites that require a real browser environment
- **Verification Flow Handling** - Automatically work through common challenge and verification steps where access is permitted
- **Browser Automation** - Full browser control with 20+ action types including click, type, scroll, and JavaScript execution
- **Session Management** - Maintain cookies and state across multiple requests
- **Proxy Support** - Built-in proxy rotation with country selection, premium, and mobile proxy options
- **All HTTP Methods** - Support for GET, POST, PUT, DELETE, and PATCH requests
- **Data Extraction** - CSS selectors, regex patterns, screenshots, and video recording

### Quick Start

#### Input Configuration

```json
{
    "scrappeyApiKey": "your-api-key",
    "url": "https://example.com",
    "cmd": "request.get"
}
```

Get your API key at <https://scrappey.com>

#### Basic GET Request

```json
{
    "scrappeyApiKey": "your-api-key",
    "url": "https://httpbin.rs/get",
    "cmd": "request.get"
}
```

#### POST Request with Data

```json
{
    "scrappeyApiKey": "your-api-key",
    "url": "https://httpbin.rs/post",
    "cmd": "request.post",
    "postData": {
        "username": "user",
        "password": "pass"
    }
}
```

#### Browser Automation

```json
{
    "scrappeyApiKey": "your-api-key",
    "url": "https://example.com/login",
    "browserActions": [
        { "type": "wait_for_selector", "cssSelector": "#login-form" },
        { "type": "type", "cssSelector": "#username", "text": "myuser" },
        { "type": "type", "cssSelector": "#password", "text": "mypassword" },
        { "type": "click", "cssSelector": "#submit", "waitForSelector": ".dashboard" }
    ]
}
```

#### Automatic Verification Handling

```json
{
    "scrappeyApiKey": "your-api-key",
    "url": "https://example.com",
    "automaticallySolveCaptchas": true
}
```

#### Geo-Targeted Requests

```json
{
    "scrappeyApiKey": "your-api-key",
    "url": "https://example.com",
    "proxyCountry": "UnitedStates",
    "premiumProxy": true
}
```

#### Screenshot Capture

```json
{
    "scrappeyApiKey": "your-api-key",
    "url": "https://example.com",
    "screenshot": true,
    "screenshotUpload": true
}
```

### Input Options

#### Required

| Option | Type | Description |
|--------|------|-------------|
| `scrappeyApiKey` | string | Your Scrappey API key |
| `url` | string | Target URL |

#### HTTP Method

| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `cmd` | string | `request.get` | HTTP method: `request.get`, `request.post`, `request.put`, `request.delete`, `request.patch` |
| `postData` | object | - | Request body for POST, PUT, PATCH |
| `requestType` | string | `browser` | `browser` for full rendering, `request` for HTTP-only (faster) |

#### Proxy Configuration

| Option | Type | Description |
|--------|------|-------------|
| `proxy` | string | Custom proxy URL (http://user:pass@ip:port) |
| `proxyCountry` | string | Request a proxy from a specific country (e.g., UnitedStates, Germany) |
| `premiumProxy` | boolean | Use premium residential proxies |
| `mobileProxy` | boolean | Use mobile carrier proxies |

#### Verification Handling

| Option | Type | Description |
|--------|------|-------------|
| `automaticallySolveCaptchas` | boolean | Automatically work through detected verification steps |

#### Response Options

| Option | Type | Description |
|--------|------|-------------|
| `screenshot` | boolean | Capture page screenshot |
| `screenshotUpload` | boolean | Upload screenshot to storage |
| `video` | boolean | Record browser session |
| `cssSelector` | string | Extract content by CSS selector |
| `innerText` | boolean | Include page text content |
| `includeImages` | boolean | Include image URLs |
| `includeLinks` | boolean | Include link URLs |

#### Session and Cookies

| Option | Type | Description |
|--------|------|-------------|
| `session` | string | Session ID for state persistence |
| `cookiejar` | array | Cookies to set before request |
| `localStorage` | object | Local storage data to set |

### Output Data

```json
{
    "url": "https://example.com",
    "cmd": "request.get",
    "verified": true,
    "statusCode": 200,
    "currentUrl": "https://example.com",
    "userAgent": "Mozilla/5.0...",
    "cookies": [],
    "cookieString": "",
    "responseHeaders": {},
    "requestHeaders": {},
    "html": "<html>...</html>",
    "innerText": "Page content...",
    "ipInfo": {
        "country": "United States",
        "city": "New York"
    },
    "session": "session-id",
    "timeElapsed": 1234,
    "screenshot": "base64...",
    "screenshotUrl": "https://...",
    "timestamp": "2025-01-09T12:00:00.000Z"
}
```

### Browser Actions

Execute automated browser interactions:

| Action | Description |
|--------|-------------|
| `click` | Click an element |
| `type` | Type text into input |
| `wait` | Wait for milliseconds |
| `wait_for_selector` | Wait for element to appear |
| `scroll` | Scroll to element or bottom |
| `hover` | Hover over element |
| `keyboard` | Press keyboard keys |
| `dropdown` | Select dropdown option |
| `execute_js` | Run JavaScript code |
| `if` | Conditional execution |
| `while` | Loop execution |
| `goto` | Navigate to URL |

#### Example: Login Flow

```json
{
    "browserActions": [
        { "type": "wait_for_selector", "cssSelector": "#login-form" },
        { "type": "type", "cssSelector": "#email", "text": "user@example.com" },
        { "type": "type", "cssSelector": "#password", "text": "password123" },
        { "type": "click", "cssSelector": "#submit", "waitForSelector": ".dashboard" },
        { "type": "execute_js", "code": "document.querySelector('.user-name').innerText" }
    ]
}
```

### Error Codes

| Code | Description | Solution |
|------|-------------|----------|
| CODE-0001 | Server overload | Retry with backoff |
| CODE-0002 | Request not completed | Try a different proxy |
| CODE-0007 | Proxy error | Check proxy credentials |
| CODE-0010 | Access not completed | Use a premium proxy |
| CODE-0029 | Too many sessions | Wait for cleanup |
| CODE-0032 | Verification not completed | Retry request |

### Local Development

#### Setup

```bash
git clone https://github.com/pim97/apify-scrappey
cd apify-scrappey
npm install
```

#### Run Locally

```bash
## Set input in storage/key_value_stores/default/INPUT.json
npm run start:dev
```

#### Run Tests

```bash
npm test
```

#### Build

```bash
npm run build
```

#### Deploy to Apify

```bash
apify login
apify push
```

### CI/CD

The repository includes GitHub Actions for automatic deployment:

1. Tests run on every push and pull request
2. Deployment to Apify triggers on:
   - Push to main/master branch
   - New release published

Set `APIFY_TOKEN` secret in your repository settings.

### Resources

- [Scrappey Documentation](https://docs.scrappey.com)
- [Scrappey API Reference](https://docs.scrappey.com/api-reference)
- [Apify SDK Documentation](https://docs.apify.com/sdk/js)
- [Request Builder](https://app.scrappey.com/#/builder)

### Support

- [Scrappey Support](https://scrappey.com/support)
- [GitHub Issues](https://github.com/pim97/apify-scrappey/issues)
- [Apify Discord](https://discord.com/invite/jyEM2PRvMU)

### License

MIT License

# Actor input Schema

## `scrappeyApiKey` (type: `string`):

Your Scrappey API key. Get one at https://scrappey.com

## `url` (type: `string`):

The URL of the website to scrape

## `cmd` (type: `string`):

The HTTP method to use for the request

## `postData` (type: `object`):

Data to send with POST, PUT, or PATCH requests. Can be form data or JSON.

## `requestType` (type: `string`):

Use 'browser' for full browser rendering (slower, handles JS) or 'request' for HTTP-only mode (faster, cheaper)

## `proxyCountry` (type: `string`):

Request a proxy from a specific country (e.g., UnitedStates, Germany, UnitedKingdom)

## `proxy` (type: `string`):

Custom proxy URL in format: http://username:password@ip:port

## `premiumProxy` (type: `boolean`):

Use premium residential proxies for better success rates

## `mobileProxy` (type: `boolean`):

Use mobile carrier proxies

## `session` (type: `string`):

Session ID for maintaining state across requests. Leave empty for automatic session.

## `browserActions` (type: `array`):

Array of browser automation actions (click, type, scroll, wait, etc.)

## `automaticallySolveCaptchas` (type: `boolean`):

Automatically work through detected verification widgets where access is permitted

## `alwaysLoad` (type: `array`):

Verification widget types to always load handlers for

## `cloudflareBypass` (type: `boolean`):

Enable additional optimizations for sites with interstitial challenge pages

## `datadomeBypass` (type: `boolean`):

Enable additional optimizations for sites with advanced verification flows

## `kasadaBypass` (type: `boolean`):

Enable additional optimizations for sites with client-side verification

## `screenshot` (type: `boolean`):

Capture a screenshot of the page

## `screenshotUpload` (type: `boolean`):

Upload screenshot to storage and return URL

## `video` (type: `boolean`):

Record browser session as video

## `cssSelector` (type: `string`):

Extract content matching this CSS selector

## `innerText` (type: `boolean`):

Include the inner text content of the page

## `includeImages` (type: `boolean`):

Include all image URLs found on the page

## `includeLinks` (type: `boolean`):

Include all link URLs found on the page

## `customHeaders` (type: `object`):

Custom HTTP headers to send with the request

## `cookiejar` (type: `array`):

Array of cookies to set before making the request

## `localStorage` (type: `object`):

Local storage data to set in the browser

## `interceptFetchRequest` (type: `string`):

URL pattern(s) to intercept and return (comma-separated for multiple)

## `fullPageLoad` (type: `boolean`):

Wait for full page load including all resources

## `timeout` (type: `integer`):

Request timeout in milliseconds

## Actor input object example

```json
{
  "url": "https://httpbin.rs/get",
  "cmd": "request.get",
  "requestType": "browser",
  "premiumProxy": false,
  "mobileProxy": false,
  "browserActions": [],
  "automaticallySolveCaptchas": false,
  "cloudflareBypass": false,
  "datadomeBypass": false,
  "kasadaBypass": false,
  "screenshot": false,
  "screenshotUpload": false,
  "video": false,
  "innerText": false,
  "includeImages": false,
  "includeLinks": false,
  "fullPageLoad": false,
  "timeout": 300000
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "url": "https://httpbin.rs/get",
    "browserActions": []
};

// Run the Actor and wait for it to finish
const run = await client.actor("dormic/apify-scrappey").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "url": "https://httpbin.rs/get",
    "browserActions": [],
}

# Run the Actor and wait for it to finish
run = client.actor("dormic/apify-scrappey").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "url": "https://httpbin.rs/get",
  "browserActions": []
}' |
apify call dormic/apify-scrappey --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,dormic/apify-scrappey"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/cLo1vfjaaMlAzaQao/builds/mI5zqpUKTsi4lGKxU/openapi.json