# apify-scrapers Live Website Scraper (`ai_category/apify-scrapers`) Actor

apify-scrapers collects live public records from linkedin.com.

- **URL**: https://apify.com/ai_category/apify-scrapers.md
- **Developed by:** [AI Category](https://apify.com/ai_category) (community)
- **Categories:** Lead generation, Automation, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.99 / 1,000 scraped results

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### What does apify-scrapers do?

apify-scrapers uses the shared live-site runtime to search for relevant public records on [linkedin.com](https://linkedin.com). It builds requests from a target-site URL or search query and records only values received from the live page or public response. The output URLs retain the target site as their source. No sample or static records are inserted.

**Current setup:** ready to request live pages from [linkedin.com](https://linkedin.com). The prefilled query is **software engineer**. Results depend on the current response from that website and may be empty or blocked by the site.

### Why use apify-scrapers?

Use this actor when you want a repeatable local run for search results on the actor's configured website. The input keeps the actor name, target domain, target type, and query together so the request can be checked before it runs. The crawler can follow discovered item links for detail enrichment within the configured limit. Each record keeps its source URL and collection time for review.

The run depends on the target site's current availability and public response. Some sites limit automated traffic, require a sign-in, or return a verification page. In those cases the run reports a blocked status or no results instead of inventing records.

### What data can apify-scrapers extract?

The exact fields depend on what the target site exposes for the selected search results. Common output fields are:

| Field | Description |
| --- | --- |
| `actorName` | The actor identity recorded for each live item. |
| `sourceUrl` | The live target search or listing URL that produced the item. |
| `itemUrl` | The matching target-site item URL when one is exposed. |
| `title` | The title or name parsed from the live page or response. |
| `description / text` | Visible summary or content returned by the target. |
| `searchQuery` | The query used for this item, when the run used search. |
| `scrapedAt` | The time the live item was collected. |
| `rawMetadata` | Additional source fields retained when available. |
| `text` | A target-specific value, included when exposed by the live search results source. |
| `imageUrl` | A target-specific value, included when exposed by the live search results source. |
| `category` | A target-specific value, included when exposed by the live search results source. |

Type-specific values are retained when they appear in the live response; unavailable fields are left empty or omitted.

### How to scrape linkedin.com

1. Open the input settings and confirm the target domain is linkedin.com.
2. Supply a direct URL from that website in `startUrls`, or edit `searchQueries` to the exact term you want to search.
3. Set `maxItems` to control the number of records. Production/schema defaults request 20 valid unique items (up to 50 when explicitly changed). Local input.json explicitly requests 10 for a smoke test.
4. Run `npm start` from this actor folder.
5. Review the dataset and the run summary under `storage/`; each output item includes its source URL and timestamp.

### Cost and run size

A local run uses your machine, network connection, and the configured request limits. This project does not add a per-result fee. If you run it on a hosted Actor platform, charges depend on the current account plan and the run's actual compute and proxy settings. Check the platform's current pricing before increasing the item count or frequency. Keeping `maxItems`, `maxPages`, and `detailPages` low is useful for a small validation run.

### Input

- **Local input file:** `input.json` is used for local `npm start`; the batch runner mirrors it to `storage/key_value_stores/default/INPUT.json`.
- **Actor:** `apify-scrapers` (folder: `apify-scrapers`).
- **Target type:** `search`.
- **Target domain:** `linkedin.com`.
- **Prefilled query:** `software engineer`.
- **Direct URLs:** add one or more live URLs from the target website to `startUrls`.
- **Local smoke-test limit:** `10`; **production default:** `20`; **maximum allowed by the schema:** `50`.
- **Detail enrichment:** `detailPages` limits how many discovered item pages are opened.

See the input tab for the full configuration. Use a URL and query that belong to the actor's target website; results are filtered to the requested target and query.

### Output

The dataset contains live records collected during the current run. Fields are populated only when the target page or response provides them, so two runs can return different items and optional fields. The actor does not seed the dataset with fixed examples. Download or inspect the dataset in JSON, CSV, or another format supported by the runner.

### Proxy and retry configuration

Production defaults use the configured public API directly when applicable, automatic Apify Proxy for other websites, or GOOGLE_SERP for supported Google search endpoints. The input tab shows the exact default. Set RESIDENTIAL only when appropriate and permitted by your account/budget. Requests rotate proxy sessions on retry, respect per-host request delays, and use bounded exponential backoff. A failed proxy access check is reported as a failure, not silently bypassed. CAPTCHA and login walls remain access limitations; no fake records replace blocked responses.

### Tips and troubleshooting

- Use a direct results URL when the website's search route needs extra filters or location settings.
- If the run reports no results, check that the query matches the target site's own search syntax and that the URL opens publicly.
- If the run reports blocked, review the target site's response and provide a permitted public URL. The crawler will not treat a verification page as a result.
- Keep the detail page limit low when validating a new query, then increase it if the target exposes useful item pages.

### Responsible use and support

Use this actor only with pages you are permitted to access and in line with the target website's terms. It is designed for public information and does not request private account data. Public pages can still contain personal information; collect and retain only what you have a legitimate reason to use, and follow applicable privacy rules. For issues, include the actor name, target domain, query or URL, and run summary when reporting a problem.

# Changelog

This Actor's version history is a separate document: https://apify.com/ai_category/apify-scrapers/changelog.md

# Actor input Schema

## `actorName` (type: `string`):

Actor identity used in output records.

## `targetType` (type: `string`):

Expected live data type for apify-scrapers.

## `sourceDomain` (type: `string`):

Live target website domain configured for this actor.

## `startUrls` (type: `array`):

Direct page, API or search-result URLs from the target website.

## `searchQueries` (type: `array`):

Search terms for apify-scrapers on the target website.

## `blocked` (type: `boolean`):

True when no safe actor-specific live source can be used.

## `blockedReason` (type: `string`):

Why this actor is blocked or needs a dedicated implementation.

## `maxItems` (type: `integer`):

Valid unique results requested per run. Local smoke-test prefill is 10; production default is 20 (maximum 50).

## `maxPages` (type: `integer`):

Maximum live pages requested in one run.

## `detailPages` (type: `integer`):

Maximum discovered item/detail pages to enrich after the listing or API response.

## `proxyConfiguration` (type: `object`):

Leave disabled to start direct. When a target blocks the run, autoProxy retries that host through Apify datacenter proxy, then UNBLOCKER; Google search hosts use GOOGLE_SERP first.

## `autoProxy` (type: `boolean`):

Direct first; on HTTP 401/403/429 or a CAPTCHA page retry via Apify Proxy (GOOGLE_SERP for Google, otherwise datacenter then UNBLOCKER).

## `requestDelayMillis` (type: `integer`):

Minimum request spacing per hostname in milliseconds; bounds concurrency pressure.

## `maxRetries` (type: `integer`):

Retries temporary HTTP/network failures with bounded exponential backoff and proxy session rotation. This excludes the initial attempt.

## Actor input object example

```json
{
  "actorName": "apify-scrapers",
  "targetType": "search",
  "sourceDomain": "linkedin.com",
  "startUrls": [],
  "searchQueries": [
    "software engineer"
  ],
  "blocked": false,
  "blockedReason": "",
  "maxItems": 10,
  "maxPages": 3,
  "detailPages": 10,
  "proxyConfiguration": {
    "useApifyProxy": false
  },
  "autoProxy": true,
  "requestDelayMillis": 500,
  "maxRetries": 2
}
```

# Actor output Schema

## `dataset` (type: `string`):

No description

## `runSummary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "sourceDomain": "linkedin.com",
    "startUrls": [],
    "searchQueries": [
        "software engineer"
    ],
    "maxItems": 10,
    "maxPages": 3,
    "detailPages": 10,
    "proxyConfiguration": {
        "useApifyProxy": false
    },
    "autoProxy": true,
    "requestDelayMillis": 0,
    "maxRetries": 0
};

// Run the Actor and wait for it to finish
const run = await client.actor("ai_category/apify-scrapers").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "sourceDomain": "linkedin.com",
    "startUrls": [],
    "searchQueries": ["software engineer"],
    "maxItems": 10,
    "maxPages": 3,
    "detailPages": 10,
    "proxyConfiguration": { "useApifyProxy": False },
    "autoProxy": True,
    "requestDelayMillis": 0,
    "maxRetries": 0,
}

# Run the Actor and wait for it to finish
run = client.actor("ai_category/apify-scrapers").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "sourceDomain": "linkedin.com",
  "startUrls": [],
  "searchQueries": [
    "software engineer"
  ],
  "maxItems": 10,
  "maxPages": 3,
  "detailPages": 10,
  "proxyConfiguration": {
    "useApifyProxy": false
  },
  "autoProxy": true,
  "requestDelayMillis": 0,
  "maxRetries": 0
}' |
apify call ai_category/apify-scrapers --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,ai_category/apify-scrapers"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/rb51pAVQ2gYlX3GN6/builds/d6DvodZg9Y7TPaDPz/openapi.json