# Website Data Scraper (`reapx/website-data-scraper`) Actor

Read a list of web pages into a structured content file. Keep titles, page text, response status, metadata, canonical links, and source URLs together for research, migration, or search work.

- **URL**: https://apify.com/reapx/website-data-scraper.md
- **Developed by:** [ReapX](https://apify.com/reapx) (community)
- **Categories:** SEO tools, Developer tools, Lead generation
- **Stats:** 1 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.85 / 1,000 jobs

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Website Data Scraper

Read a list of web pages into a structured content file. Keep titles, page text, response status, metadata, canonical links, and source URLs together for research, migration, or search work.

![Website Data Scraper interface](https://reapx.dev/assets/products/website-data-scraper/readme.png)

![Website Data Scraper interface](https://reapx.dev/assets/products/website-data-scraper/readme.png?v=20260826)

### What it returns

Each row keeps the the web source record beside the fields needed to use it. The opening set is `requestedUrl`, `finalUrl`, `responseStatus`, `title`, `metaDescription`, `canonical`, `links`, `jsonLd`, `jsonLdTypes`, `jsonLdErrors`, `text`, and `sourceHeaders`. The complete schema is declared before the run, and dataset views keep related fields together without changing the underlying row.

### Input

Website Data Scraper accepts source URLs. Run controls stay in the same form.

| Field | What it controls | Starting value |
| --- | --- | --- |
| `startUrls` | Paste exact URLs from the open web, one per line. | `["https://www.iana.org/help/example-domains"]` |
| `maxItems` | Stop after this many dataset rows. | `20` |
| `maxSeconds` | Stop after this many seconds and keep completed rows. | `120` |

#### Example input

```json
{
  "startUrls": [
    "https://www.iana.org/help/example-domains"
  ],
  "maxItems": 3,
  "maxSeconds": 120
}
```

No field is required. Start with the filled example, then replace only the target values needed for the job. Run controls can stay at their starting values for the first collection.

### Dataset fields

| Field | Type |
| --- | --- |
| `requestedUrl` | `string` |
| `finalUrl` | `string` |
| `responseStatus` | `integer` |
| `title` | `string` |
| `metaDescription` | `string` |
| `canonical` | `string` |
| `links` | `array:string` |
| `jsonLd` | `array:object` |
| `jsonLdTypes` | `array:string` |
| `jsonLdErrors` | `array:object` |
| `text` | `string` |
| `sourceHeaders` | `object` |
| `bodySha256` | `string` |
| `contentType` | `string` |
| `wireBytes` | `integer` |
| `decodedBytes` | `integer` |
| `route` | `string` |
| `scrapedAt` | `string` |

### Dataset views

Views are working surfaces for review and export. They select and order fields while leaving the stored row unchanged.

| View | Opening fields |
| --- | --- |
| `overview` | `title`, `metaDescription`, `text`, `bodySha256`, `contentType`, `scrapedAt`, `requestedUrl`, and `finalUrl` |
| `identity` | `title` |
| `content` | `title`, `requestedUrl`, `metaDescription`, `text`, `bodySha256`, and `contentType` |
| `timing` | `title`, `requestedUrl`, and `scrapedAt` |
| `provenance` | `requestedUrl`, `finalUrl`, `sourceHeaders`, and `route` |
| `details` | `title`, `requestedUrl`, `responseStatus`, `canonical`, `links`, `jsonLd`, `jsonLdTypes`, and `jsonLdErrors` |

![Website Data Scraper input and result demonstration](https://reapx.dev/assets/products/website-data-scraper/demo.webp)

![Website Data Scraper input and result demonstration](https://reapx.dev/assets/products/website-data-scraper/demo.webp?v=20260826)

### Output and exports

| Output | Type | Destination |
| --- | --- | --- |
| `results` | `string` | `{{links.apiDefaultDatasetUrl}}/items` |
| `json` | `string` | `{{links.apiDefaultDatasetUrl}}/items?clean=true&format=json` |
| `csv` | `string` | `{{links.apiDefaultDatasetUrl}}/items?clean=true&format=csv` |
| `excel` | `string` | `{{links.apiDefaultDatasetUrl}}/items?clean=true&format=xlsx` |
| `jsonl` | `string` | `{{links.apiDefaultDatasetUrl}}/items?clean=true&format=jsonl` |

Completed rows are available in the Apify dataset as JSON, CSV, Excel, and JSONL exports. The run output also carries the declared links above for API clients and automations.

### Pricing

$1 per 1,000 jobs on the Free plan. Other Apify plans use the rates shown in the Pricing tab.

### Console, API, schedules, and MCP

Runs can begin in Apify Console, from a saved task, or through the Actor API. A schedule can reuse the same input, and a run-finished webhook can pass the dataset or run ID to the next system.

```text
POST https://api.apify.com/v2/acts/XPS2cILOF68CbPrj8/runs
GET  https://api.apify.com/v2/datasets/{datasetId}/items
```

For MCP selection, use **Website Data Scraper**. Its machine entry carries the same description, input field names, no-required-field contract, output types, dataset fields, views, and pricing facts as this document.

### Saved tasks

Twenty saved-task products cover distinct lookup, comparison, research, operations, automation, and export jobs:

- **Website Data headers full record**: buyer-job; opens `overview`.
- **Website Data identity titles description set**: buyer-job; opens `identity`.
- **Website Data titles and meta description set**: buyer-job; opens `content`.
- **Website Data timing collection times publishing file**: buyer-job; opens `timing`.
- **Website Data source requested URLs origin check**: buyer-job; opens `provenance`.
- **Website Data links json ld detail**: buyer-job; opens `details`.
- **Website Data headers and meta description detail handoff**: buyer-job; opens `overview`.
- **Website Data titles identity reading list**: buyer-job; opens `identity`.
- **Website Data content meta description reading list**: buyer-job; opens `content`.
- **Website Data collection times timing time window**: buyer-job; opens `timing`.
- **Website Data final URL source result lineage**: buyer-job; opens `provenance`.
- **Website Data json ld detail handoff**: buyer-job; opens `details`.
- **Website Data core headers detail file**: buyer-job; opens `overview`.
- **Website Data titles text identity**: buyer-job; opens `identity`.
- **Website Data text content text file**: buyer-job; opens `content`.
- **Website Data collection times titles timing**: buyer-job; opens `timing`.
- **Website Data routes requested URLs source**: buyer-job; opens `provenance`.
- **Website Data json ld errors and download sizes detail file**: buyer-job; opens `details`.
- **Website Data headers core record view**: buyer-job; opens `overview`.
- **Website Data title and copy review**: buyer-job; opens `identity`.

### Integrations

Use the dataset API from any HTTP client, export rows to a spreadsheet, or send the run ID through an Apify webhook. Saved tasks give schedules and automation tools a stable input without changing the Actor contract.

### Related products

- [Website Profile Scraper](https://apify.com/reapx/website-profile-scraper)
- [Website Email Scraper](https://apify.com/reapx/website-email-scraper)
- [Website Lead Scraper](https://apify.com/reapx/website-lead-scraper)
- [Web Event Data Scraper](https://apify.com/reapx/web-event-data-scraper)
- [Website Contact Scraper](https://apify.com/reapx/website-contact-scraper)

### When a run needs attention

- **No rows:** Open the target in a browser, check spelling and source visibility, then retry the saved example before widening the input.
- **A field is empty:** Check the field beside its source URL. A missing source value stays empty instead of being replaced with a guess.
- **A target fails:** Keep successful targets in the dataset, then retry only the affected input.
- **An automation cannot find results:** Read the dataset ID from the run and request its items endpoint directly.

### FAQ

#### What do I get back from one run?

One row per general with 18 declared fields, opening on requestedUrl, finalUrl, responseStatus and title. The schema is published before the run, so you know the shape before you spend anything.

#### Do I need a website account or login?

No. The run works from the website sources you supply in the input. Nothing is posted, changed or accessed on your behalf.

#### What does a run cost?

The current rate is shown on the Pricing tab and is charged per row you receive, so a run that finds nothing costs close to nothing. Cap the run with the item limit when you want a predictable ceiling.

#### Can I try it before committing budget?

Yes. Cap the run with the item limit in the input and inspect the first rows. The cap is enforced before charging, so a trial run stays a trial.

#### What do I put in the input?

The staged input is already usable: startUrls, maxItems and maxSeconds. Replace the staged target with your own list when you are ready to run for real.

#### Are any fields required?

No field is required. Every input carries a working default, so the Actor can be started as-is and refined afterwards.

#### Can I schedule this to run on its own?

Yes. Save the input as an Apify task and attach a schedule. Keep separate tasks when different teams need different targets or delivery paths.

#### Do I need to configure proxies?

No. Network access is handled inside the Actor and needs no proxy configuration from you.

#### How fresh is the data?

Every row is collected during the run you start, not served from a cache. Re-run the same input whenever you need the current state of a website general.

#### Can I use the results commercially?

The Actor collects publicly accessible website information. You remain responsible for how you use it, including any privacy or contractual obligations that apply to your business.

#### How do I compare two runs?

Keep requestedUrl and finalUrl as your join key and diff the exports. The identity fields stay stable across runs, which is what makes a comparison meaningful.

#### What happens if a source fails mid-run?

The run continues through the remaining sources and finishes with what it collected. Partial results are still written to the dataset rather than discarded.

#### Can I limit how long a run takes?

Yes. The maxItems input caps the run. Use it when you need a predictable cost and a predictable finish time.

#### How do I report a problem?

Open an issue on the Actor with the run ID, the input you used and the field or row that needs attention. The run ID lets the exact execution be inspected.

#### How is this different from Website Profile Scraper?

Website Data Scraper answers one job: Collect the records behind a list of web pages.. Website Profile Scraper covers a different question on the same platform. Run both when you need both sides.

#### What is `responseStatus` for?

It records how the row was resolved, so you can filter to the rows you trust instead of treating every row as equally certain.

#### How do I read the output without scrolling through JSON?

Open the overview view on the Output tab. 6 views ship with the Actor (overview, identity, content, timing, provenance and details), each grouping the fields that belong to one question.

#### How do I get the data into my own tools?

Export the dataset as JSON, CSV, Excel or XML, call the dataset API directly, or attach a run-finished webhook and collect the dataset reference as soon as the run ends.

#### Can an agent or LLM call this?

Yes. Website Data Scraper is exposed over MCP with the same description, no-required-field input contract and output types shown here, so an agent can select and call it without a human in the loop.

#### Why is a value empty on some rows?

website does not expose every field on every general. An absent value stays empty rather than being filled with a guess, so a row never invents a fact it did not receive.

#### A run returned fewer rows than I expected. Why?

The usual causes are a narrow source list, an item cap still set low, or a source that genuinely holds less than expected. Widen the input or raise the cap and run again.

### Support

Use the Actor issue form for product questions, broken source routes, schema mismatches, and feedback. Include the smallest input that reproduces the problem. That is enough to locate the run and its dataset without sharing an entire working list.

Use this Actor only for data you are allowed to collect. Follow source terms, privacy law, and your own retention policy.

# Actor input Schema

## `startUrls` (type: `array`):

Paste exact URLs from the open web, one per line.

## `maxItems` (type: `integer`):

Stop after this many dataset rows.

## `maxSeconds` (type: `integer`):

Stop after this many seconds and keep completed rows.

## Actor input object example

```json
{
  "startUrls": [
    "https://www.iana.org/help/example-domains"
  ],
  "maxItems": 20,
  "maxSeconds": 120
}
```

# Actor output Schema

## `results` (type: `string`):

Open all returned Website Data Scraper rows with 18 declared fields and 6 working views.

## `json` (type: `string`):

Retrieve clean Website Data Scraper records for API, MCP, or webhook use.

## `csv` (type: `string`):

Download the Website Data Scraper table for spreadsheets and data tools.

## `excel` (type: `string`):

Open the Website Data Scraper rows as an Excel workbook.

## `jsonl` (type: `string`):

Stream one clean Website Data Scraper record per line for downstream processing.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://www.iana.org/help/example-domains"
    ],
    "maxItems": 20,
    "maxSeconds": 120
};

// Run the Actor and wait for it to finish
const run = await client.actor("reapx/website-data-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://www.iana.org/help/example-domains"],
    "maxItems": 20,
    "maxSeconds": 120,
}

# Run the Actor and wait for it to finish
run = client.actor("reapx/website-data-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://www.iana.org/help/example-domains"
  ],
  "maxItems": 20,
  "maxSeconds": 120
}' |
apify call reapx/website-data-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,reapx/website-data-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/XPS2cILOF68CbPrj8/builds/j1eEngGHymHTadegy/openapi.json