Website Data Scraper

Read a list of web pages into a structured content file. Keep titles, page text, response status, metadata, canonical links, and source URLs together for research, migration, or search work.

What it returns

Each row keeps the the web source record beside the fields needed to use it. The opening set is requestedUrl, finalUrl, responseStatus, title, metaDescription, canonical, links, jsonLd, jsonLdTypes, jsonLdErrors, text, and sourceHeaders. The complete schema is declared before the run, and dataset views keep related fields together without changing the underlying row.

Input

Website Data Scraper accepts source URLs. Run controls stay in the same form.

Field	What it controls	Starting value
startUrls	Paste exact URLs from the open web, one per line.	["https://www.iana.org/help/example-domains"]
maxItems	Stop after this many dataset rows.	20
maxSeconds	Stop after this many seconds and keep completed rows.	120

Example input

{
  "startUrls": [
    "https://www.iana.org/help/example-domains"
  ],
  "maxItems": 3,
  "maxSeconds": 120
}

No field is required. Start with the filled example, then replace only the target values needed for the job. Run controls can stay at their starting values for the first collection.

Dataset fields

Field	Type
requestedUrl	string
finalUrl	string
responseStatus	integer
title	string
metaDescription	string
canonical	string
links	array:string
jsonLd	array:object
jsonLdTypes	array:string
jsonLdErrors	array:object
text	string
sourceHeaders	object
bodySha256	string
contentType	string
wireBytes	integer
decodedBytes	integer
route	string
scrapedAt	string

Dataset views

Views are working surfaces for review and export. They select and order fields while leaving the stored row unchanged.

View	Opening fields
overview	title, metaDescription, text, bodySha256, contentType, scrapedAt, requestedUrl, and finalUrl
identity	title
content	title, requestedUrl, metaDescription, text, bodySha256, and contentType
timing	title, requestedUrl, and scrapedAt
provenance	requestedUrl, finalUrl, sourceHeaders, and route
details	title, requestedUrl, responseStatus, canonical, links, jsonLd, jsonLdTypes, and jsonLdErrors

Output and exports

Output	Type	Destination
results	string	{{links.apiDefaultDatasetUrl}}/items
json	string	{{links.apiDefaultDatasetUrl}}/items?clean=true&format=json
csv	string	{{links.apiDefaultDatasetUrl}}/items?clean=true&format=csv
excel	string	{{links.apiDefaultDatasetUrl}}/items?clean=true&format=xlsx
jsonl	string	{{links.apiDefaultDatasetUrl}}/items?clean=true&format=jsonl

Completed rows are available in the Apify dataset as JSON, CSV, Excel, and JSONL exports. The run output also carries the declared links above for API clients and automations.

Pricing

$1 per 1,000 jobs on the Free plan. Other Apify plans use the rates shown in the Pricing tab.

Console, API, schedules, and MCP

Runs can begin in Apify Console, from a saved task, or through the Actor API. A schedule can reuse the same input, and a run-finished webhook can pass the dataset or run ID to the next system.

POST https://api.apify.com/v2/acts/XPS2cILOF68CbPrj8/runs
GET  https://api.apify.com/v2/datasets/{datasetId}/items

For MCP selection, use Website Data Scraper. Its machine entry carries the same description, input field names, no-required-field contract, output types, dataset fields, views, and pricing facts as this document.

Saved tasks

Twenty saved-task products cover distinct lookup, comparison, research, operations, automation, and export jobs:

• Website Data headers full record: buyer-job; opens overview.
• Website Data identity titles description set: buyer-job; opens identity.
• Website Data titles and meta description set: buyer-job; opens content.
• Website Data timing collection times publishing file: buyer-job; opens timing.
• Website Data source requested URLs origin check: buyer-job; opens provenance.
• Website Data links json ld detail: buyer-job; opens details.
• Website Data headers and meta description detail handoff: buyer-job; opens overview.
• Website Data titles identity reading list: buyer-job; opens identity.
• Website Data content meta description reading list: buyer-job; opens content.
• Website Data collection times timing time window: buyer-job; opens timing.
• Website Data final URL source result lineage: buyer-job; opens provenance.
• Website Data json ld detail handoff: buyer-job; opens details.
• Website Data core headers detail file: buyer-job; opens overview.
• Website Data titles text identity: buyer-job; opens identity.
• Website Data text content text file: buyer-job; opens content.
• Website Data collection times titles timing: buyer-job; opens timing.
• Website Data routes requested URLs source: buyer-job; opens provenance.
• Website Data json ld errors and download sizes detail file: buyer-job; opens details.
• Website Data headers core record view: buyer-job; opens overview.
• Website Data title and copy review: buyer-job; opens identity.

Integrations

Use the dataset API from any HTTP client, export rows to a spreadsheet, or send the run ID through an Apify webhook. Saved tasks give schedules and automation tools a stable input without changing the Actor contract.

Related products

• Website Profile Scraper
• Website Email Scraper
• Website Lead Scraper
• Web Event Data Scraper
• Website Contact Scraper

When a run needs attention

• No rows: Open the target in a browser, check spelling and source visibility, then retry the saved example before widening the input.
• A field is empty: Check the field beside its source URL. A missing source value stays empty instead of being replaced with a guess.
• A target fails: Keep successful targets in the dataset, then retry only the affected input.
• An automation cannot find results: Read the dataset ID from the run and request its items endpoint directly.

FAQ

What do I get back from one run?

One row per general with 18 declared fields, opening on requestedUrl, finalUrl, responseStatus and title. The schema is published before the run, so you know the shape before you spend anything.

Do I need a website account or login?

No. The run works from the website sources you supply in the input. Nothing is posted, changed or accessed on your behalf.

What does a run cost?

The current rate is shown on the Pricing tab and is charged per row you receive, so a run that finds nothing costs close to nothing. Cap the run with the item limit when you want a predictable ceiling.

Can I try it before committing budget?

Yes. Cap the run with the item limit in the input and inspect the first rows. The cap is enforced before charging, so a trial run stays a trial.

What do I put in the input?

The staged input is already usable: startUrls, maxItems and maxSeconds. Replace the staged target with your own list when you are ready to run for real.

Are any fields required?

No field is required. Every input carries a working default, so the Actor can be started as-is and refined afterwards.

Can I schedule this to run on its own?

Yes. Save the input as an Apify task and attach a schedule. Keep separate tasks when different teams need different targets or delivery paths.

Do I need to configure proxies?

No. Network access is handled inside the Actor and needs no proxy configuration from you.

How fresh is the data?

Every row is collected during the run you start, not served from a cache. Re-run the same input whenever you need the current state of a website general.

Can I use the results commercially?

The Actor collects publicly accessible website information. You remain responsible for how you use it, including any privacy or contractual obligations that apply to your business.

How do I compare two runs?

Keep requestedUrl and finalUrl as your join key and diff the exports. The identity fields stay stable across runs, which is what makes a comparison meaningful.

What happens if a source fails mid-run?

The run continues through the remaining sources and finishes with what it collected. Partial results are still written to the dataset rather than discarded.

Can I limit how long a run takes?

Yes. The maxItems input caps the run. Use it when you need a predictable cost and a predictable finish time.

How do I report a problem?

Open an issue on the Actor with the run ID, the input you used and the field or row that needs attention. The run ID lets the exact execution be inspected.

How is this different from Website Profile Scraper?

Website Data Scraper answers one job: Collect the records behind a list of web pages.. Website Profile Scraper covers a different question on the same platform. Run both when you need both sides.

What is responseStatus for?

It records how the row was resolved, so you can filter to the rows you trust instead of treating every row as equally certain.

How do I read the output without scrolling through JSON?

Open the overview view on the Output tab. 6 views ship with the Actor (overview, identity, content, timing, provenance and details), each grouping the fields that belong to one question.

How do I get the data into my own tools?

Export the dataset as JSON, CSV, Excel or XML, call the dataset API directly, or attach a run-finished webhook and collect the dataset reference as soon as the run ends.

Can an agent or LLM call this?

Yes. Website Data Scraper is exposed over MCP with the same description, no-required-field input contract and output types shown here, so an agent can select and call it without a human in the loop.

Why is a value empty on some rows?

website does not expose every field on every general. An absent value stays empty rather than being filled with a guess, so a row never invents a fact it did not receive.

A run returned fewer rows than I expected. Why?

The usual causes are a narrow source list, an item cap still set low, or a source that genuinely holds less than expected. Widen the input or raise the cap and run again.

Support

Use the Actor issue form for product questions, broken source routes, schema mismatches, and feedback. Include the smallest input that reproduces the problem. That is enough to locate the run and its dataset without sharing an entire working list.

Use this Actor only for data you are allowed to collect. Follow source terms, privacy law, and your own retention policy.