# Intelligent Website Scraper (`scrapers-hub/intelligent-website-scraper`) Actor

Intelligent Website Scraper crawls a site to a set depth and returns cleaned, task-shaped content plus raw HTML, title, metadata and timestamps. 🧠 Built for RAG ingestion, LLM training data, knowledge bases and content migration.

- **URL**: https://apify.com/scrapers-hub/intelligent-website-scraper.md
- **Developed by:** [Scrapers Hub](https://apify.com/scrapers-hub) (community)
- **Categories:** AI, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.99 / 1,000 results

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## 🕸️ Intelligent Website Scraper – Website Content Extraction & Page Text Crawler

The **Intelligent Website Scraper** crawls any website with a real headless browser, strips away scripts, styles and media, and returns the clean readable text of every page it visits alongside a task-labelled content excerpt and structural page metrics. Give it a list of start URLs, choose a task type, and decide whether to follow internal links — the scraper handles JavaScript rendering, same-domain link discovery and text normalisation, and pushes one tidy record per page into an Apify dataset.

This website scraper is deliberately straightforward about what it does. It uses Playwright through Crawlee, so pages built with React, Vue, Angular or any other client-side framework render fully before extraction, which is where simple HTTP-based scrapers fail. The text you get back is the page's visible content with `<script>`, `<style>`, `<noscript>`, `<iframe>`, `<img>`, `<svg>` and `<video>` elements removed, joined into a single normalised string. Alongside it you get a word count, a link count and an image count for every page — the kind of structural signal that makes content audits, site inventories and page-quality checks possible at scale.

***

### 📊 What Data Can You Extract with This Website Scraper?

Every page the crawler visits produces one dataset row. The fields fall into five natural groups.

| Category | Fields | What You Get |
|---|---|---|
| 🔗 Page identity | `url`, `title` | The exact page URL crawled and the rendered document title |
| 🏷️ Task labelling | `taskType` | Which processing mode produced this record — summarise, products, services or FAQs |
| ✂️ Processed excerpt | `processedContent` | A task-labelled excerpt of the page's leading content, prefixed with the extraction mode |
| 📃 Page text | `rawContent` | The cleaned visible text of the page, captured from the fully rendered DOM |
| 📐 Structural metrics | `metadata` | An object with `wordCount`, `linksFound` and `imagesFound` for the page |
| 🕒 Run metadata | `scrapedAt` | ISO-8601 UTC timestamp recording when the page was crawled |

The `metadata` object is the quietly valuable part. `wordCount` reflects the full text length of the page before truncation, so you can identify thin content even when the stored excerpt is short; `linksFound` and `imagesFound` count every anchor and image element in the raw HTML, which makes navigation-heavy hub pages and media-heavy landing pages immediately distinguishable from ordinary article pages.

***

### 🌟 Key Features of the Intelligent Website Scraper

| Feature | Description |
|---|---|
| 🌐 Full JavaScript rendering | Uses Playwright through Crawlee, so single-page applications and client-rendered content are captured after the page has hydrated |
| 🔗 Same-domain link following | Enable `followInternalLinks` and the crawler enqueues internal links automatically, expanding from your start URLs across the site |
| 🎯 Four task modes | `summarize`, `extractProducts`, `extractServices` and `extractFAQs` label and frame the extracted excerpt for your downstream workflow |
| 🧹 Clean text extraction | Scripts, styles, noscript blocks, iframes, images, SVGs and video elements are removed before text is collected |
| 📐 Per-page structural metrics | Word count, anchor count and image count are computed for every page and returned in the `metadata` object |
| 📋 Multiple start URLs | Supply an array of entry points to crawl several sections, subdomains paths or landing pages in a single run |
| 🛡️ Built-in crawl ceiling | A maximum of 50 requests per crawl keeps runs bounded and predictable rather than expanding without limit |
| ⏱️ Timestamped records | Every row carries an ISO-8601 UTC `scrapedAt` value, so repeated crawls can be compared chronologically |
| 📤 Incremental dataset pushes | Records are written as each page finishes, so partial results are available while a crawl is still running |

***

### 🚀 Why Choose This Intelligent Website Scraper?

**It renders the page properly.** A large share of the modern web serves an almost empty HTML shell and builds the content in the browser. Fetching that HTML with a plain HTTP request returns nothing useful. This website scraper drives a real Chromium instance through Playwright and reads the DOM after rendering, so what you extract is what a visitor actually sees.

**Text and structure in one pass.** Most scrapers give you either the content or the page statistics, never both. Every record here pairs the cleaned `rawContent` with a `metadata` object carrying `wordCount`, `linksFound` and `imagesFound`, which means a single crawl supports both content analysis and technical site auditing.

**Predictable, bounded crawls.** The crawler is capped at 50 requests per run. That is a deliberate design decision, not a limitation to work around: it keeps compute usage and run times foreseeable, prevents an accidental `followInternalLinks` setting from crawling a hundred-thousand-page site, and makes cost estimation simple.

**Honest, deterministic extraction.** The task modes apply straightforward, rule-based content selection rather than a language model. That means results are reproducible, there is no model latency or token cost, and nothing is ever hallucinated — the text in your dataset came verbatim from the page.

***

### 📥 Input

The Intelligent Website Scraper takes a list of start URLs plus three crawl and processing controls. Only `startUrls` is required.

```json
{
  "startUrls": [
    { "url": "https://example.com" }
  ],
  "taskType": "summarize",
  "maxDepth": 1,
  "followInternalLinks": false
}
```

#### 🔧 Intelligent Website Scraper Input Fields

| Field | Type | Required | Default | Description |
|---|---|---|---|---|
| `startUrls` | array | ✅ Yes | — | List of URLs to scrape and process. Each entry is an object with a `url` key, in Apify's request-list format. |
| `taskType` | string | ❌ No | `"summarize"` | Type of content processing task to perform. One of `summarize` (summarise entire site content), `extractProducts` (extract product information), `extractServices` (extract service offerings) or `extractFAQs` (extract FAQ content). |
| `maxDepth` | integer | ❌ No | `1` | Maximum depth for internal link following — `0` for only the start URL, `1` for the start URL and its direct links, and so on. |
| `followInternalLinks` | boolean | ❌ No | `false` | Whether to follow internal links on the same domain. |

#### 💡 Input Examples

**Single-page content capture**

```json
{
  "startUrls": [{ "url": "https://example.com/about" }],
  "taskType": "summarize",
  "followInternalLinks": false
}
```

**Site section crawl for service offerings**

```json
{
  "startUrls": [{ "url": "https://example.com/services" }],
  "taskType": "extractServices",
  "maxDepth": 1,
  "followInternalLinks": true
}
```

**Multi-entry product catalogue sweep**

```json
{
  "startUrls": [
    { "url": "https://example.com/shop/mens" },
    { "url": "https://example.com/shop/womens" },
    { "url": "https://example.com/shop/accessories" }
  ],
  "taskType": "extractProducts",
  "maxDepth": 1,
  "followInternalLinks": true
}
```

***

### 📤 Output

Each dataset item represents one crawled page.

```json
{
  "url": "https://example.com/services",
  "title": "Our Services | Example Ltd",
  "taskType": "extractServices",
  "processedContent": "Services found in text (manual extraction): Our Services Consulting Implementation Managed Support We work with mid-market operations teams to design, deploy and run...",
  "rawContent": "Our Services Consulting Implementation Managed Support We work with mid-market operations teams to design, deploy and run the systems that keep their business moving. Every engagement starts with a discovery workshop...",
  "scrapedAt": "2026-08-17T09:12:44.183920Z",
  "metadata": {
    "wordCount": 1284,
    "linksFound": 47,
    "imagesFound": 12
  }
}
```

#### 🧾 Intelligent Website Scraper Output Fields

| Field | Type | Description |
|---|---|---|
| `url` | string | null | Canonical URL of the scraped item — the page that was crawled. |
| `title` | string | null | Title of the item, taken from the rendered document title. |
| `taskType` | string | null | Task type of the item — the processing mode used for this record. |
| `processedContent` | string | null | Processed content of the item — a task-labelled excerpt of the page's leading text. |
| `rawContent` | string | null | Raw content of the item — the cleaned visible page text. |
| `scrapedAt` | string | null | When this record was scraped, as an ISO-8601 UTC timestamp. |
| `metadata` | object | null | Metadata of the item, containing `wordCount`, `linksFound` and `imagesFound`. |

The `metadata` object breaks down as follows.

| Metadata Key | Type | Description |
|---|---|---|
| `wordCount` | integer | Number of whitespace-separated words in the page's full cleaned text. |
| `linksFound` | integer | Number of anchor elements present in the page's raw HTML. |
| `imagesFound` | integer | Number of image elements present in the page's raw HTML. |

***

### 💻 How to Use the Intelligent Website Scraper (Step by Step)

#### Step 1: Choose Your Start URLs Carefully

Start URLs determine everything that follows. If you want a single page, list that page. If you want a section of a site, list the section's index page and enable link following. If the site has several disconnected areas you care about — a blog, a product catalogue, a support centre — list one entry point for each rather than relying on the crawler to find its way between them. Because runs are capped at 50 requests, well-chosen entry points matter far more here than they would on an uncapped crawler.

#### Step 2: Decide Whether to Follow Internal Links

`followInternalLinks` defaults to `false`, which means the scraper visits only the URLs you supplied. That is the right setting for targeted extraction where you already know which pages you want. Set it to `true` and the crawler enqueues links on the same domain as it goes, discovering pages you did not list. This is the right setting for site inventories and content audits, but it consumes your 50-request budget quickly on link-dense sites.

#### Step 3: Select the Task Type

The `taskType` value determines how the extracted excerpt is labelled and framed in `processedContent`, and it is written to every record so you can filter a mixed dataset later. Use `summarize` for general content capture, `extractProducts` when crawling catalogue and product pages, `extractServices` for service and solution pages, and `extractFAQs` for help centres and support content. The underlying text extraction is the same across modes; what changes is the labelling and how you will interpret the excerpt downstream.

#### Step 4: Set the Crawl Depth

`maxDepth` describes how far from your start URLs the crawler should travel: `0` means the start URLs alone, `1` means the start URLs plus pages linked directly from them, and higher values extend further. In practice the 50-request ceiling binds before deep values become meaningful on most sites, so depth is best thought of as a coarse control used alongside `followInternalLinks` rather than a precise budget.

#### Step 5: Run the Crawler and Monitor Progress

Start the run from the Apify Console or trigger it over the API. The log records each URL as it begins scraping and notes which task type is being applied. Because a headless browser is doing the work, expect each page to take noticeably longer than a plain HTTP fetch would — that time is what buys you correctly rendered JavaScript content. Records are pushed to the dataset as each page completes, so you can inspect early results immediately.

#### Step 6: Review Text and Structural Metrics Together

When the run finishes, read `rawContent` and `metadata` side by side. A page with a high `wordCount` but a short stored `rawContent` simply means the page is longer than the stored excerpt. A page with a very high `linksFound` relative to `wordCount` is almost certainly a navigation hub, category index or sitemap rather than a content page. A high `imagesFound` with a low `wordCount` signals a gallery or visual landing page. These ratios let you classify pages without reading a single one.

#### Step 7: Export and Feed Downstream Systems

Export the dataset as JSON, CSV or Excel from the Console, or pull it through the API. For content audits, load the rows into a spreadsheet and sort on `metadata.wordCount` to surface thin pages. For search or AI pipelines, index `rawContent` keyed on `url`, keeping `title` and `scrapedAt` as metadata so results stay attributable and you can tell how fresh each record is.

***

### 🔌 API Access & Integrations

Run the Intelligent Website Scraper synchronously and receive dataset items in a single call:

```bash
curl -X POST "https://api.apify.com/v2/acts/scrapers-hub~intelligent-website-scraper/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "startUrls": [{ "url": "https://example.com" }],
    "taskType": "summarize",
    "maxDepth": 1,
    "followInternalLinks": true
  }'
```

The equivalent run in Python using the official client, flagging thin content as it goes:

```python
from apify_client import ApifyClient

client = ApifyClient("YOUR_TOKEN")

run_input = {
    "startUrls": [{"url": "https://example.com"}],
    "taskType": "summarize",
    "maxDepth": 1,
    "followInternalLinks": True,
}

run = client.actor("scrapers-hub/intelligent-website-scraper").call(run_input=run_input)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    meta = item.get("metadata") or {}
    words = meta.get("wordCount", 0)
    flag = "THIN" if words < 300 else "ok"
    print(flag, words, meta.get("linksFound"), meta.get("imagesFound"), item.get("url"))
```

The actor also connects to Zapier, Make, Google Sheets, Slack and any custom endpoint through Apify webhooks, so crawled page content and metrics can flow into a content audit spreadsheet or a team channel as soon as a run completes.

***

### 💡 Best Use Cases for Website Content Extraction Data

#### 📋 Content Audits and Site Inventories

Enable `followInternalLinks` and let the crawler build an inventory of a site section. Sorting the resulting rows on `metadata.wordCount` surfaces thin pages that need attention, while `title` and `url` together give you the page list content teams actually work from. It is a full audit skeleton from one run.

#### 🛒 Product and Catalogue Page Capture

Set `taskType` to `extractProducts` and point the start URLs at category pages. Because Playwright renders client-side catalogue interfaces before extraction, product names and descriptions that never appear in the raw HTML are captured in `rawContent`, and `imagesFound` gives a quick proxy for how richly each category page is illustrated.

#### 🧰 Service and Solution Page Analysis

Agencies and consultancies describe their offerings across a handful of service pages. Crawling those with `taskType` set to `extractServices` produces a structured record per offering, with the labelled `processedContent` making it obvious at a glance which extraction mode produced each row when you merge datasets from several sites.

#### ❓ Help Centre and FAQ Content Extraction

Support content is usually spread across many small pages. Running with `extractFAQs` and internal link following captures them as individual records, each with its own `title` and `rawContent` — the natural shape for loading a help centre into a search index or a customer support assistant.

#### 🔍 Search Index and AI Knowledge Base Building

The `rawContent` field paired with `url` and `title` is exactly what a search index or vector store needs. The `scrapedAt` timestamp lets you implement freshness policies, re-crawling and replacing records older than a chosen threshold rather than rebuilding the whole index.

#### 🏗️ Competitive Content Benchmarking

Crawl a competitor's key sections and compare `metadata.wordCount` distributions against your own equivalent pages. Depth of content is one of the more measurable differences between sites, and `linksFound` additionally reveals how heavily they cross-link, which is a visible part of their internal linking strategy.

#### 🖥️ JavaScript Site Verification

For teams migrating to a client-rendered framework, the question of whether content is actually reaching the rendered DOM is a real one. Running this website scraper against key pages and checking `rawContent` and `wordCount` gives a direct answer, because it reads the same post-render DOM a browser produces.

***

### ⚙️ Tips for Better Website Scraping Results

- **Respect the 50-request ceiling when planning.** With link following enabled, a link-dense start page can consume the entire budget on navigation pages. Point start URLs at content sections rather than the homepage when you want content records.
- **Use several targeted start URLs instead of one broad crawl.** Listing five section entry points with shallow following usually yields a more useful dataset than one homepage entry crawling outward at random.
- **Turn link following off for known URL lists.** If you already have the pages you want, `followInternalLinks: false` spends every request on a page you care about rather than on discovery.
- **Read `wordCount` rather than measuring `rawContent`.** The word count reflects the page's full cleaned text, whereas the stored `rawContent` is an excerpt. For content-length analysis, always use the metric.
- **Allow more time than an HTTP scraper would take.** Every page is rendered in a real browser. That is what makes JavaScript sites work, and it is inherently slower than fetching HTML — budget accordingly on scheduled runs.
- **Re-crawl on a schedule and diff on `scrapedAt`.** Comparing consecutive runs on `title`, `wordCount` and `linksFound` per URL turns a one-off snapshot into genuine change detection across a site.

***

### 🛠️ Troubleshooting

**Why did my crawl stop before covering the whole site?**
The crawler is capped at 50 requests per run. On any site of moderate size with `followInternalLinks` enabled, that ceiling is reached quickly. Split the work into several runs with different start URLs, each targeting a specific section, rather than expecting one run to cover everything.

**Why is `rawContent` shorter than the `wordCount` suggests?**
The `wordCount` metric is calculated across the page's complete cleaned text, while `rawContent` stores a leading portion of it. For long pages the two will legitimately disagree. Use `wordCount` for any length-based analysis and treat `rawContent` as a representative excerpt.

**Why is the extracted text empty or unexpectedly short on some pages?**
Either the page genuinely has little text — a gallery, a video landing page, a redirect stub — or its content loads after the point at which the page was read. Pages with heavy lazy loading or content triggered by scrolling may not have rendered fully. Check `metadata.imagesFound` against `wordCount` to distinguish a visual page from a rendering problem.

**Why did the crawler not follow any links?**
`followInternalLinks` defaults to `false`, so unless you explicitly set it to `true`, only the URLs in `startUrls` are visited. Link discovery is also restricted to the same domain, so links pointing to subdomains or external sites are never enqueued.

**Why does `processedContent` look like a truncated version of `rawContent`?**
That is expected. The task modes apply rule-based selection: each prefixes a label describing the extraction mode and then includes the leading portion of the page text. The modes label and frame the content rather than performing language-model interpretation, which is what keeps output deterministic and free of fabricated detail.

***

### ❓ Frequently Asked Questions About Website Scraping

**What does this Intelligent Website Scraper do?**
It crawls the URLs you supply using a headless Chromium browser, waits for the page to render, removes scripts, styles and media elements, and extracts the remaining visible text. Each page becomes one dataset record containing the URL, title, cleaned text, a task-labelled excerpt, a timestamp and structural page metrics.

**Does the website scraper handle JavaScript-rendered sites?**
Yes. It runs Playwright through Crawlee, driving a real browser, so single-page applications and any content injected client-side are present in the DOM by the time text is extracted. This is the main reason to choose it over a plain HTTP fetcher.

**How many pages can the scraper crawl in one run?**
The crawler is limited to 50 requests per run. For larger jobs, split the work across several runs with different start URLs, each scoped to a particular section of the site.

**What is the difference between `processedContent` and `rawContent`?**
`rawContent` is the cleaned visible text of the page. `processedContent` is a task-labelled excerpt built from that text, prefixed with a label naming the extraction mode used. Both come verbatim from the page; neither is generated or paraphrased.

**Do the task types use an AI model?**
No. The four task modes apply deterministic, rule-based content selection and labelling. There is no language model in the pipeline, which means no token cost, no model latency, and no possibility of invented content in the output.

**Which task type should I choose?**
Pick the one matching your intent: `summarize` for general content capture, `extractProducts` for catalogue pages, `extractServices` for service and solution pages, and `extractFAQs` for help centre content. The choice is written to every record's `taskType` field, so mixed datasets stay filterable.

**How does `maxDepth` interact with `followInternalLinks`?**
`followInternalLinks` decides whether links are discovered at all; `maxDepth` describes how far from the start URLs the crawl may travel. With following disabled, depth has no effect. With it enabled, the 50-request ceiling usually binds before deep values become meaningful.

**Can the scraper crawl links to other domains?**
No. Link discovery uses a same-domain strategy, so only links on the same domain as the page being crawled are enqueued. To cover several domains, list a start URL for each.

**What exactly is stripped out of the page text?**
Script, style, noscript, iframe, image, SVG and video elements are removed before text is collected. What remains is the visible textual content, with whitespace normalised into a single continuous string.

**What do the numbers in the `metadata` object mean?**
`wordCount` is the number of whitespace-separated words in the page's full cleaned text. `linksFound` is the count of anchor elements in the raw HTML, and `imagesFound` is the count of image elements. Together they characterise the page's structure without you needing to read it.

**Does this website scraper need proxy configuration?**
There is no proxy field in the input schema, so there is nothing for you to configure. The crawler operates with its default networking behaviour.

**Can I use the scraper for content audits and SEO analysis?**
Yes, and it is one of the strongest fits. The combination of `title`, `url` and `metadata.wordCount` across a crawled section gives you a thin-content report immediately, while `linksFound` gives a view of internal linking density per page.

**How do I export the scraped website data?**
From the Apify Console Dataset tab you can download JSON, JSONL, CSV, Excel, XML or RSS. Programmatically, use the dataset API or the `apify_client` iteration pattern shown in the API section above.

**Can I schedule the website scraper to run automatically?**
Yes. Apify schedules can trigger the actor on any cron expression, and webhooks can fire on run completion. Comparing consecutive runs by `url` and `scrapedAt` gives you change detection across a site.

**Will the crawler pick up content that loads only when you scroll?**
Not reliably. The page is read after its initial render, so content that appears in response to scrolling, hovering or other interaction may not be present. Pages relying heavily on lazy loading will return less text than a human scrolling through them would see.

***

### 🆘 Support & Feedback

Hit an error, seeing empty content on a page you expect to work, or need behaviour the current task types do not cover? Open a ticket on the **Issues** tab of this actor with the URL and the run ID. That is the fastest route to a fix and leaves a public record other users can learn from.

Need something custom? Higher crawl limits, site-specific extraction rules, structured field extraction, or a private variant wired into your own content pipeline — email **scraperhubapi@gmail.com** with a description of what you are trying to achieve and an example URL.

If this Intelligent Website Scraper saves you time, please leave a review on the actor page. Ratings and written feedback genuinely shape what gets built and prioritised next.

***

### ⚖️ Disclaimer

This Intelligent Website Scraper accesses only publicly available web pages at URLs you explicitly supply, or pages reachable by following public internal links from them. It does not authenticate, does not bypass paywalls, logins or access controls, and does not attempt to circumvent any technical protection measure.

You are responsible for ensuring that your use of this website scraper complies with applicable law, with the robots directives and terms of service of the sites you crawl, and with any contractual obligations you have to those site operators. Website content is protected by copyright belonging to its publisher; extracting text does not transfer any rights, and republication or commercial reuse of scraped content may require permission from the rights holder.

Where extracted page text contains personal data — names, contact details, biographical information, user-generated content or any other information relating to an identifiable person — you act as the data controller for that processing. Ensure you have a lawful basis under the GDPR, the UK GDPR, the CCPA or any other applicable privacy framework, apply data minimisation, retain records no longer than necessary, and honour data subject access, objection and erasure requests.

If you believe content extracted through this actor should be removed, or you are a site operator or rights holder with a concern about a specific page, contact **scraperhubapi@gmail.com** and the request will be reviewed promptly.

# Actor input Schema

## `startUrls` (type: `array`):

List of URLs to scrape and process

## `taskType` (type: `string`):

Type of content processing task to perform

## `maxDepth` (type: `integer`):

Maximum depth for internal link following (0 for only start URL, 1 for start URL and its direct links, etc.)

## `followInternalLinks` (type: `boolean`):

Whether to follow internal links on the same domain

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://example.com"
    }
  ],
  "taskType": "summarize",
  "maxDepth": 1,
  "followInternalLinks": false
}
```

# Actor output Schema

## `results` (type: `string`):

Records scraped by Intelligent Website Scraper, stored in the run's default dataset.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://example.com"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("scrapers-hub/intelligent-website-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://example.com" }] }

# Run the Actor and wait for it to finish
run = client.actor("scrapers-hub/intelligent-website-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://example.com"
    }
  ]
}' |
apify call scrapers-hub/intelligent-website-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,scrapers-hub/intelligent-website-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/HJ4udnHyUcIf32MvC/builds/9OFYavIklw7AkmswC/openapi.json