🕸️ Intelligent Website Scraper – Website Content Extraction & Page Text Crawler

The Intelligent Website Scraper crawls any website with a real headless browser, strips away scripts, styles and media, and returns the clean readable text of every page it visits alongside a task-labelled content excerpt and structural page metrics. Give it a list of start URLs, choose a task type, and decide whether to follow internal links — the scraper handles JavaScript rendering, same-domain link discovery and text normalisation, and pushes one tidy record per page into an Apify dataset.

This website scraper is deliberately straightforward about what it does. It uses Playwright through Crawlee, so pages built with React, Vue, Angular or any other client-side framework render fully before extraction, which is where simple HTTP-based scrapers fail. The text you get back is the page's visible content with <script>, <style>, <noscript>, <iframe>, <img>, <svg> and <video> elements removed, joined into a single normalised string. Alongside it you get a word count, a link count and an image count for every page — the kind of structural signal that makes content audits, site inventories and page-quality checks possible at scale.

---

📊 What Data Can You Extract with This Website Scraper?

Every page the crawler visits produces one dataset row. The fields fall into five natural groups.

Category	Fields	What You Get
🔗 Page identity	url, title	The exact page URL crawled and the rendered document title
🏷️ Task labelling	taskType	Which processing mode produced this record — summarise, products, services or FAQs
✂️ Processed excerpt	processedContent	A task-labelled excerpt of the page's leading content, prefixed with the extraction mode
📃 Page text	rawContent	The cleaned visible text of the page, captured from the fully rendered DOM
📐 Structural metrics	metadata	An object with wordCount, linksFound and imagesFound for the page
🕒 Run metadata	scrapedAt	ISO-8601 UTC timestamp recording when the page was crawled

The metadata object is the quietly valuable part. wordCount reflects the full text length of the page before truncation, so you can identify thin content even when the stored excerpt is short; linksFound and imagesFound count every anchor and image element in the raw HTML, which makes navigation-heavy hub pages and media-heavy landing pages immediately distinguishable from ordinary article pages.

---

🌟 Key Features of the Intelligent Website Scraper

Feature	Description
🌐 Full JavaScript rendering	Uses Playwright through Crawlee, so single-page applications and client-rendered content are captured after the page has hydrated
🔗 Same-domain link following	Enable followInternalLinks and the crawler enqueues internal links automatically, expanding from your start URLs across the site
🎯 Four task modes	summarize, extractProducts, extractServices and extractFAQs label and frame the extracted excerpt for your downstream workflow
🧹 Clean text extraction	Scripts, styles, noscript blocks, iframes, images, SVGs and video elements are removed before text is collected
📐 Per-page structural metrics	Word count, anchor count and image count are computed for every page and returned in the metadata object
📋 Multiple start URLs	Supply an array of entry points to crawl several sections, subdomains paths or landing pages in a single run
🛡️ Built-in crawl ceiling	A maximum of 50 requests per crawl keeps runs bounded and predictable rather than expanding without limit
⏱️ Timestamped records	Every row carries an ISO-8601 UTC scrapedAt value, so repeated crawls can be compared chronologically
📤 Incremental dataset pushes	Records are written as each page finishes, so partial results are available while a crawl is still running

---

🚀 Why Choose This Intelligent Website Scraper?

It renders the page properly. A large share of the modern web serves an almost empty HTML shell and builds the content in the browser. Fetching that HTML with a plain HTTP request returns nothing useful. This website scraper drives a real Chromium instance through Playwright and reads the DOM after rendering, so what you extract is what a visitor actually sees.

Text and structure in one pass. Most scrapers give you either the content or the page statistics, never both. Every record here pairs the cleaned rawContent with a metadata object carrying wordCount, linksFound and imagesFound, which means a single crawl supports both content analysis and technical site auditing.

Predictable, bounded crawls. The crawler is capped at 50 requests per run. That is a deliberate design decision, not a limitation to work around: it keeps compute usage and run times foreseeable, prevents an accidental followInternalLinks setting from crawling a hundred-thousand-page site, and makes cost estimation simple.

Honest, deterministic extraction. The task modes apply straightforward, rule-based content selection rather than a language model. That means results are reproducible, there is no model latency or token cost, and nothing is ever hallucinated — the text in your dataset came verbatim from the page.

---

📥 Input

The Intelligent Website Scraper takes a list of start URLs plus three crawl and processing controls. Only startUrls is required.

{
  "startUrls": [
    { "url": "https://example.com" }
  ],
  "taskType": "summarize",
  "maxDepth": 1,
  "followInternalLinks": false
}

🔧 Intelligent Website Scraper Input Fields

Field	Type	Required	Default	Description
startUrls	array	✅ Yes	—	List of URLs to scrape and process. Each entry is an object with a url key, in Apify's request-list format.
taskType	string	❌ No	"summarize"	Type of content processing task to perform. One of summarize (summarise entire site content), extractProducts (extract product information), extractServices (extract service offerings) or extractFAQs (extract FAQ content).
maxDepth	integer	❌ No	1	Maximum depth for internal link following — 0 for only the start URL, 1 for the start URL and its direct links, and so on.
followInternalLinks	boolean	❌ No	false	Whether to follow internal links on the same domain.

💡 Input Examples

Single-page content capture

{
  "startUrls": [{ "url": "https://example.com/about" }],
  "taskType": "summarize",
  "followInternalLinks": false
}

Site section crawl for service offerings

{
  "startUrls": [{ "url": "https://example.com/services" }],
  "taskType": "extractServices",
  "maxDepth": 1,
  "followInternalLinks": true
}

Multi-entry product catalogue sweep

{
  "startUrls": [
    { "url": "https://example.com/shop/mens" },
    { "url": "https://example.com/shop/womens" },
    { "url": "https://example.com/shop/accessories" }
  ],
  "taskType": "extractProducts",
  "maxDepth": 1,
  "followInternalLinks": true
}

---

📤 Output

Each dataset item represents one crawled page.

{
  "url": "https://example.com/services",
  "title": "Our Services | Example Ltd",
  "taskType": "extractServices",
  "processedContent": "Services found in text (manual extraction): Our Services Consulting Implementation Managed Support We work with mid-market operations teams to design, deploy and run...",
  "rawContent": "Our Services Consulting Implementation Managed Support We work with mid-market operations teams to design, deploy and run the systems that keep their business moving. Every engagement starts with a discovery workshop...",
  "scrapedAt": "2026-08-17T09:12:44.183920Z",
  "metadata": {
    "wordCount": 1284,
    "linksFound": 47,
    "imagesFound": 12
  }
}

🧾 Intelligent Website Scraper Output Fields

Field	Type	Description
url	string | null	Canonical URL of the scraped item — the page that was crawled.
title	string | null	Title of the item, taken from the rendered document title.
taskType	string | null	Task type of the item — the processing mode used for this record.
processedContent	string | null	Processed content of the item — a task-labelled excerpt of the page's leading text.
rawContent	string | null	Raw content of the item — the cleaned visible page text.
scrapedAt	string | null	When this record was scraped, as an ISO-8601 UTC timestamp.
metadata	object | null	Metadata of the item, containing wordCount, linksFound and imagesFound.

The metadata object breaks down as follows.

Metadata Key	Type	Description
wordCount	integer	Number of whitespace-separated words in the page's full cleaned text.
linksFound	integer	Number of anchor elements present in the page's raw HTML.
imagesFound	integer	Number of image elements present in the page's raw HTML.

---

💻 How to Use the Intelligent Website Scraper (Step by Step)

Step 1: Choose Your Start URLs Carefully

Start URLs determine everything that follows. If you want a single page, list that page. If you want a section of a site, list the section's index page and enable link following. If the site has several disconnected areas you care about — a blog, a product catalogue, a support centre — list one entry point for each rather than relying on the crawler to find its way between them. Because runs are capped at 50 requests, well-chosen entry points matter far more here than they would on an uncapped crawler.

Step 2: Decide Whether to Follow Internal Links

followInternalLinks defaults to false, which means the scraper visits only the URLs you supplied. That is the right setting for targeted extraction where you already know which pages you want. Set it to true and the crawler enqueues links on the same domain as it goes, discovering pages you did not list. This is the right setting for site inventories and content audits, but it consumes your 50-request budget quickly on link-dense sites.

Step 3: Select the Task Type

The taskType value determines how the extracted excerpt is labelled and framed in processedContent, and it is written to every record so you can filter a mixed dataset later. Use summarize for general content capture, extractProducts when crawling catalogue and product pages, extractServices for service and solution pages, and extractFAQs for help centres and support content. The underlying text extraction is the same across modes; what changes is the labelling and how you will interpret the excerpt downstream.

Step 4: Set the Crawl Depth

maxDepth describes how far from your start URLs the crawler should travel: 0 means the start URLs alone, 1 means the start URLs plus pages linked directly from them, and higher values extend further. In practice the 50-request ceiling binds before deep values become meaningful on most sites, so depth is best thought of as a coarse control used alongside followInternalLinks rather than a precise budget.

Step 5: Run the Crawler and Monitor Progress

Start the run from the Apify Console or trigger it over the API. The log records each URL as it begins scraping and notes which task type is being applied. Because a headless browser is doing the work, expect each page to take noticeably longer than a plain HTTP fetch would — that time is what buys you correctly rendered JavaScript content. Records are pushed to the dataset as each page completes, so you can inspect early results immediately.

Step 6: Review Text and Structural Metrics Together

When the run finishes, read rawContent and metadata side by side. A page with a high wordCount but a short stored rawContent simply means the page is longer than the stored excerpt. A page with a very high linksFound relative to wordCount is almost certainly a navigation hub, category index or sitemap rather than a content page. A high imagesFound with a low wordCount signals a gallery or visual landing page. These ratios let you classify pages without reading a single one.

Step 7: Export and Feed Downstream Systems

Export the dataset as JSON, CSV or Excel from the Console, or pull it through the API. For content audits, load the rows into a spreadsheet and sort on metadata.wordCount to surface thin pages. For search or AI pipelines, index rawContent keyed on url, keeping title and scrapedAt as metadata so results stay attributable and you can tell how fresh each record is.

---

🔌 API Access & Integrations

Run the Intelligent Website Scraper synchronously and receive dataset items in a single call:

curl -X POST "https://api.apify.com/v2/acts/scrapers-hub~intelligent-website-scraper/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "startUrls": [{ "url": "https://example.com" }],
    "taskType": "summarize",
    "maxDepth": 1,
    "followInternalLinks": true
  }'

The equivalent run in Python using the official client, flagging thin content as it goes:

from apify_client import ApifyClient

client = ApifyClient("YOUR_TOKEN")

run_input = {
    "startUrls": [{"url": "https://example.com"}],
    "taskType": "summarize",
    "maxDepth": 1,
    "followInternalLinks": True,
}

run = client.actor("scrapers-hub/intelligent-website-scraper").call(run_input=run_input)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    meta = item.get("metadata") or {}
    words = meta.get("wordCount", 0)
    flag = "THIN" if words < 300 else "ok"
    print(flag, words, meta.get("linksFound"), meta.get("imagesFound"), item.get("url"))

The actor also connects to Zapier, Make, Google Sheets, Slack and any custom endpoint through Apify webhooks, so crawled page content and metrics can flow into a content audit spreadsheet or a team channel as soon as a run completes.

---

💡 Best Use Cases for Website Content Extraction Data

📋 Content Audits and Site Inventories

Enable followInternalLinks and let the crawler build an inventory of a site section. Sorting the resulting rows on metadata.wordCount surfaces thin pages that need attention, while title and url together give you the page list content teams actually work from. It is a full audit skeleton from one run.

🛒 Product and Catalogue Page Capture

Set taskType to extractProducts and point the start URLs at category pages. Because Playwright renders client-side catalogue interfaces before extraction, product names and descriptions that never appear in the raw HTML are captured in rawContent, and imagesFound gives a quick proxy for how richly each category page is illustrated.

🧰 Service and Solution Page Analysis

Agencies and consultancies describe their offerings across a handful of service pages. Crawling those with taskType set to extractServices produces a structured record per offering, with the labelled processedContent making it obvious at a glance which extraction mode produced each row when you merge datasets from several sites.

❓ Help Centre and FAQ Content Extraction

Support content is usually spread across many small pages. Running with extractFAQs and internal link following captures them as individual records, each with its own title and rawContent — the natural shape for loading a help centre into a search index or a customer support assistant.

🔍 Search Index and AI Knowledge Base Building

The rawContent field paired with url and title is exactly what a search index or vector store needs. The scrapedAt timestamp lets you implement freshness policies, re-crawling and replacing records older than a chosen threshold rather than rebuilding the whole index.

🏗️ Competitive Content Benchmarking

Crawl a competitor's key sections and compare metadata.wordCount distributions against your own equivalent pages. Depth of content is one of the more measurable differences between sites, and linksFound additionally reveals how heavily they cross-link, which is a visible part of their internal linking strategy.

🖥️ JavaScript Site Verification

For teams migrating to a client-rendered framework, the question of whether content is actually reaching the rendered DOM is a real one. Running this website scraper against key pages and checking rawContent and wordCount gives a direct answer, because it reads the same post-render DOM a browser produces.

---

⚙️ Tips for Better Website Scraping Results

• Respect the 50-request ceiling when planning. With link following enabled, a link-dense start page can consume the entire budget on navigation pages. Point start URLs at content sections rather than the homepage when you want content records.
• Use several targeted start URLs instead of one broad crawl. Listing five section entry points with shallow following usually yields a more useful dataset than one homepage entry crawling outward at random.
• Turn link following off for known URL lists. If you already have the pages you want, followInternalLinks: false spends every request on a page you care about rather than on discovery.
• Read wordCount rather than measuring rawContent. The word count reflects the page's full cleaned text, whereas the stored rawContent is an excerpt. For content-length analysis, always use the metric.
• Allow more time than an HTTP scraper would take. Every page is rendered in a real browser. That is what makes JavaScript sites work, and it is inherently slower than fetching HTML — budget accordingly on scheduled runs.
• Re-crawl on a schedule and diff on scrapedAt. Comparing consecutive runs on title, wordCount and linksFound per URL turns a one-off snapshot into genuine change detection across a site.

---

🛠️ Troubleshooting

Why did my crawl stop before covering the whole site? The crawler is capped at 50 requests per run. On any site of moderate size with followInternalLinks enabled, that ceiling is reached quickly. Split the work into several runs with different start URLs, each targeting a specific section, rather than expecting one run to cover everything.

Why is rawContent shorter than the wordCount suggests? The wordCount metric is calculated across the page's complete cleaned text, while rawContent stores a leading portion of it. For long pages the two will legitimately disagree. Use wordCount for any length-based analysis and treat rawContent as a representative excerpt.

Why is the extracted text empty or unexpectedly short on some pages? Either the page genuinely has little text — a gallery, a video landing page, a redirect stub — or its content loads after the point at which the page was read. Pages with heavy lazy loading or content triggered by scrolling may not have rendered fully. Check metadata.imagesFound against wordCount to distinguish a visual page from a rendering problem.

Why did the crawler not follow any links? followInternalLinks defaults to false, so unless you explicitly set it to true, only the URLs in startUrls are visited. Link discovery is also restricted to the same domain, so links pointing to subdomains or external sites are never enqueued.

Why does processedContent look like a truncated version of rawContent? That is expected. The task modes apply rule-based selection: each prefixes a label describing the extraction mode and then includes the leading portion of the page text. The modes label and frame the content rather than performing language-model interpretation, which is what keeps output deterministic and free of fabricated detail.

---

❓ Frequently Asked Questions About Website Scraping

What does this Intelligent Website Scraper do? It crawls the URLs you supply using a headless Chromium browser, waits for the page to render, removes scripts, styles and media elements, and extracts the remaining visible text. Each page becomes one dataset record containing the URL, title, cleaned text, a task-labelled excerpt, a timestamp and structural page metrics.

Does the website scraper handle JavaScript-rendered sites? Yes. It runs Playwright through Crawlee, driving a real browser, so single-page applications and any content injected client-side are present in the DOM by the time text is extracted. This is the main reason to choose it over a plain HTTP fetcher.

How many pages can the scraper crawl in one run? The crawler is limited to 50 requests per run. For larger jobs, split the work across several runs with different start URLs, each scoped to a particular section of the site.

What is the difference between processedContent and rawContent? rawContent is the cleaned visible text of the page. processedContent is a task-labelled excerpt built from that text, prefixed with a label naming the extraction mode used. Both come verbatim from the page; neither is generated or paraphrased.

Do the task types use an AI model? No. The four task modes apply deterministic, rule-based content selection and labelling. There is no language model in the pipeline, which means no token cost, no model latency, and no possibility of invented content in the output.

Which task type should I choose? Pick the one matching your intent: summarize for general content capture, extractProducts for catalogue pages, extractServices for service and solution pages, and extractFAQs for help centre content. The choice is written to every record's taskType field, so mixed datasets stay filterable.

How does maxDepth interact with followInternalLinks? followInternalLinks decides whether links are discovered at all; maxDepth describes how far from the start URLs the crawl may travel. With following disabled, depth has no effect. With it enabled, the 50-request ceiling usually binds before deep values become meaningful.

Can the scraper crawl links to other domains? No. Link discovery uses a same-domain strategy, so only links on the same domain as the page being crawled are enqueued. To cover several domains, list a start URL for each.

What exactly is stripped out of the page text? Script, style, noscript, iframe, image, SVG and video elements are removed before text is collected. What remains is the visible textual content, with whitespace normalised into a single continuous string.

What do the numbers in the metadata object mean? wordCount is the number of whitespace-separated words in the page's full cleaned text. linksFound is the count of anchor elements in the raw HTML, and imagesFound is the count of image elements. Together they characterise the page's structure without you needing to read it.

Does this website scraper need proxy configuration? There is no proxy field in the input schema, so there is nothing for you to configure. The crawler operates with its default networking behaviour.

Can I use the scraper for content audits and SEO analysis? Yes, and it is one of the strongest fits. The combination of title, url and metadata.wordCount across a crawled section gives you a thin-content report immediately, while linksFound gives a view of internal linking density per page.

How do I export the scraped website data? From the Apify Console Dataset tab you can download JSON, JSONL, CSV, Excel, XML or RSS. Programmatically, use the dataset API or the apify_client iteration pattern shown in the API section above.

Can I schedule the website scraper to run automatically? Yes. Apify schedules can trigger the actor on any cron expression, and webhooks can fire on run completion. Comparing consecutive runs by url and scrapedAt gives you change detection across a site.

Will the crawler pick up content that loads only when you scroll? Not reliably. The page is read after its initial render, so content that appears in response to scrolling, hovering or other interaction may not be present. Pages relying heavily on lazy loading will return less text than a human scrolling through them would see.

---

🆘 Support & Feedback

Hit an error, seeing empty content on a page you expect to work, or need behaviour the current task types do not cover? Open a ticket on the Issues tab of this actor with the URL and the run ID. That is the fastest route to a fix and leaves a public record other users can learn from.

Need something custom? Higher crawl limits, site-specific extraction rules, structured field extraction, or a private variant wired into your own content pipeline — email scraperhubapi@gmail.com with a description of what you are trying to achieve and an example URL.

If this Intelligent Website Scraper saves you time, please leave a review on the actor page. Ratings and written feedback genuinely shape what gets built and prioritised next.

---

⚖️ Disclaimer

This Intelligent Website Scraper accesses only publicly available web pages at URLs you explicitly supply, or pages reachable by following public internal links from them. It does not authenticate, does not bypass paywalls, logins or access controls, and does not attempt to circumvent any technical protection measure.

You are responsible for ensuring that your use of this website scraper complies with applicable law, with the robots directives and terms of service of the sites you crawl, and with any contractual obligations you have to those site operators. Website content is protected by copyright belonging to its publisher; extracting text does not transfer any rights, and republication or commercial reuse of scraped content may require permission from the rights holder.

Where extracted page text contains personal data — names, contact details, biographical information, user-generated content or any other information relating to an identifiable person — you act as the data controller for that processing. Ensure you have a lawful basis under the GDPR, the UK GDPR, the CCPA or any other applicable privacy framework, apply data minimisation, retain records no longer than necessary, and honour data subject access, objection and erasure requests.

If you believe content extracted through this actor should be removed, or you are a site operator or rights holder with a concern about a specific page, contact scraperhubapi@gmail.com and the request will be reviewed promptly.