🧪 High-Volume Website Content & Media Scraper

Crawl more pages. Capture more content. Keep the output clean.

Turn public websites into structured, automation-ready data at scale. This Actor runs real browser sessions against JavaScript-heavy pages and extracts the content that matters: article text, metadata, images, videos, and links.

Feed it a list of URLs, control the crawl depth and speed, and receive consistent records ready for JSON, CSV, Excel, APIs, AI pipelines, research systems, or monitoring workflows.

Built for high-volume crawling

This is not a one-page demo scraper. It includes the controls needed for serious multi-page runs:

• Process up to 10,000 requests in a run
• Scale browser concurrency from 1 to 50 pages
• Crawl discovered links automatically
• Keep discovery restricted to the original domain
• Cap links added from every processed page
• Apply a global requests-per-minute throttle
• Retry failed pages automatically
• Set independent navigation and processing timeouts
• Preserve failed-page records and diagnostic screenshots

Run it conservatively against sensitive sites or increase concurrency when speed matters.

What it extracts

Every successfully processed page can return:

• Page URL and title
• Structured metadata
• Article-style content blocks
• Image URLs
• Embedded video URLs
• Direct video and streaming-source URLs
• Outbound links
• Content type
• Scrape and processing timestamps
• Crawl status and diagnostics

The result is a usable content and media inventory—not a thin page snapshot.

Live proof crawl

A controlled public-web test completed 4 of 4 requests successfully in 32 seconds for $0.039.

Target	Content blocks	Images	Links	Processing time	Status
NASA homepage	4	57	150	1.143 s	OK
Apify JavaScript scraping article	8	12	100	1.296 s	OK
Apple Newsroom	1	0	150	1.140 s	OK
NASA Pluto & Dwarf Planets	3	15	150	1.131 s	OK

Results reflect the public page state at test time. Dynamic websites can return different counts as their content changes.

Ideal workflows

Content and media extraction

Collect text, images, video references, metadata, and links from public pages in one run.

Website and domain crawling

Start with one or more URLs and follow discovered same-domain links to build a structured view of an entire content surface.

News, blog, and article research

Extract article-style blocks from editorial sites for monitoring, analysis, archiving, or AI processing.

Competitor monitoring

Track public landing pages, campaigns, resources, product messaging, and published media.

JavaScript-heavy websites

Use real browser rendering for pages whose meaningful content does not exist in the initial HTML response.

Public social pages

Process publicly visible social-profile or post pages when the target exposes that content without authentication.

Input example

{
  "startUrls": [
    { "url": "https://example.com/blog" },
    { "url": "https://example.com/resources" }
  ],
  "maxRequests": 500,
  "sameDomainOnly": true,
  "linkEnqueueLimit": 20,
  "minConcurrency": 2,
  "maxConcurrency": 10,
  "maxRequestsPerMinute": 120,
  "retryLimit": 2,
  "requestHandlerTimeoutSecs": 90,
  "navigationTimeoutSecs": 90,
  "waitAfterLoadSecs": 0.5,
  "includeFacebookLinks": true,
  "saveErrorScreenshots": true
}

Core crawl controls

Setting	Purpose
startUrls	Public pages where the crawl begins
maxRequests	Maximum total pages processed
sameDomainOnly	Prevent discovered links from leaving the starting domain
linkEnqueueLimit	Maximum discovered links added per page
minConcurrency	Minimum parallel browser-page count
maxConcurrency	Maximum parallel browser-page count
maxRequestsPerMinute	Global crawl throttle; use 0 for unlimited
retryLimit	Automatic retries for failed requests
navigationTimeoutSecs	Maximum navigation time
requestHandlerTimeoutSecs	Maximum total processing time per page
waitAfterLoadSecs	Extra time for client-side rendering
saveErrorScreenshots	Preserve visual diagnostics for failed pages

Structured output

Dataset records can include:

• url
• title
• meta
• articles
• images
• videos
• links
• contentType
• scraped
• scrapeTime
• processingTimeMs
• status
• error
• diag

Download the dataset directly or connect it to another application through the Apify API and integrations.

Failures stay visible

Large crawls should not become impossible to audit because several pages fail. Failed requests can still produce dataset records containing the URL, status, error details, and an optional diagnostic screenshot reference. That makes it easier to identify blocked pages, correct inputs, and rerun only what needs attention.

Public-web access

This Actor extracts content exposed by publicly accessible pages. It does not bypass logins, paywalls, privacy controls, or platform access restrictions.

---

One Actor. Thousands of pages. Structured content, media, and links ready for whatever comes next.