🧪 High-Volume Website Content & Media Scraper

Crawl more pages. Capture more content. Keep the output clean.

Turn public websites into structured, automation-ready data at scale. This Actor runs real browser sessions against JavaScript-heavy pages and extracts the content that matters: article text, metadata, images, videos, and links.

Start from exact URLs or enter keyword searches. Keyword mode discovers matching public webpages first and then runs the same structured extraction against every result.

Feed it targets, control the crawl depth and speed, and receive consistent records ready for JSON, CSV, Excel, APIs, AI pipelines, research systems, or monitoring workflows.

Built for high-volume crawling

This is not a one-page demo scraper. It includes the controls needed for serious multi-page runs:

• Process up to 10,000 requests in a run
• Scale browser concurrency from 1 to 50 pages
• Discover targets from keyword searches
• Crawl links discovered on processed pages
• Keep discovery restricted to the original domain
• Cap links added from every processed page
• Apply a global requests-per-minute throttle
• Retry failed pages automatically
• Set independent navigation and processing timeouts
• Route browser traffic through optional Apify or custom proxies
• Preserve failed-page records and diagnostic screenshots

Run it conservatively against sensitive sites or increase concurrency when speed matters.

What it extracts

Every successfully processed page can return:

• Page URL and title
• Structured metadata
• Article-style content blocks
• Image URLs
• Embedded video URLs
• Direct video and streaming-source URLs
• Outbound links
• Content type
• Scrape and processing timestamps
• Crawl status and diagnostics

The result is a usable content and media inventory—not a thin page snapshot.

Lazy image sources are retained when a page declares them even if the browser has not loaded their pixels yet. Known small images are still filtered, and image output remains capped at 120 URLs per record.

Live proof crawl

A controlled public-web test completed 4 of 4 requests successfully in 32 seconds for $0.039.

Target	Content blocks	Images	Links	Processing time	Status
NASA homepage	4	57	150	1.143 s	OK
Apify JavaScript scraping article	8	12	100	1.296 s	OK
Apple Newsroom	1	0	150	1.140 s	OK
NASA Pluto & Dwarf Planets	3	15	150	1.131 s	OK

Results reflect the public page state at test time. Dynamic websites can return different counts as their content changes.

Ideal workflows

Content and media extraction

Collect text, images, video references, metadata, and links from public pages in one run.

Website and domain crawling

Start with one or more URLs and follow discovered same-domain links to build a structured view of an entire content surface.

Keyword discovery

Enter search queries, discover matching public webpages, and send those results through the same extraction pipeline automatically.

News, blog, and article research

Extract article-style blocks from editorial sites for monitoring, analysis, archiving, or AI processing.

Competitor monitoring

Track public landing pages, campaigns, resources, product messaging, and published media.

JavaScript-heavy websites

Use real browser rendering for pages whose meaningful content does not exist in the initial HTML response.

Public social pages

Process publicly visible social-profile or post pages when the target exposes that content without authentication.

Direct URL crawl

{
  "startUrls": [
    { "url": "https://example.com/blog" },
    { "url": "https://example.com/resources" }
  ],
  "maxRequests": 500,
  "sameDomainOnly": true,
  "linkEnqueueLimit": 20,
  "minConcurrency": 2,
  "maxConcurrency": 10,
  "maxRequestsPerMinute": 120,
  "retryLimit": 2,
  "requestHandlerTimeoutSecs": 90,
  "navigationTimeoutSecs": 90,
  "waitAfterLoadSecs": 0.5,
  "proxyConfiguration": {
    "useApifyProxy": false
  },
  "includeFacebookLinks": true,
  "saveErrorScreenshots": true
}

Keyword discovery crawl

{
  "searchQueries": [
    "independent pizza restaurants California",
    "California pizza restaurant contact page"
  ],
  "searchResultLimit": 20,
  "maxRequests": 50
}

Keyword results are web-search results, not an exhaustive business directory. For geographic business discovery with addresses, phone numbers, ratings, and map listings, use Universal Data Harvester's google_maps mode.

Tracked Bing result links are resolved before they are crawled. When a query contains an explicit site: scope or quoted phrase, results that do not match those constraints are discarded. If the search provider returns only unrelated or malformed results, the Actor reports zero discovered targets instead of scraping an unrelated page.

Core crawl controls

Setting	Purpose
startUrls	Public pages where the crawl begins
searchQueries	Keywords used to discover public target pages
searchResultLimit	Maximum discovered targets per query
maxRequests	Maximum total pages processed
sameDomainOnly	Prevent discovered links from leaving the starting domain
linkEnqueueLimit	Maximum discovered links added per page
minConcurrency	Minimum parallel browser-page count
maxConcurrency	Maximum parallel browser-page count
maxRequestsPerMinute	Global crawl throttle; use 0 for unlimited
retryLimit	Automatic retries for failed requests
navigationTimeoutSecs	Maximum navigation time
requestHandlerTimeoutSecs	Maximum total processing time per page
waitAfterLoadSecs	Extra time for client-side rendering
proxyConfiguration	Optional Apify Proxy or custom proxy URLs; disabled by default
saveErrorScreenshots	Preserve visual diagnostics for failed pages

Proxy use is opt-in. Leave useApifyProxy set to false for direct connections, enable Apify Proxy in the input editor, or supply custom proxy URLs through the same proxy configuration control.

Standby status API

Standby mode exposes two lightweight status endpoints:

• GET / returns readiness and explains that scraping is started as a normal Actor run.
• GET /health returns the Standby server health status.

These endpoints do not perform a scrape. Submit the documented Actor input through a normal Console, API, task, schedule, or integration run to create dataset results. This keeps the existing batch workflow unchanged while giving Apify's readiness probe and monitoring clients a truthful HTTP interface.

Structured output

Dataset records can include:

• url
• title
• meta
• articles
• images
• videos
• links
• contentType
• scraped
• scrapeTime
• processingTimeMs
• status
• error
• diag

Download the dataset directly or connect it to another application through the Apify API and integrations.

Failures stay visible

Large crawls should not become impossible to audit because several pages fail. Failed requests can still produce dataset records containing the URL, status, error details, and an optional diagnostic screenshot reference. That makes it easier to identify blocked pages, correct inputs, and rerun only what needs attention.

Public-web access

This Actor extracts content exposed by publicly accessible pages. It does not bypass logins, paywalls, privacy controls, or platform access restrictions.

---

One Actor. Thousands of pages. Structured content, media, and links ready for whatever comes next.