General Web Scraper & AI Data Extraction

Scrape any website and get clean, machine-readable content — markdown, plain text, or HTML — ready for AI consumption. Uses a headless browser (Playwright) for JavaScript-heavy sites or raw HTTP (Cheerio) for fast, lightweight scraping. Content is automatically extracted via Mozilla Readability, stripping out navigation, ads, footers, and sidebars so you get only the main article.

Perfect for RAG pipelines, LLM training data, content analysis, knowledge base building, and research.

---

Features

• ✅ Universal scraping — works on any website, JS-rendered or static
• ✅ Clean content extraction — Mozilla Readability strips clutter (nav, ads, footers, sidebars)
• ✅ Multiple output formats — Markdown (default), plain text, or raw HTML
• ✅ Smart crawling — configurable depth, same-domain filtering, and max page limits
• ✅ Two crawling engines — Playwright (headless browser, handles JS) or Cheerio (raw HTTP, 10x faster)
• ✅ Apify proxy — automatic proxy rotation to avoid IP blocking
• ✅ Graceful error handling — per-page failures don't crash the run
• ✅ AI-ready output — clean markdown is ideal for RAG, LLM training, and content analysis

Use Cases

Use case	Description
RAG knowledge base	Scrape documentation sites, blogs, or wikis and feed clean markdown into vector DBs
LLM training data	Extract article content from thousands of pages for dataset creation
Competitive research	Monitor competitor websites, product pages, and pricing
Content analysis	Extract and analyze article content, sentiment, or topics at scale
Documentation archiving	Crawl and save entire documentation sites as clean markdown
News monitoring	Scrape news articles and extract the main content without ads or clutter
Lead generation	Extract business data from directory sites and listings

Input Configuration

Field	Type	Default	Description
startUrls	array	Required	One or more URLs to start scraping from
maxPages	integer	25	Maximum pages to scrape (0 = unlimited)
maxCrawlDepth	integer	2	Maximum link depth from start URLs (0 = start URLs only)
sameDomain	boolean	true	Only crawl links within the same domain
crawlerType	enum	playwright:chrome	playwright:chrome (JS rendering) or cheerio (raw HTTP)
outputFormat	enum	markdown	markdown, text, or html
extractMainContent	boolean	true	Strip nav/footer/ads via Readability, or get full page content
proxyConfiguration	object	Apify proxy	Proxy settings for IP rotation

Example Input

{
    "startUrls": [
        { "url": "https://example.com/blog" }
    ],
    "maxPages": 50,
    "maxCrawlDepth": 3,
    "sameDomain": true,
    "crawlerType": "playwright:chrome",
    "outputFormat": "markdown",
    "extractMainContent": true
}

Output Format

Each scraped page produces a dataset item with the following structure:

Field	Type	Description
url	string	The scraped page URL
title	string	Page title extracted from HTML
content	string	Extracted content in requested format (markdown/text/html)
excerpt	string	Short description or meta description
textLength	integer	Length of the extracted text in characters
crawlDepth	integer	Link depth from start URLs
crawledAt	string	ISO timestamp of when the page was scraped

Example Output (Markdown)

{
    "url": "https://example.com/blog/article",
    "title": "How Web Scraping Powers AI",
    "content": "# How Web Scraping Powers AI\n\nWeb scraping is the foundation of modern AI training pipelines...",
    "excerpt": "Learn how web scraping provides the data that fuels AI models",
    "textLength": 3421,
    "crawlDepth": 1,
    "crawledAt": "2026-07-04T20:30:00.000Z"
}

Quick Start

1. Add start URLs — enter the URLs you want to scrape
2. Configure settings — adjust max pages, depth, and output format
3. Run the Actor — results appear in the dataset within seconds to minutes
4. Export results — download as JSON, CSV, XML, or Excel

Performance Tips

• Use Cheerio (crawlerType: cheerio) for static websites — it's 10x faster and cheaper than Playwright
• Set maxCrawlDepth: 0 if you only need the start URLs themselves
• Set extractMainContent: false if you need the full page HTML including navigation
• Use sameDomain: false only if you need to crawl across multiple domains
• For large crawls (>100 pages), start with a small test run to validate the output format

Pricing

This Actor uses pay-per-event (PPE) pricing:

• Per Actor start — a small fee each time the Actor runs
• Per dataset item — a small fee per scraped page
• Platform usage costs (compute, storage, proxy) are included in the pricing

See the Monetization section in Apify Console for current prices.

Limitations

• Free tier limit: The Apify free plan has limited platform credits. Large crawls may require upgrading your Apify plan
• JavaScript-heavy SPAs: Some single-page applications may not render all content within the navigation timeout
• Login-required content: This Actor does not handle authentication — use it on publicly accessible pages only
• Rate limiting: Some websites may block aggressive crawling. Use proxyConfiguration and reasonable maxPages values

FAQ

Q: Can I scrape sites that require login?
A: No. This Actor only scrapes publicly accessible pages.

Q: What's the difference between Playwright and Cheerio?
A: Playwright uses a real headless Chrome browser — it renders JavaScript and handles modern web apps. Cheerio makes raw HTTP requests (no JS rendering) and is 10x faster and cheaper. Use Cheerio for static HTML sites.

Q: How is the content cleaned?
A: Mozilla Readability (the same algorithm behind Firefox's Reader Mode) extracts the main article content, stripping navigation, ads, footers, and sidebars.

Q: Can I scrape multiple domains at once?
A: Yes — set sameDomain: false and add URLs from different domains to startUrls. Be mindful of rate limits.

Q: What does "crawl depth" mean?
A: Depth 0 = the start URLs. Depth 1 = links found on start URLs. Depth 2 = links found on depth-1 pages, etc.

Technical Details

• Runtime: Node.js 24 + Playwright (headless Chrome)
• SDK: Apify SDK v3 + Crawlee v3
• Content extraction: Mozilla Readability via JSDOM
• Markdown conversion: Turndown
• Docker base: apify/actor-node-playwright-chrome:24-1.60.0

Changelog

v0.0.1 (2026-07-04)

• Initial release
• Playwright and Cheerio crawling engines
• Readability-based content extraction
• Markdown, text, and HTML output formats
• Configurable crawl depth and page limits
• Apify proxy integration
• Graceful abort handling

---

Built with Apify SDK and Crawlee