Cheerio Web Scraper - Fast HTTP Crawler & CSS Extractor

Scrape static websites quickly without running a browser. This Actor downloads pages with raw HTTP requests, parses HTML with Cheerio, follows links recursively, and writes clean structured records to an Apify dataset.

Use CSS extraction rules for a quick no-code setup, or add an advanced JavaScript page function when you need complete control. Run it from Apify Console, API, schedules, webhooks, or an MCP-connected AI agent.

What you can scrape

• Product catalogs, prices, stock status, and product metadata
• Article directories, documentation, blogs, and news sites
• Business directories, public listings, and lead pages
• SEO metadata, headings, canonical URLs, images, and JSON-LD
• Static HTML, JSON, XML, and text endpoints

Why use this Actor

• Fast and inexpensive: raw HTTP requests avoid browser startup and rendering costs.
• No-code extraction: map field names to CSS selectors in the input form.
• Developer control: return any JSON-compatible data from a JavaScript page function.
• Recursive crawling: follow selected links with include/exclude globs and depth limits.
• Production controls: retries, concurrency, timeouts, proxy rotation, headers, and robots.txt.
• Ready to integrate: export JSON, CSV, Excel, XML, or RSS and connect through API or MCP.

Quick start: CSS rules

{
  "startUrls": [{ "url": "https://books.toscrape.com/" }],
  "extractionRules": [
    { "name": "heading", "selector": "h1", "type": "text" },
    { "name": "products", "selector": ".product_pod h3 a", "type": "attribute", "attribute": "title", "multiple": true }
  ],
  "linkSelector": "a[href]",
  "globs": [{ "glob": "https://books.toscrape.com/catalogue/**" }],
  "maxCrawlingDepth": 1,
  "maxPagesPerCrawl": 100,
  "maxResultsPerCrawl": 100
}

Each extraction rule supports:

Field	Meaning
name	Output field name
selector	CSS selector evaluated with Cheerio
type	text, html, or attribute
attribute	Attribute name such as href, src, or content
multiple	Return all matches as an array

Advanced JavaScript extraction

When pageFunction is supplied, its return value becomes the page output. The context contains $, request, response, body, json, crawler, Actor, log, and customData.

async function pageFunction({ $, request }) {
    return $('.product_pod').map((_, element) => ({
        sourceUrl: request.loadedUrl || request.url,
        title: $(element).find('h3 a').attr('title'),
        price: $(element).find('.price_color').text().trim(),
        availability: $(element).find('.availability').text().trim(),
    })).get();
}

Returning an array creates one dataset item for each element. Return null to skip output for a page.

Recursive crawling

Set linkSelector to a CSS selector such as a[href]. Use globs to include only desired URLs and excludes to reject assets, account pages, or other unwanted routes. sameDomainOnly is enabled by default, and maxCrawlingDepth prevents unbounded discovery.

Limits and responsible use

This is an HTTP crawler. It does not execute client-side JavaScript, click buttons, or solve browser challenges. Use a Playwright or Puppeteer scraper for pages whose content appears only after browser rendering.

Respect website terms, robots.txt, privacy rights, copyright, and applicable law. The Actor enables respectRobotsTxtFile by default. Lower concurrency for fragile sites and use proxies only when you are authorized to access the target.

Output

The default output contains one item per page or one item per object returned by your page function. Automatic mode can include URL, HTTP status, title, description, canonical URL, language, H1 headings, Open Graph image, JSON-LD, and your custom fields.

Pricing

The Actor uses pay-per-result pricing. A result is one item written to the default dataset. Failed requests are logged but are not emitted as billable result items. Use maxResultsPerCrawl and Apify's maximum charge control to cap every run.

FAQ

Can it render JavaScript?

No. That is why it is faster and cheaper than browser-based crawlers. It is best for server-rendered HTML and public data endpoints.

Can I send authenticated requests?

Yes. Add authorization, cookies, or other headers under additionalHeaders, or attach headers to individual Start URLs. Only provide credentials you are authorized to use.

Can it crawl an entire site?

Yes. Configure linkSelector, URL globs, exclusions, maximum depth, page limit, and result limit. Start small before increasing limits.

Does it work with AI agents?

Yes. Run it through the Apify API or expose it through Apify MCP so an agent can collect current structured web data on demand.

Run with the Apify API

curl -X POST \
  "https://api.apify.com/v2/acts/groupoject~cheerio-web-scraper/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "startUrls":[{"url":"https://books.toscrape.com/"}],
    "extractionRules":[
      {"name":"heading","selector":"h1","type":"text"}
    ],
    "maxPagesPerCrawl":10,
    "maxResultsPerCrawl":10
  }'

Extraction recipes

SEO metadata crawler

Enable automatic page metadata and JSON-LD, set linkSelector to a[href], constrain URLs with same-domain globs, and export titles, descriptions, canonicals, headings, language, images, and structured data.

Product catalog scraper

Use one rule per product field such as title, price, availability, image, SKU, or product URL. Return arrays from pageFunction when one category page contains several products.

JSON endpoint collector

Start from public JSON URLs and use the advanced page function's json value to normalize each response into dataset rows.

Page-function context

The async function receives Cheerio $, request, response, raw body, parsed json, crawler, Actor, log, and user-defined customData. Return an object, an array of objects, or null.

Troubleshooting

• Empty selectors usually mean the content is client-rendered or the CSS selector does not match the returned HTML.
• HTTP 403 or 429 responses may require slower concurrency, authorized headers, or an appropriate proxy.
• Unexpected external URLs should be constrained with sameDomainOnly, globs, and exclusions.
• Large crawls should begin with strict page and result caps before scaling.