# Cheerio Web Scraper - Fast HTTP Crawler & CSS Extractor (`groupoject/cheerio-web-scraper`) Actor

Crawl static websites at high speed with HTTP requests and Cheerio. Extract structured data with no-code CSS rules or a JavaScript page function. Supports recursive links, proxies, JSON-LD, metadata, robots.txt, API, and MCP workflows.

- **URL**: https://apify.com/groupoject/cheerio-web-scraper.md
- **Developed by:** [Group Oject](https://apify.com/groupoject) (community)
- **Categories:** Developer tools, Automation, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 web page results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Cheerio Web Scraper - Fast HTTP Crawler & CSS Extractor

Scrape static websites quickly without running a browser. This Actor downloads pages with raw HTTP requests, parses HTML with Cheerio, follows links recursively, and writes clean structured records to an Apify dataset.

Use **CSS extraction rules** for a quick no-code setup, or add an **advanced JavaScript page function** when you need complete control. Run it from Apify Console, API, schedules, webhooks, or an MCP-connected AI agent.

### What you can scrape

- Product catalogs, prices, stock status, and product metadata
- Article directories, documentation, blogs, and news sites
- Business directories, public listings, and lead pages
- SEO metadata, headings, canonical URLs, images, and JSON-LD
- Static HTML, JSON, XML, and text endpoints

### Why use this Actor

- **Fast and inexpensive:** raw HTTP requests avoid browser startup and rendering costs.
- **No-code extraction:** map field names to CSS selectors in the input form.
- **Developer control:** return any JSON-compatible data from a JavaScript page function.
- **Recursive crawling:** follow selected links with include/exclude globs and depth limits.
- **Production controls:** retries, concurrency, timeouts, proxy rotation, headers, and robots.txt.
- **Ready to integrate:** export JSON, CSV, Excel, XML, or RSS and connect through API or MCP.

### Quick start: CSS rules

```json
{
  "startUrls": [{ "url": "https://books.toscrape.com/" }],
  "extractionRules": [
    { "name": "heading", "selector": "h1", "type": "text" },
    { "name": "products", "selector": ".product_pod h3 a", "type": "attribute", "attribute": "title", "multiple": true }
  ],
  "linkSelector": "a[href]",
  "globs": [{ "glob": "https://books.toscrape.com/catalogue/**" }],
  "maxCrawlingDepth": 1,
  "maxPagesPerCrawl": 100,
  "maxResultsPerCrawl": 100
}
```

Each extraction rule supports:

| Field | Meaning |
|---|---|
| `name` | Output field name |
| `selector` | CSS selector evaluated with Cheerio |
| `type` | `text`, `html`, or `attribute` |
| `attribute` | Attribute name such as `href`, `src`, or `content` |
| `multiple` | Return all matches as an array |

### Advanced JavaScript extraction

When `pageFunction` is supplied, its return value becomes the page output. The context contains `$`, `request`, `response`, `body`, `json`, `crawler`, `Actor`, `log`, and `customData`.

```javascript
async function pageFunction({ $, request }) {
    return $('.product_pod').map((_, element) => ({
        sourceUrl: request.loadedUrl || request.url,
        title: $(element).find('h3 a').attr('title'),
        price: $(element).find('.price_color').text().trim(),
        availability: $(element).find('.availability').text().trim(),
    })).get();
}
```

Returning an array creates one dataset item for each element. Return `null` to skip output for a page.

### Recursive crawling

Set `linkSelector` to a CSS selector such as `a[href]`. Use `globs` to include only desired URLs and `excludes` to reject assets, account pages, or other unwanted routes. `sameDomainOnly` is enabled by default, and `maxCrawlingDepth` prevents unbounded discovery.

### Limits and responsible use

This is an HTTP crawler. It does not execute client-side JavaScript, click buttons, or solve browser challenges. Use a Playwright or Puppeteer scraper for pages whose content appears only after browser rendering.

Respect website terms, robots.txt, privacy rights, copyright, and applicable law. The Actor enables `respectRobotsTxtFile` by default. Lower concurrency for fragile sites and use proxies only when you are authorized to access the target.

### Output

The default output contains one item per page or one item per object returned by your page function. Automatic mode can include URL, HTTP status, title, description, canonical URL, language, H1 headings, Open Graph image, JSON-LD, and your custom fields.

### Pricing

The Actor uses pay-per-result pricing. A result is one item written to the default dataset. Failed requests are logged but are not emitted as billable result items. Use `maxResultsPerCrawl` and Apify's maximum charge control to cap every run.

### FAQ

#### Can it render JavaScript?

No. That is why it is faster and cheaper than browser-based crawlers. It is best for server-rendered HTML and public data endpoints.

#### Can I send authenticated requests?

Yes. Add authorization, cookies, or other headers under `additionalHeaders`, or attach headers to individual Start URLs. Only provide credentials you are authorized to use.

#### Can it crawl an entire site?

Yes. Configure `linkSelector`, URL globs, exclusions, maximum depth, page limit, and result limit. Start small before increasing limits.

#### Does it work with AI agents?

Yes. Run it through the Apify API or expose it through Apify MCP so an agent can collect current structured web data on demand.

### Run with the Apify API

```bash
curl -X POST \
  "https://api.apify.com/v2/acts/groupoject~cheerio-web-scraper/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "startUrls":[{"url":"https://books.toscrape.com/"}],
    "extractionRules":[
      {"name":"heading","selector":"h1","type":"text"}
    ],
    "maxPagesPerCrawl":10,
    "maxResultsPerCrawl":10
  }'
```

### Extraction recipes

#### SEO metadata crawler

Enable automatic page metadata and JSON-LD, set `linkSelector` to `a[href]`, constrain URLs with same-domain globs, and export titles, descriptions, canonicals, headings, language, images, and structured data.

#### Product catalog scraper

Use one rule per product field such as title, price, availability, image, SKU, or product URL. Return arrays from `pageFunction` when one category page contains several products.

#### JSON endpoint collector

Start from public JSON URLs and use the advanced page function's `json` value to normalize each response into dataset rows.

### Page-function context

The async function receives Cheerio `$`, `request`, `response`, raw `body`, parsed `json`, `crawler`, `Actor`, `log`, and user-defined `customData`. Return an object, an array of objects, or `null`.

### Troubleshooting

- Empty selectors usually mean the content is client-rendered or the CSS selector does not match the returned HTML.
- HTTP 403 or 429 responses may require slower concurrency, authorized headers, or an appropriate proxy.
- Unexpected external URLs should be constrained with `sameDomainOnly`, globs, and exclusions.
- Large crawls should begin with strict page and result caps before scaling.

# Actor input Schema

## `startUrls` (type: `array`):

Pages where the crawl begins. Static HTML, JSON, XML, and text responses are supported.

## `extractionRules` (type: `array`):

No-code extraction rules. Each rule needs a name and CSS selector. Use type text, html, or attribute; set multiple to return every match.

## `pageFunction` (type: `string`):

Optional async JavaScript function executed for every page. When supplied, its return value replaces CSS-rule output. Context includes $, request, response, body, json, crawler, Actor, log, and customData.

## `linkSelector` (type: `string`):

CSS selector for links to follow. Leave empty to scrape only Start URLs.

## `globs` (type: `array`):

Only follow links matching these patterns. Empty means all links selected above.

## `excludes` (type: `array`):

Never follow links matching these patterns.

## `sameDomainOnly` (type: `boolean`):

Prevent discovered links from leaving the domains supplied in Start URLs.

## `maxCrawlingDepth` (type: `integer`):

How many link levels beyond each Start URL can be crawled.

## `maxPagesPerCrawl` (type: `integer`):

Stop after processing this many HTTP pages.

## `maxResultsPerCrawl` (type: `integer`):

Stop writing results after this many dataset items.

## `includePageMetadata` (type: `boolean`):

Include title, description, canonical URL, language, H1 headings, and Open Graph image.

## `includeJsonLd` (type: `boolean`):

Parse valid structured data from application/ld+json script elements.

## `removeEmptyFields` (type: `boolean`):

Omit null, empty-string, and empty-array fields from automatic results.

## `proxyConfiguration` (type: `object`):

Use Apify Proxy or your own proxy URLs for protected websites.

## `additionalHeaders` (type: `object`):

Headers sent with every request, such as Authorization or Accept-Language.

## `maxConcurrency` (type: `integer`):

Maximum number of pages fetched at the same time.

## `maxRequestRetries` (type: `integer`):

Retry failed HTTP requests this many times.

## `requestTimeoutSecs` (type: `integer`):

Maximum time allowed for a page request.

## `respectRobotsTxtFile` (type: `boolean`):

Check and honor each site's robots.txt rules.

## `customData` (type: `object`):

Arbitrary values available as context.customData in the page function.

## `debugLog` (type: `boolean`):

Enable verbose logs for troubleshooting.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://books.toscrape.com/"
    }
  ],
  "extractionRules": [
    {
      "name": "pageTitle",
      "selector": "h1",
      "type": "text"
    },
    {
      "name": "products",
      "selector": ".product_pod h3 a",
      "type": "attribute",
      "attribute": "title",
      "multiple": true
    }
  ],
  "pageFunction": "",
  "linkSelector": "a[href]",
  "globs": [
    {
      "glob": "https://books.toscrape.com/catalogue/**"
    }
  ],
  "excludes": [
    {
      "glob": "**/*.{jpg,jpeg,png,gif,svg,pdf,zip}"
    }
  ],
  "sameDomainOnly": true,
  "maxCrawlingDepth": 1,
  "maxPagesPerCrawl": 100,
  "maxResultsPerCrawl": 100,
  "includePageMetadata": true,
  "includeJsonLd": true,
  "removeEmptyFields": true,
  "proxyConfiguration": {
    "useApifyProxy": false
  },
  "additionalHeaders": {},
  "maxConcurrency": 20,
  "maxRequestRetries": 3,
  "requestTimeoutSecs": 45,
  "respectRobotsTxtFile": true,
  "customData": {},
  "debugLog": false
}
```

# Actor output Schema

## `dataset` (type: `string`):

Structured records returned by CSS rules or the page function.

## `output` (type: `string`):

Result count and completion time.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://books.toscrape.com/"
        }
    ],
    "extractionRules": [
        {
            "name": "pageTitle",
            "selector": "h1",
            "type": "text"
        },
        {
            "name": "products",
            "selector": ".product_pod h3 a",
            "type": "attribute",
            "attribute": "title",
            "multiple": true
        }
    ],
    "linkSelector": "a[href]",
    "globs": [
        {
            "glob": "https://books.toscrape.com/catalogue/**"
        }
    ],
    "excludes": [
        {
            "glob": "**/*.{jpg,jpeg,png,gif,svg,pdf,zip}"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("groupoject/cheerio-web-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://books.toscrape.com/" }],
    "extractionRules": [
        {
            "name": "pageTitle",
            "selector": "h1",
            "type": "text",
        },
        {
            "name": "products",
            "selector": ".product_pod h3 a",
            "type": "attribute",
            "attribute": "title",
            "multiple": True,
        },
    ],
    "linkSelector": "a[href]",
    "globs": [{ "glob": "https://books.toscrape.com/catalogue/**" }],
    "excludes": [{ "glob": "**/*.{jpg,jpeg,png,gif,svg,pdf,zip}" }],
}

# Run the Actor and wait for it to finish
run = client.actor("groupoject/cheerio-web-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://books.toscrape.com/"
    }
  ],
  "extractionRules": [
    {
      "name": "pageTitle",
      "selector": "h1",
      "type": "text"
    },
    {
      "name": "products",
      "selector": ".product_pod h3 a",
      "type": "attribute",
      "attribute": "title",
      "multiple": true
    }
  ],
  "linkSelector": "a[href]",
  "globs": [
    {
      "glob": "https://books.toscrape.com/catalogue/**"
    }
  ],
  "excludes": [
    {
      "glob": "**/*.{jpg,jpeg,png,gif,svg,pdf,zip}"
    }
  ]
}' |
apify call groupoject/cheerio-web-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,groupoject/cheerio-web-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/iFgdsVkeB8h8vgVh0/builds/LiSh3zV1B9MuHeLIK/openapi.json