# Rust Web Scraper (`automation-lab/rust-web-scraper`) Actor

Crawl anonymous public URLs with a native Rust HTTP crawler and export URL, status, title, text, links, timing, and selected CSS fields.

- **URL**: https://apify.com/automation-lab/rust-web-scraper.md
- **Developed by:** [Stas Persiianenko](https://apify.com/automation-lab) (community)
- **Categories:** Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.44 / 1,000 item processeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Rust Web Scraper

**Rust Web Scraper** crawls supplied anonymous public URLs with a native asynchronous Rust HTTP worker. It exports one typed record per fetched page with the requested and final URL, HTTP status, title, normalized text, links, custom CSS-selected fields, crawl depth, timestamp, and response time.

Use it when you need a fast, bounded HTTP crawl for public pages without paying for a browser on every request. The crawler is source-agnostic: provide one page, several unrelated sites, or same-host starting points for a scheduled content pipeline.

### What does Rust Web Scraper do?

The Actor combines:

- a native Tokio and Reqwest worker for concurrent HTTP requests;
- structural HTML parsing with the Rust `scraper` crate;
- same-host link discovery with page and depth limits;
- named CSS selectors for reusable structured fields;
- Apify datasets, schedules, API access, webhooks, and integrations;
- pay-per-event billing only for the run start and accepted page records.

It does not start a browser or residential proxy automatically. This keeps ordinary public-page collection quick and predictable.

### Who is it for?

- **Developers** building repeatable web datasets or testing Rust-powered collection.
- **Data engineers** feeding normalized page text into ETL, search, or RAG pipelines.
- **Researchers** collecting a bounded set of public project or documentation pages.
- **Content teams** exporting titles, links, headings, and custom fields to a spreadsheet.
- **Automation builders** scheduling the same URL set and comparing datasets downstream.

### Why use a native Rust crawler?

The page-fetching and parsing work runs in a compiled Rust binary. Concurrency is capped at 6 and each response is streamed into a 2 MB bounded buffer, preserving headroom within the Actor's 256 MB allocation. The TypeScript adapter is intentionally small and uses the Apify SDK for lifecycle, billing, and dataset writes.

This implementation favors deterministic public HTTP content over browser emulation. You control concurrency, retries, depth, body-text size, and the exact number of page records.

### What data can I extract?

| Field | Description |
| --- | --- |
| `url` | Normalized URL that was queued |
| `finalUrl` | URL after validated redirects |
| `statusCode` | Final HTTP response status |
| `contentType` | Response content type, when supplied |
| `title` | HTML document title, or `null` |
| `text` | Normalized body text, truncated by `maxTextChars` |
| `links` | Unique normalized absolute HTTP(S) links |
| `selected` | Arrays keyed by your named CSS field definitions |
| `depth` | Link distance from a supplied start URL |
| `fetchedAt` | ISO 8601 fetch timestamp |
| `responseTimeMs` | Request and parsing duration in milliseconds |

Non-HTML responses can still produce status and content-type evidence, but HTML-only fields remain empty.

### How to scrape public websites

1. Open the Actor input page.
2. Add one or more public HTTP(S) start URLs.
3. Set `maxPages` and `maxDepth` for a bounded crawl.
4. Keep `sameDomainOnly` enabled unless you intentionally need cross-domain discovery.
5. Optionally define named CSS fields.
6. Run the Actor and open the **Crawled pages** dataset view.
7. Export JSON, CSV, Excel, XML, or RSS from the dataset API.

A useful first run is the prefilled Rust Learn page with five pages and one link level.

### Input parameters

| Input | Type | Default | Purpose |
| --- | --- | ---: | --- |
| `startUrls` | array | required | Up to 100 public HTTP(S) starting pages |
| `maxPages` | integer | `25` | Global record limit, from 1 to 10,000 |
| `maxDepth` | integer | `1` | Link levels to follow; `0` fetches only supplied URLs |
| `maxConcurrency` | integer | `6` | Simultaneous requests, from 1 to 6; capped for predictable memory use |
| `sameDomainOnly` | boolean | `true` | Restrict discovered links to supplied hostnames |
| `includeLinks` | boolean | `true` | Include extracted links in each record |
| `maxTextChars` | integer | `100000` | Per-page normalized text limit; `0` omits text |
| `selectors` | array | `[]` | Up to 20 named CSS field definitions |
| `requestTimeoutSecs` | integer | `30` | Timeout for an individual attempt |
| `maxRetries` | integer | `2` | Bounded transient retries |
| `userAgent` | string | Actor agent | Optional responsible crawler identity |

Each CSS field accepts `name`, `selector`, optional `attribute`, and `multiple`. Without `attribute`, the Actor extracts normalized element text.

### CSS field example

```json
{
  "startUrls": [{ "url": "https://www.rust-lang.org/learn" }],
  "maxPages": 5,
  "maxDepth": 1,
  "maxConcurrency": 4,
  "selectors": [
    { "name": "headings", "selector": "h1, h2", "multiple": true },
    { "name": "canonical", "selector": "link[rel=canonical]", "attribute": "href" }
  ]
}
```

Selector names become keys under `selected`. Every value is an array so downstream schemas remain stable whether there are zero, one, or many matches.

### Output example

This shortened record reflects the current Rust Learn page behavior:

```json
{
  "url": "https://www.rust-lang.org/learn",
  "finalUrl": "https://www.rust-lang.org/learn",
  "statusCode": 200,
  "contentType": "text/html; charset=utf-8",
  "title": "Learn Rust - Rust Programming Language",
  "text": "Affectionately nicknamed the book...",
  "links": ["https://www.rust-lang.org/tools/install"],
  "selected": {
    "headings": ["Learn Rust", "Get started with Rust", "Documentation", "Master Rust"],
    "canonical": []
  },
  "depth": 0,
  "fetchedAt": "2026-08-30T20:00:00.000Z",
  "responseTimeMs": 184
}
```

The full `text` and `links` arrays are preserved in the dataset unless their inputs disable or truncate them.

### How much does it cost to scrape public web pages?

The Actor uses pay-per-event pricing:

- one `start` event per run;
- one `item` event for each accepted page record;
- failed, rejected, duplicate, or unsaved pages do not create an item event.

Your Apify plan tier determines the exact event price shown in Console before a run. A tight `maxPages` value is the simplest budget cap. Platform compute is included under the Actor's applicable pay-per-event pricing configuration.

### Performance tips

- Start with concurrency 3–6 and reduce it when a target rate-limits requests.
- Use `maxDepth: 0` for a supplied URL batch with no discovery.
- Keep `sameDomainOnly` enabled to avoid an accidental web-wide crawl.
- Set `includeLinks: false` when only text or CSS fields matter.
- Lower `maxTextChars` for compact datasets and faster downstream processing.
- Split very different sites into separate scheduled tasks when they need different rates.
- Use stable CSS selectors tied to document structure rather than volatile classes.

### Limits and failure behavior

The Actor supports anonymous public HTTP(S) pages. It does not claim support for:

- login-required or account-specific content;
- CAPTCHAs and anti-bot challenges;
- JavaScript-only content that is absent from server HTML;
- residential-proxy or browser fallback;
- responses larger than the 2 MB streamed-body cap;
- private, loopback, link-local, or reserved network targets.

Redirects are bounded and each destination is validated. Invalid input and exhausted terminal request failures cause a non-zero Actor run rather than silently returning a misleading empty success.

### Scheduled collection workflow

Create an Apify Task with a fixed URL set and schedule it hourly, daily, or weekly. Each run receives its own dataset. Connect a webhook or integration to process successful datasets, then compare `title`, `text`, `links`, or `selected` fields in your own database.

The Actor does not maintain change history itself. Dataset retention and comparison remain explicit downstream steps, which avoids hidden state between runs.

### Integrations

Results work with:

- Apify dataset exports in JSON, CSV, Excel, XML, and RSS;
- Google Sheets and other Apify integrations;
- Make, Zapier, and n8n through Apify modules or HTTP calls;
- webhooks that trigger after successful runs;
- vector stores and RAG pipelines after your preferred chunking stage;
- custom applications using the Apify API client.

### Run with the Apify API

Replace `APIFY_TOKEN` with a token stored securely outside source code.

#### cURL

```bash
curl -X POST \
  "https://api.apify.com/v2/acts/automation-lab~rust-web-scraper/run-sync-get-dataset-items?token=APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://www.rust-lang.org/learn"}],"maxPages":5,"maxDepth":1}'
```

#### JavaScript

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/rust-web-scraper').call({
  startUrls: [{ url: 'https://www.rust-lang.org/learn' }],
  maxPages: 5,
  maxDepth: 1,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);
```

#### Python

```python
from apify_client import ApifyClient
import os

client = ApifyClient(os.environ['APIFY_TOKEN'])
run = client.actor('automation-lab/rust-web-scraper').call(run_input={
    'startUrls': [{'url': 'https://www.rust-lang.org/learn'}],
    'maxPages': 5,
    'maxDepth': 1,
})
items = client.dataset(run['defaultDatasetId']).list_items().items
print(items)
```

### Use with MCP and AI assistants

Add the Actor to Claude Code:

```bash
claude mcp add --transport http apify \
  "https://mcp.apify.com?tools=automation-lab/rust-web-scraper"
```

#### Claude Desktop, Cursor, and VS Code

Use the same remote MCP server URL in Claude Desktop, Cursor, or VS Code:

```json
{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=automation-lab/rust-web-scraper"
    }
  }
}
```

Example prompts:

- “Use Rust Web Scraper to fetch the Rust Learn page and return all H1 and H2 text.”
- “Crawl up to 20 same-domain pages from this public documentation URL and save titles and links.”
- “Fetch these three public URLs concurrently and export normalized text for my data pipeline.”

### Responsible use and legality

Only collect content you are authorized to access. Review the target site's terms, robots policy, copyright rules, privacy requirements, and applicable law. Use conservative concurrency and identify your crawler when appropriate. Do not use this Actor to access private networks, evade authentication, bypass technical restrictions, or collect sensitive personal data.

You are responsible for your input URLs, collection purpose, retention, and downstream use.

### Troubleshooting

#### Why did the run fail on a URL that opens in my browser?

The page may require JavaScript, cookies, login, a browser fingerprint, or an anti-bot challenge. This Actor intentionally performs direct anonymous HTTP requests. Use a public server-rendered URL or a dedicated source Actor when the source needs browser behavior.

#### Why did I receive fewer pages than `maxPages`?

`maxPages` is a ceiling, not a promise. The crawl may exhaust unique links, stay within a restrictive hostname, encounter non-HTML pages, or stop at the configured depth.

#### Why is a selected field empty?

Check the raw server HTML, not only the rendered browser DOM. Verify selector case and structure, or set `attribute` when the value lives in an HTML attribute.

### FAQ

#### Is the crawler really written in Rust?

Yes. Network fetching, concurrency, redirects, link discovery, HTML parsing, and field extraction run in a compiled Rust binary. The Apify SDK adapter streams records into the platform.

#### Can it crawl several websites in one run?

Yes. Supply up to 100 start URLs. The global `maxPages` limit applies across all of them.

#### Does it execute JavaScript?

No. It extracts server-delivered HTML. This is a deliberate throughput and cost choice.

#### Can it use a proxy?

The initial version does not expose or automatically enable proxy routing. There is no unmeasured residential fallback.

#### Can I export to Excel?

Yes. Open the run dataset and choose Excel, CSV, JSON, XML, or another supported dataset format.

### Related automation-lab Actors

For specialized workflows, consider these automation-lab Actors:

- [Sitewide Broken Link Checker](https://apify.com/automation-lab/sitewide-broken-link-checker) for link availability and failure reporting.
- [Multi-page On-page SEO Audit Crawler](https://apify.com/automation-lab/multi-page-on-page-seo-audit) for deterministic SEO fields and issue flags.
- [Website HTML & Text Change Monitor](https://apify.com/automation-lab/website-html-text-change-monitor) for versioned snapshots and machine-readable changes.

Choose this Rust Web Scraper when the buyer job is a general bounded public-page dataset with custom CSS fields rather than a specialized audit or monitor.

### Support

For a reproducible report, include the public URL, input with secrets removed, run ID, expected field, actual field, and whether the content appears in server HTML. Do not post access tokens, cookies, private URLs, or personal data.

# Actor input Schema

## `startUrls` (type: `array`):

Public HTTP(S) pages where crawling begins. Hostnames resolving to private or reserved networks are rejected.

## `maxPages` (type: `integer`):

Maximum number of page records across all start URLs.

## `maxDepth` (type: `integer`):

How many link levels to follow from each start page. Use 0 to fetch only supplied URLs.

## `maxConcurrency` (type: `integer`):

Maximum simultaneous Rust HTTP requests. Capped at 6 to preserve memory headroom within the 256 MB Actor allocation.

## `sameDomainOnly` (type: `boolean`):

Follow only links whose hostname matches one of the supplied start URLs.

## `includeLinks` (type: `boolean`):

Include normalized absolute page links in each output record.

## `maxTextChars` (type: `integer`):

Truncate normalized visible body text to this length. Use 0 to omit text.

## `selectors` (type: `array`):

Optional named CSS selectors. Each field exports text or one HTML attribute as an array.

## `requestTimeoutSecs` (type: `integer`):

Timeout for each HTTP request attempt.

## `maxRetries` (type: `integer`):

Bounded retries for transient request and server failures.

## `userAgent` (type: `string`):

Optional HTTP User-Agent override. Respect each target site's terms and crawl policy.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://www.rust-lang.org/"
    }
  ],
  "maxPages": 20,
  "maxDepth": 1,
  "maxConcurrency": 6,
  "sameDomainOnly": true,
  "includeLinks": true,
  "maxTextChars": 100000,
  "selectors": [],
  "requestTimeoutSecs": 30,
  "maxRetries": 2
}
```

# Actor output Schema

## `overview` (type: `string`):

Open URL, status, title, text, links, and selected CSS fields in the overview view.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://www.rust-lang.org/"
        }
    ],
    "maxPages": 20
};

// Run the Actor and wait for it to finish
const run = await client.actor("automation-lab/rust-web-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://www.rust-lang.org/" }],
    "maxPages": 20,
}

# Run the Actor and wait for it to finish
run = client.actor("automation-lab/rust-web-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://www.rust-lang.org/"
    }
  ],
  "maxPages": 20
}' |
apify call automation-lab/rust-web-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,automation-lab/rust-web-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/IEeA6Z629JRXc5byy/builds/axWkHR2Lc2CMgtZiL/openapi.json