Rust Web Scraper

Rust Web Scraper crawls supplied anonymous public URLs with a native asynchronous Rust HTTP worker. It exports one typed record per fetched page with the requested and final URL, HTTP status, title, normalized text, links, custom CSS-selected fields, crawl depth, timestamp, and response time.

Use it when you need a fast, bounded HTTP crawl for public pages without paying for a browser on every request. The crawler is source-agnostic: provide one page, several unrelated sites, or same-host starting points for a scheduled content pipeline.

What does Rust Web Scraper do?

The Actor combines:

• a native Tokio and Reqwest worker for concurrent HTTP requests;
• structural HTML parsing with the Rust scraper crate;
• same-host link discovery with page and depth limits;
• named CSS selectors for reusable structured fields;
• Apify datasets, schedules, API access, webhooks, and integrations;
• pay-per-event billing only for the run start and accepted page records.

It does not start a browser or residential proxy automatically. This keeps ordinary public-page collection quick and predictable.

Who is it for?

• Developers building repeatable web datasets or testing Rust-powered collection.
• Data engineers feeding normalized page text into ETL, search, or RAG pipelines.
• Researchers collecting a bounded set of public project or documentation pages.
• Content teams exporting titles, links, headings, and custom fields to a spreadsheet.
• Automation builders scheduling the same URL set and comparing datasets downstream.

Why use a native Rust crawler?

The page-fetching and parsing work runs in a compiled Rust binary. Concurrency is capped at 6 and each response is streamed into a 2 MB bounded buffer, preserving headroom within the Actor's 256 MB allocation. The TypeScript adapter is intentionally small and uses the Apify SDK for lifecycle, billing, and dataset writes.

This implementation favors deterministic public HTTP content over browser emulation. You control concurrency, retries, depth, body-text size, and the exact number of page records.

What data can I extract?

Field	Description
url	Normalized URL that was queued
finalUrl	URL after validated redirects
statusCode	Final HTTP response status
contentType	Response content type, when supplied
title	HTML document title, or null
text	Normalized body text, truncated by maxTextChars
links	Unique normalized absolute HTTP(S) links
selected	Arrays keyed by your named CSS field definitions
depth	Link distance from a supplied start URL
fetchedAt	ISO 8601 fetch timestamp
responseTimeMs	Request and parsing duration in milliseconds

Non-HTML responses can still produce status and content-type evidence, but HTML-only fields remain empty.

How to scrape public websites

1. Open the Actor input page.
2. Add one or more public HTTP(S) start URLs.
3. Set maxPages and maxDepth for a bounded crawl.
4. Keep sameDomainOnly enabled unless you intentionally need cross-domain discovery.
5. Optionally define named CSS fields.
6. Run the Actor and open the Crawled pages dataset view.
7. Export JSON, CSV, Excel, XML, or RSS from the dataset API.

A useful first run is the prefilled Rust Learn page with five pages and one link level.

Input parameters

Input	Type	Default	Purpose
startUrls	array	required	Up to 100 public HTTP(S) starting pages
maxPages	integer	25	Global record limit, from 1 to 10,000
maxDepth	integer	1	Link levels to follow; 0 fetches only supplied URLs
maxConcurrency	integer	6	Simultaneous requests, from 1 to 6; capped for predictable memory use
sameDomainOnly	boolean	true	Restrict discovered links to supplied hostnames
includeLinks	boolean	true	Include extracted links in each record
maxTextChars	integer	100000	Per-page normalized text limit; 0 omits text
selectors	array	[]	Up to 20 named CSS field definitions
requestTimeoutSecs	integer	30	Timeout for an individual attempt
maxRetries	integer	2	Bounded transient retries
userAgent	string	Actor agent	Optional responsible crawler identity

Each CSS field accepts name, selector, optional attribute, and multiple. Without attribute, the Actor extracts normalized element text.

CSS field example

{
  "startUrls": [{ "url": "https://www.rust-lang.org/learn" }],
  "maxPages": 5,
  "maxDepth": 1,
  "maxConcurrency": 4,
  "selectors": [
    { "name": "headings", "selector": "h1, h2", "multiple": true },
    { "name": "canonical", "selector": "link[rel=canonical]", "attribute": "href" }
  ]
}

Selector names become keys under selected. Every value is an array so downstream schemas remain stable whether there are zero, one, or many matches.

Output example

This shortened record reflects the current Rust Learn page behavior:

{
  "url": "https://www.rust-lang.org/learn",
  "finalUrl": "https://www.rust-lang.org/learn",
  "statusCode": 200,
  "contentType": "text/html; charset=utf-8",
  "title": "Learn Rust - Rust Programming Language",
  "text": "Affectionately nicknamed the book...",
  "links": ["https://www.rust-lang.org/tools/install"],
  "selected": {
    "headings": ["Learn Rust", "Get started with Rust", "Documentation", "Master Rust"],
    "canonical": []
  },
  "depth": 0,
  "fetchedAt": "2026-08-30T20:00:00.000Z",
  "responseTimeMs": 184
}

The full text and links arrays are preserved in the dataset unless their inputs disable or truncate them.

How much does it cost to scrape public web pages?

The Actor uses pay-per-event pricing:

• one start event per run;
• one item event for each accepted page record;
• failed, rejected, duplicate, or unsaved pages do not create an item event.

Your Apify plan tier determines the exact event price shown in Console before a run. A tight maxPages value is the simplest budget cap. Platform compute is included under the Actor's applicable pay-per-event pricing configuration.

Performance tips

• Start with concurrency 3–6 and reduce it when a target rate-limits requests.
• Use maxDepth: 0 for a supplied URL batch with no discovery.
• Keep sameDomainOnly enabled to avoid an accidental web-wide crawl.
• Set includeLinks: false when only text or CSS fields matter.
• Lower maxTextChars for compact datasets and faster downstream processing.
• Split very different sites into separate scheduled tasks when they need different rates.
• Use stable CSS selectors tied to document structure rather than volatile classes.

Limits and failure behavior

The Actor supports anonymous public HTTP(S) pages. It does not claim support for:

• login-required or account-specific content;
• CAPTCHAs and anti-bot challenges;
• JavaScript-only content that is absent from server HTML;
• residential-proxy or browser fallback;
• responses larger than the 2 MB streamed-body cap;
• private, loopback, link-local, or reserved network targets.

Redirects are bounded and each destination is validated. Invalid input and exhausted terminal request failures cause a non-zero Actor run rather than silently returning a misleading empty success.

Scheduled collection workflow

Create an Apify Task with a fixed URL set and schedule it hourly, daily, or weekly. Each run receives its own dataset. Connect a webhook or integration to process successful datasets, then compare title, text, links, or selected fields in your own database.

The Actor does not maintain change history itself. Dataset retention and comparison remain explicit downstream steps, which avoids hidden state between runs.

Integrations

Results work with:

• Apify dataset exports in JSON, CSV, Excel, XML, and RSS;
• Google Sheets and other Apify integrations;
• Make, Zapier, and n8n through Apify modules or HTTP calls;
• webhooks that trigger after successful runs;
• vector stores and RAG pipelines after your preferred chunking stage;
• custom applications using the Apify API client.

Run with the Apify API

Replace APIFY_TOKEN with a token stored securely outside source code.

cURL

curl -X POST \
  "https://api.apify.com/v2/acts/automation-lab~rust-web-scraper/run-sync-get-dataset-items?token=APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://www.rust-lang.org/learn"}],"maxPages":5,"maxDepth":1}'

JavaScript

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/rust-web-scraper').call({
  startUrls: [{ url: 'https://www.rust-lang.org/learn' }],
  maxPages: 5,
  maxDepth: 1,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Python

from apify_client import ApifyClient
import os

client = ApifyClient(os.environ['APIFY_TOKEN'])
run = client.actor('automation-lab/rust-web-scraper').call(run_input={
    'startUrls': [{'url': 'https://www.rust-lang.org/learn'}],
    'maxPages': 5,
    'maxDepth': 1,
})
items = client.dataset(run['defaultDatasetId']).list_items().items
print(items)

Use with MCP and AI assistants

Add the Actor to Claude Code:

claude mcp add --transport http apify \
  "https://mcp.apify.com?tools=automation-lab/rust-web-scraper"

Claude Desktop, Cursor, and VS Code

Use the same remote MCP server URL in Claude Desktop, Cursor, or VS Code:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=automation-lab/rust-web-scraper"
    }
  }
}

Example prompts:

• “Use Rust Web Scraper to fetch the Rust Learn page and return all H1 and H2 text.”
• “Crawl up to 20 same-domain pages from this public documentation URL and save titles and links.”
• “Fetch these three public URLs concurrently and export normalized text for my data pipeline.”

Responsible use and legality

Only collect content you are authorized to access. Review the target site's terms, robots policy, copyright rules, privacy requirements, and applicable law. Use conservative concurrency and identify your crawler when appropriate. Do not use this Actor to access private networks, evade authentication, bypass technical restrictions, or collect sensitive personal data.

You are responsible for your input URLs, collection purpose, retention, and downstream use.

Troubleshooting

Why did the run fail on a URL that opens in my browser?

The page may require JavaScript, cookies, login, a browser fingerprint, or an anti-bot challenge. This Actor intentionally performs direct anonymous HTTP requests. Use a public server-rendered URL or a dedicated source Actor when the source needs browser behavior.

Why did I receive fewer pages than maxPages?

maxPages is a ceiling, not a promise. The crawl may exhaust unique links, stay within a restrictive hostname, encounter non-HTML pages, or stop at the configured depth.

Why is a selected field empty?

Check the raw server HTML, not only the rendered browser DOM. Verify selector case and structure, or set attribute when the value lives in an HTML attribute.

FAQ

Is the crawler really written in Rust?

Yes. Network fetching, concurrency, redirects, link discovery, HTML parsing, and field extraction run in a compiled Rust binary. The Apify SDK adapter streams records into the platform.

Can it crawl several websites in one run?

Yes. Supply up to 100 start URLs. The global maxPages limit applies across all of them.

Does it execute JavaScript?

No. It extracts server-delivered HTML. This is a deliberate throughput and cost choice.

Can it use a proxy?

The initial version does not expose or automatically enable proxy routing. There is no unmeasured residential fallback.

Can I export to Excel?

Yes. Open the run dataset and choose Excel, CSV, JSON, XML, or another supported dataset format.

Related automation-lab Actors

For specialized workflows, consider these automation-lab Actors:

• Sitewide Broken Link Checker for link availability and failure reporting.
• Multi-page On-page SEO Audit Crawler for deterministic SEO fields and issue flags.
• Website HTML & Text Change Monitor for versioned snapshots and machine-readable changes.

Choose this Rust Web Scraper when the buyer job is a general bounded public-page dataset with custom CSS fields rather than a specialized audit or monitor.

Support

For a reproducible report, include the public URL, input with secrets removed, run ID, expected field, actual field, and whether the content appears in server HTML. Do not post access tokens, cookies, private URLs, or personal data.