Python Web Scraper — Scrape or Crawl Any Website

🔗 Part of the Apify actors collection — actors that chain: scrape → clean → use.

Scrape any website — or crawl a whole section of one — with your own Python page function. This is a custom Python scraper running a real headless Chromium browser through Playwright, so JavaScript-rendered pages work the same as static ones. You decide what gets extracted; the Actor handles the browser, the crawl queue, the retries and the limits.

You pay $0.003 per record delivered. Nothing else — not pages visited, not retries, not time.

What you get back, and how fast

Your records, not ours. Whatever your page function returns is the dataset record. We add exactly one field and remove none: waitUntilReached — false when the page did not reach your waitUntil state within its share of requestTimeoutSecs, so the content we extracted may be incomplete. It is a fact about the page, not about your extraction, which is why we attach it rather than leaving it to your function; if you already return a field by that name, yours wins.

The default page function returns url, title, text (first 5 000 characters of the body) and textTruncated — true when the page held more text than that. Two flags, two different truths: textTruncated means we stopped reading, waitUntilReached: false means the page had not finished arriving. A cut record must never look like a complete one. A run works before you have written a line of your own.

Measured on our last front-door check: 5 pages of a static site in 28 seconds, one browser, no proxy. JavaScript-heavy pages are slower — that is Chromium rendering, not queue overhead — and a wider crawl scales with maxConcurrency rather than with time per page.

$3.00 per 1 000 records ($0.003 each). A run that delivers nothing costs nothing: pages visited, retries and browser time are not billed.

---

What can this Python web scraper do?

• Scrape a single page — give it a URL and a page function, get a record back.
• Crawl a whole section of a site — follow links by CSS selector, restrict them with glob patterns, stop at a hard request limit.
• Extract exactly the shape you want — your function returns a Python dict, and that dict is the output record. No fixed template to fight, no fields you have to accept.
• Run JavaScript-heavy pages — real Chromium, real rendering, configurable wait condition.
• Chain into other Actors — the output dataset feeds straight into Dataset Deduplicator & Cleaner or any Apify integration.

What websites can I scrape with it?

Any site that a browser can open without credentials, and that allows crawling in its robots.txt.

That includes: documentation sites, product catalogues, listing pages, blogs, news archives, government registers, price pages, job boards that aren't behind a login, and the long tail of sites nobody has written a dedicated scraper for.

It does not include sites behind industrial anti-bot protection or a login — see When is this the wrong tool? below. That section is there because a wasted run costs you money, and we would rather you not spend it.

How do I use it?

Two fields are all you need. The prefilled example works as-is — press Start and you get a record back.

{
  "startUrls": [{ "url": "https://apify.com" }],
  "pageFunction": "async def page_function(page, context, request):\n    return {\n        \"url\": page.url,\n        \"title\": await page.title()\n    }"
}

Result:

{ "url": "https://apify.com", "title": "Apify: Full-stack web scraping and data extraction platform" }

How do I write the page function?

It is an ordinary Python async function with a fixed signature:

async def page_function(page, context, request):
    # page      – Playwright Page instance (page.locator, page.title, page.content, …)
    # context   – Actor context (key-value store, dataset, logging)
    # request   – Current Request object (request.url, request.user_data, …)
    # Return a dict, or a list of dicts. Each dict becomes one result item.
    return {
        "url": page.url,
        "title": await page.title(),
        "h1": await page.locator("h1").first.inner_text(),
    }

Rules worth knowing:

• Return a dict → one record. Return a list of dicts → many records from one page.
• Return None → no record from this page, and no charge. That is the supported way of saying "this page had nothing for me".
• The function is compiled before the crawl starts. A syntax error fails the run in the first second, not after an hour of crawling.
• An exception thrown for one page does not kill the run. That page is counted as an error, the crawl continues, and the count is reported in the final status message.

How do I crawl and follow links?

{
  "startUrls": [{ "url": "https://example.com/products" }],
  "linkSelector": "a.product-link",
  "includeGlobs": ["https://example.com/products/*"],
  "maxRequestsPerCrawl": 200,
  "pageFunction": "async def page_function(page, context, request):\n    return {\n        \"url\": page.url,\n        \"name\": await page.locator(\"h1\").inner_text(),\n        \"price\": await page.locator(\".price\").inner_text()\n    }"
}

• linkSelector finds links on each visited page.
• includeGlobs decides which of those links are worth enqueueing. Without it, a crawl will happily wander off into the rest of the site — and every page it visits is a page you waited for.
• maxRequestsPerCrawl is a hard stop on total requests.

⬇️ Input

Field	Type	Default	Description
startUrls	array of objects	—	Seed URLs, as { "url": "…" }. At least one.
pageFunction	string	—	Python source of the async page function.
linkSelector	string	—	CSS selector for links to follow. Empty = no crawling.
includeGlobs	array of strings	—	Glob patterns a URL must match to be enqueued.
maxRequestsPerCrawl	integer	100	Hard limit on requests for the whole run.
maxConcurrency	integer	5	Parallel browser contexts.
requestTimeoutSecs	integer	90	Budget for one page: reaching waitUntil, then extraction and your page function. Measured: at 45 s every page on a documentation site ran out of time before load; at a larger budget none did — and the crawl finished faster.
waitUntil	string	load	Page state to reach before extracting: load, domcontentloaded, networkidle. Reached on a bounded share of requestTimeoutSecs; pages that run out of time are still extracted and counted in the run status.
respectRobotsTxt	boolean	true	Obey robots.txt. Disallowed URLs are skipped before being visited.
proxyConfiguration	object	Apify proxy	Proxy settings.
blockResources	boolean	true	Drop images, fonts, media and stylesheets before they are downloaded. They do not change the DOM you read, and you pay for the bytes. Turn it off only if your page function needs to look at an image or a computed style.
sessionMode	string	auto	auto lets Crawlee rotate sessions and retire the ones a site starts refusing. sticky keeps one small pool so a login or a cart survives across pages. off disables the pool entirely.
maxItems	integer	100	Stop after this many records — and since you pay per record, this is your cost ceiling: 100 records is $0.30. Set 0 for no cap, up to the 50 000 safety limit per run.
deliverEmptyRecords	boolean	false	Deliver — and charge for — records from pages that came back with nothing (a bot wall, a timeout mid-render). Off by default: an empty record costs you full price for no data. Turn it on when you want a row for every page attempted.

⬆️ Output — what you get back

The shape of each record is yours. Whatever dictionary your page function returns is written to the dataset unchanged. That is the whole point of this Actor, and it is why it ships no fixed output schema: any schema would be a promise about code you wrote, not code we wrote.

One record per page

{ "url": "https://example.com/products/wrench", "name": "Torque wrench 1/2\"", "price": "€89.00" }

Many records from one page

Return a list, and each element becomes its own record — useful for a listing page:

async def page_function(page, context, request):
    rows = []
    for card in await page.locator(".product-card").all():
        rows.append({
            "name": await card.locator(".name").inner_text(),
            "price": await card.locator(".price").inner_text(),
        })
    return rows

What the Actor guarantees, and what your page function decides

The Actor guarantees:

• every record you return is written exactly once, and charged exactly once;
• a page that throws is isolated — the crawl continues and you get the records that worked;
• maxRequestsPerCrawl and maxItems are hard stops, so a crawl cannot run away with your budget;
• requestTimeoutSecs bounds a page that never settles;
• a run that delivers nothing because something broke fails loudly instead of reporting success.

Your page function decides everything else: which elements to read, how to name the fields, what to skip.

What happens when something fails?

Errors are not written into your dataset as fake records — you are never charged for a failure. They are counted and reported, so a run always tells you what actually happened.

Situation	What the Actor does
pageFunction has a syntax error	Fails before the crawl starts, with the syntax error in the status message.
startUrls is empty	Fails immediately with a clear message.
One page throws inside your function	Counted as a page error, crawl continues. Reported as (N page(s) errored in the page function).
A URL is unreachable after retries	Counted as a failed request. Reported as (N request(s) failed at network/navigation).
Zero records, and pages errored or requests failed	The run fails, with the reason: every page errored in the page function, or all requests failed (site unreachable/blocked?). A broken run never reports success.
Zero records, and nothing errored	Succeeds with Scraped 0 item(s) — your selectors matched nothing. Costs you nothing.
maxItems reached	Stops cleanly, status says (reached maxItems=N).
Billing call fails repeatedly	The run stops rather than doing unpaid work, and says how many items it delivered first.

That fifth row is the one that matters most. A scraper that returns an empty dataset and calls it success is the single most expensive failure mode there is, because your pipeline keeps running on nothing. This Actor refuses to do it.

How much does it cost?

• $0.003 per record delivered (result-item), flat, regardless of how many pages were walked to find it.
• An empty run costs nothing. No record, no charge.
• Nothing is charged for pages visited, retries, browser time, or bandwidth.
• To try it cheaply, set a small maxItems — you pay only for what you actually receive.

For 1 000 records that is $3.00. A crawl of 200 pages that yields 200 records costs $0.60, whether those pages took two minutes or twenty.

When is this the wrong tool?

Be honest with yourself about the target before you spend a run:

• Sites behind industrial anti-bot protection — major marketplaces, search engines, large social networks. They rate-limit or block datacenter IP addresses regardless of how good the browser automation is. Those targets need a residential proxy, and a generic crawler is not the reason they work or fail.
• Content behind a login. This Actor does not carry your credentials, by design.
• A site somebody already solved well. If a dedicated Actor exists for your target, it will handle that site's quirks better than a page function you write today.

Use this when you need your own extraction logic on a site nobody has built a dedicated tool for.

FAQ

Does this Actor get access to my Apify account?

No. It runs with limited permissions, which means it can read its own input and write to its own run's storage, and nothing else. Your other datasets, key-value stores, tokens and Actors are out of reach.

This is worth checking before you run anything, not just this Actor. Some Actors — including a well-known general-purpose scraper — require full permissions, and running one grants it access to your whole account. That is not carelessness on their part: an Actor that lets you paste your own JavaScript has to hand that code real credentials. But it is a decision you should make knowingly, and Apify asks you to approve it once, in a dialog that is easy to click past.

You can verify any Actor for yourself:

curl -s -H "Authorization: Bearer $APIFY_TOKEN" \
  https://api.apify.com/v2/acts/eszetael_lab~reliable-playwright-scraper \
  | grep -o '"actorPermissionLevel":"[^"]*"'

Because this Actor runs your page function inside your own run, it never needs more than that.

Can I use it with the Apify API?

Yes. Start it like any Actor — POST /v2/acts/eszetael_lab~reliable-playwright-scraper/runs with your input as the JSON body, then read the run's dataset. Everything this Actor does is available through the standard API, CLI and client libraries.

Can I use it through an MCP server?

Yes. It is exposed through Apify's Actors MCP server like any other public Actor, so an AI agent can call it as a tool. It is also enabled for agentic payments, meaning an agent can run it and be charged directly without a human in the loop.

Can I schedule it to run every day?

Yes — use Apify Schedules. A common pattern is a daily crawl with a maxItems cap, chained into the Dataset Deduplicator & Cleaner so you only ever look at what is new.

Does it respect robots.txt?

By default, yes. Disallowed URLs are skipped before they are visited, so a blocked page never costs you a request. Turn respectRobotsTxt off only for sites you own or are otherwise authorised to crawl.

Can it log in to a website?

No, and that is deliberate. This Actor does not carry credentials. If a site requires a session, this is the wrong tool.

Where does my page function run?

Inside your own run, on your own account, like any other code you run on the platform. A per-page timeout bounds runaway code, and maxRequestsPerCrawl bounds runaway crawls.

Is it legal to scrape with this Actor?

Scraping publicly available data is broadly lawful in the EU and the US, but "broadly" is not "always". You are responsible for ensuring your use complies with applicable law and the target site's terms of service, and for not collecting personal data you have no basis to collect. robots.txt is respected by default because it is the site's own machine-readable statement of what it permits.

Your feedback

Found a bug, or a site where this behaves badly? Open an issue on the Actor's Issues tab. Real failure reports are worth more to us than feature requests.

Ready-made setups

Each of these is this Actor with the input already filled in — open it, press Run, then change the target to yours. No configuration to read first.

• Collect Titles and Meta Descriptions Across a Site
• Crawl a Website to Clean Text for RAG
• Extract Every Link From a Website
• Extract HTML Tables From Web Pages as JSON
• Find Which Pages of a Site Mention a Keyword
• Scrape a Page That Only Renders With JavaScript
• Get the Text and Title of a Single Web Page

Related actors

Three tools built to chain into each other — scrape, then clean, then use.

• Dataset Deduplicator & Cleaner — pass this Actor's dataset ID straight in to remove duplicates across runs and clean the fields before analysis. Six times cheaper than a scraper, because it processes data you already paid for.
• Bluesky Scraper — when your target is Bluesky rather than a website, use the protocol directly instead of a browser: no login, no proxy, and an incremental mode that returns only new posts.

All three are on pay-per-result pricing, and an empty run costs nothing in every one of them.

If it worked for you

Ratings feed the Apify quality score, which decides whether anyone finds this Actor at all — and it is the one part we cannot build ourselves. If it did the job, a rating takes a few seconds.

If it did not, the issue tab is more useful than a low rating with no detail: tell us the input that failed and what you expected, and it becomes a test case in the next release.