# Crawl4AI (`janbuchar/crawl4ai`) Actor

Wraps the Crawl4AI open-source library for retrieving text content from websites.

- **URL**: https://apify.com/janbuchar/crawl4ai.md
- **Developed by:** [Jan Buchar](https://apify.com/janbuchar) (community)
- **Categories:** AI
- **Stats:** 809 total users, 3 monthly users, 100.0% runs succeeded, 16 bookmarks
- **User rating**: 3.26 out of 5 stars

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## 🔥🕷️ Crawl4AI

An Apify Actor wrapping [Crawl4AI](https://crawl4ai.com/), the open-source LLM-friendly web crawler and scraper.

The Actor can:

- Crawl and scrape websites with a real browser, following links to a configurable depth.
- Produce clean Markdown, suitable for RAG pipelines or direct ingestion into large language models.
- Extract structured data using CSS selectors or an LLM of your choice.
- Route traffic through Apify Proxy, and resume where it left off if the run gets migrated.
- Scale concurrency to the memory you give the run, instead of guessing.

### Concurrency and memory

**Maximum concurrency** is an upper bound. The crawler watches how much of the run's memory allocation is
actually in use and stops opening new pages as it gets close, resuming once memory frees up — so a value that
is slightly too ambitious costs you speed rather than a crashed run.

If you see `deferred, low memory` in the run status, give the run more memory or lower the concurrency; pages
deferred that way can end up skipped.

### Usage

1. Enter your start URLs.
2. Choose how far the crawler should follow links, and optionally narrow the crawl down with URL globs or
   allowed domains.
3. Pick an extraction strategy if you want structured data on top of the Markdown.
4. Run the Actor.

### How much will it cost?

Apify provides $5 free usage credits every month on the [Apify Free plan](https://apify.com/pricing). With
Crawl4AI, you can enjoy a certain number of results per month for free.

For larger data needs, consider upgrading to the [$49/month Starter plan](https://apify.com/pricing) for
increased monthly results volume. Or opt for the [Scale plan](https://apify.com/pricing) for even higher result
limits.

### Results

Every crawled page becomes one dataset item. The page content itself is stored in the run's key-value store,
and the dataset links to it:

```json
[
    {
        "url": "https://docs.crawl4ai.com/",
        "success": true,
        "statusCode": 200,
        "depth": 0,
        "parentUrl": null,
        "markdown": "https://api.apify.com/v2/key-value-stores/m1Sqnke1KWM0AI8co/records/content_4242.md",
        "html": "https://api.apify.com/v2/key-value-stores/m1Sqnke1KWM0AI8co/records/content_4242.html",
        "screenshot_url": null,
        "pdf_url": null,
        "extracted_content": null,
        "metadata": {
            "title": "Home - Crawl4AI Documentation",
            "description": "🚀🤖 Crawl4AI, Open-source LLM-Friendly Web Crawler & Scraper"
        }
    }
]
```

If you enable a **Markdown content filter**, a pruned `fitMarkdown` link is added alongside the raw one.

### Crawl strategies

- **Breadth-first** — visits every page at one depth before going deeper. The sane default.
- **Depth-first** — follows one branch as far as it goes. Crawls one page at a time, so it is slower.
- **Best-first** — pages whose URL matches your **Keywords** jump the queue. Use it when you only have budget
  for part of a site. Approximate: matching pages are crawled sooner, but pages are not globally ordered by
  score, and a page already queued is not re-prioritised if it turns up again. The effect is strongest at low
  **Maximum concurrency**, since pages crawled in parallel finish in whatever order they finish.

### Extraction strategies

- **JsonCSS** — declarative CSS-selector extraction, no LLM involved. See the
  [Crawl4AI docs](https://docs.crawl4ai.com/extraction/no-llm-strategies/) for the schema format.
- **LLM** — pass an instruction and, optionally, a JSON schema. Any provider supported by LiteLLM works; you
  supply the model identifier and API token.

The **Cosine** strategy from earlier versions of this Actor has been removed — it pulled in PyTorch and
sentence-transformers for a feature almost nobody used. Its input fields are still accepted so that old
configurations do not break, but selecting it fails the run with an explanation.

### Interruptions and progress

Pages waiting to be crawled live in the run's request queue, so you can watch a long crawl progress in the
Apify console, and an interrupted run — migrated to another machine, aborted, or resurrected — carries on
from where it stopped rather than starting over.

# Actor input Schema

## `startUrls` (type: `array`):

URLs to start with

## `maxCrawlDepth` (type: `integer`):

How many levels of links should the crawler follow. The Start URLs have a depth of 0, all links found there have a depth of 1, and so on.

## `maxPages` (type: `integer`):

Stop the crawl after this many pages have been crawled successfully. Leave empty for no limit.

## `crawlStrategy` (type: `string`):

How the crawler traverses the link graph. `bestFirst` visits the most promising URLs first and requires **Keywords** to be useful.

## `keywords` (type: `array`):

Used by the `bestFirst` strategy to score URLs. URLs containing these words are crawled sooner.

## `includeExternalLinks` (type: `boolean`):

Follow links pointing outside of the domain of the page they were found on. Combine with **Allowed domains** to keep the crawl bounded.

## `includeUrlGlobs` (type: `array`):

Only follow links matching at least one of these glob patterns, e.g. `*/docs/*`. Start URLs are always crawled.

## `excludeUrlGlobs` (type: `array`):

Never follow links matching any of these glob patterns.

## `allowedDomains` (type: `array`):

Only follow links on these domains. Leave empty to allow all.

## `blockedDomains` (type: `array`):

Never follow links on these domains.

## `respectRobotsTxt` (type: `boolean`):

Skip URLs disallowed by the target website's robots.txt.

## `proxyConfiguration` (type: `object`):

Select proxies to be used by your crawler.

## `maxConcurrency` (type: `integer`):

Upper limit on how many pages are crawled in parallel. The crawler backs off below this on its own when the run approaches its memory allocation, so treat it as a ceiling rather than a target.

## `maxRetries` (type: `integer`):

How many times to retry a page that failed to load.

## `pageTimeoutSecs` (type: `integer`):

How long to wait for a page to load before giving up, in seconds.

## `waitUntil` (type: `string`):

Which page lifecycle event to consider as "loaded".

## `magicMode` (type: `boolean`):

Masks browser automation signals, simulates human-like behavior, overrides navigator properties, handles cookie consent popups, manages browser fingerprinting and randomizes timing patterns.

## `scanFullPage` (type: `boolean`):

Scroll to the bottom of the page before extracting content. Necessary for lazy-loaded and infinite-scroll pages.

## `cssSelector` (type: `string`):

Ensures only a part of the website is processed

## `excludedTags` (type: `array`):

HTML tags to strip before extracting content, e.g. `nav`, `footer`, `aside`.

## `markdownContentFilter` (type: `string`):

Produces an additional pruned `fit_markdown` alongside the raw one. `pruning` drops low-density boilerplate, `bm25` keeps content relevant to a query.

## `markdownContentFilterQuery` (type: `string`):

Query used by the BM25 markdown content filter.

## `waitFor` (type: `string`):

CSS selector or JS function to wait for before processing a page

## `jsCode` (type: `string`):

Function executed for each request

## `saveScreenshots` (type: `boolean`):

Should the crawler save screenshots?

## `savePdfs` (type: `boolean`):

Render each page to a PDF and store it in the key-value store.

## `extractionStrategy` (type: `string`):

Specifies how structured data is extracted from each page

## `llmProvider` (type: `string`):

LiteLLM-style model identifier of the LLM used to extract data, e.g. `openai/gpt-4o` or `anthropic/claude-sonnet-4-5`.

## `llmApiToken` (type: `string`):

Token used for authentication with the LLM provider

## `llmBaseUrl` (type: `string`):

Custom API endpoint of the LLM provider.

## `llmInstruction` (type: `string`):

Instructions (prompt) for the LLM

## `llmSchema` (type: `string`):

JSON schema describing the shape of the data the LLM should return. Leave empty to let the LLM decide.

## `jsonCssSchema` (type: `string`):

Description of the data to be extracted, see https://docs.crawl4ai.com/extraction/no-llm-strategies/

## `cosineSemanticFilter` (type: `string`):

**Removed.** The cosine extraction strategy required PyTorch and is no longer supported. Use the LLM or JsonCSS strategy instead.

## `cosineWordCountThreshold` (type: `integer`):

**Removed.** See above.

## `cosineLinkageMethod` (type: `string`):

**Removed.** See above.

## `cosineTopK` (type: `integer`):

**Removed.** See above.

## `cosineModelName` (type: `string`):

**Removed.** See above.

## `debugLog` (type: `boolean`):

Print debug-level log messages

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.crawl4ai.com"
    }
  ],
  "maxCrawlDepth": 2,
  "crawlStrategy": "bfs",
  "includeExternalLinks": false,
  "respectRobotsTxt": false,
  "proxyConfiguration": {
    "useApifyProxy": true
  },
  "maxConcurrency": 5,
  "maxRetries": 2,
  "pageTimeoutSecs": 60,
  "waitUntil": "domcontentloaded",
  "magicMode": true,
  "scanFullPage": false,
  "markdownContentFilter": "none",
  "saveScreenshots": false,
  "savePdfs": false,
  "debugLog": false
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `content` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.crawl4ai.com"
        }
    ],
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("janbuchar/crawl4ai").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://docs.crawl4ai.com" }],
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("janbuchar/crawl4ai").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.crawl4ai.com"
    }
  ],
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call janbuchar/crawl4ai --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,janbuchar/crawl4ai"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/leuS7kGvGuH31ddts/builds/KXEpqxU5kj4HFUbdk/openapi.json