🔥🕷️ Crawl4AI

An Apify Actor wrapping Crawl4AI, the open-source LLM-friendly web crawler and scraper.

The Actor can:

• Crawl and scrape websites with a real browser, following links to a configurable depth.
• Produce clean Markdown, suitable for RAG pipelines or direct ingestion into large language models.
• Extract structured data using CSS selectors or an LLM of your choice.
• Route traffic through Apify Proxy, and resume where it left off if the run gets migrated.
• Scale concurrency to the memory you give the run, instead of guessing.

Concurrency and memory

Maximum concurrency is an upper bound. The crawler watches how much of the run's memory allocation is actually in use and stops opening new pages as it gets close, resuming once memory frees up — so a value that is slightly too ambitious costs you speed rather than a crashed run.

If you see deferred, low memory in the run status, give the run more memory or lower the concurrency; pages deferred that way can end up skipped.

Usage

1. Enter your start URLs.
2. Choose how far the crawler should follow links, and optionally narrow the crawl down with URL globs or allowed domains.
3. Pick an extraction strategy if you want structured data on top of the Markdown.
4. Run the Actor.

How much will it cost?

Apify provides $5 free usage credits every month on the Apify Free plan. With Crawl4AI, you can enjoy a certain number of results per month for free.

For larger data needs, consider upgrading to the $49/month Starter plan for increased monthly results volume. Or opt for the Scale plan for even higher result limits.

Results

Every crawled page becomes one dataset item. The page content itself is stored in the run's key-value store, and the dataset links to it:

[
    {
        "url": "https://docs.crawl4ai.com/",
        "success": true,
        "statusCode": 200,
        "depth": 0,
        "parentUrl": null,
        "markdown": "https://api.apify.com/v2/key-value-stores/m1Sqnke1KWM0AI8co/records/content_4242.md",
        "html": "https://api.apify.com/v2/key-value-stores/m1Sqnke1KWM0AI8co/records/content_4242.html",
        "screenshot_url": null,
        "pdf_url": null,
        "extracted_content": null,
        "metadata": {
            "title": "Home - Crawl4AI Documentation",
            "description": "🚀🤖 Crawl4AI, Open-source LLM-Friendly Web Crawler & Scraper"
        }
    }
]

If you enable a Markdown content filter, a pruned fitMarkdown link is added alongside the raw one.

Crawl strategies

• Breadth-first — visits every page at one depth before going deeper. The sane default.
• Depth-first — follows one branch as far as it goes. Crawls one page at a time, so it is slower.
• Best-first — pages whose URL matches your Keywords jump the queue. Use it when you only have budget for part of a site. Approximate: matching pages are crawled sooner, but pages are not globally ordered by score, and a page already queued is not re-prioritised if it turns up again. The effect is strongest at low Maximum concurrency, since pages crawled in parallel finish in whatever order they finish.

Extraction strategies

• JsonCSS — declarative CSS-selector extraction, no LLM involved. See the Crawl4AI docs for the schema format.
• LLM — pass an instruction and, optionally, a JSON schema. Any provider supported by LiteLLM works; you supply the model identifier and API token.

The Cosine strategy from earlier versions of this Actor has been removed — it pulled in PyTorch and sentence-transformers for a feature almost nobody used. Its input fields are still accepted so that old configurations do not break, but selecting it fails the run with an explanation.

Interruptions and progress

Pages waiting to be crawled live in the run's request queue, so you can watch a long crawl progress in the Apify console, and an interrupted run — migrated to another machine, aborted, or resurrected — carries on from where it stopped rather than starting over.