Crawl4AI
Pricing
Pay per usage
Wraps the Crawl4AI open-source library for retrieving text content from websites.
Pricing
Pay per usage
Rating
3.3
(2)
Developer
Jan Buchar
Maintained by CommunityActor stats
16
Bookmarked
806
Total users
5
Monthly active users
50 days
Issues response
a month ago
Last modified
Categories
Share
How many levels of links should the crawler follow. The Start URLs have a depth of 0, all links found there have a depth of 1, and so on.
Stop the crawl after this many pages have been crawled successfully. Leave empty for no limit.
How the crawler traverses the link graph. bestFirst visits the most promising URLs first and requires Keywords to be useful.
Used by the bestFirst strategy to score URLs. URLs containing these words are crawled sooner.
Follow links pointing outside of the domain of the page they were found on. Combine with Allowed domains to keep the crawl bounded.
Only follow links matching at least one of these glob patterns, e.g. */docs/*. Start URLs are always crawled.
Never follow links matching any of these glob patterns.
Only follow links on these domains. Leave empty to allow all.
Skip URLs disallowed by the target website's robots.txt.
Upper limit on how many pages are crawled in parallel. The crawler backs off below this on its own when the run approaches its memory allocation, so treat it as a ceiling rather than a target.
How many times to retry a page that failed to load.
How long to wait for a page to load before giving up, in seconds.
Which page lifecycle event to consider as "loaded".
Masks browser automation signals, simulates human-like behavior, overrides navigator properties, handles cookie consent popups, manages browser fingerprinting and randomizes timing patterns.
Scroll to the bottom of the page before extracting content. Necessary for lazy-loaded and infinite-scroll pages.
HTML tags to strip before extracting content, e.g. nav, footer, aside.
Produces an additional pruned fit_markdown alongside the raw one. pruning drops low-density boilerplate, bm25 keeps content relevant to a query.
Query used by the BM25 markdown content filter.
CSS selector or JS function to wait for before processing a page
Function executed for each request
Should the crawler save screenshots?
Render each page to a PDF and store it in the key-value store.
Specifies how structured data is extracted from each page
LiteLLM-style model identifier of the LLM used to extract data, e.g. openai/gpt-4o or anthropic/claude-sonnet-4-5.
JSON schema describing the shape of the data the LLM should return. Leave empty to let the LLM decide.
Description of the data to be extracted, see https://docs.crawl4ai.com/extraction/no-llm-strategies/
Removed. The cosine extraction strategy required PyTorch and is no longer supported. Use the LLM or JsonCSS strategy instead.
