Website Content Crawler (Anti-Bot) - Markdown for LLMs

Crawl a whole website, a docs section or a blog and get the clean content of every page as Markdown and plain text, ready for LLMs, RAG pipelines, vector databases, AI agents and fine-tuning. Every page is loaded through an anti-bot unblocker, so sites behind Cloudflare, DataDome, PerimeterX (HUMAN), Akamai and similar walls are crawled like any other site: no proxies to configure, no browser settings to tune, no CAPTCHA services.

The input uses the same field names as the most popular website content crawlers (startUrls, maxCrawlPages, maxCrawlDepth, includeUrlGlobs, excludeUrlGlobs, removeElementsCssSelector, ...), and the output has the same shape (url, markdown, text, crawl, metadata), so switching an existing pipeline over takes minutes.

Features

• Gets through anti-bot walls - Cloudflare challenges, DataDome, PerimeterX, Akamai: pages come back as real content, not as "Just a moment..." screens.
• Clean Markdown - headings, lists, tables, links (absolute URLs) and fenced code blocks with their language; menus, footers, cookie banners and scripts removed.
• Main content extraction - keeps the article or documentation body and drops sidebars, or keep the whole page; plus your own CSS selectors to remove or keep.
• Smart crawl scope - stays under the start URL's path by default (https://example.com/docs/ crawls /docs/... only), or use include / exclude URL globs.
• Sitemaps - optionally read the site's sitemaps to find pages that are not linked.
• Duplicates skipped - pages whose canonical URL was already crawled are skipped, and not billed.
• Page metadata - title, description, language, canonical URL, author, keywords, Open Graph tags and JSON-LD.
• Pay only for content - pages that fail, are blocked, do not exist (404) or have no readable text are listed in the ERRORS record and never billed.
• Resumes after platform migrations - the crawl state is saved, so a moved run does not start over or bill pages twice.

Use cases

• RAG and chatbots - feed product docs, help centers and knowledge bases into a vector database.
• LLM fine-tuning and evaluation datasets - collect clean text from many sites.
• AI agents - give an agent the current content of a site that blocks ordinary crawlers.
• Content monitoring and SEO - snapshot a site's pages, titles and descriptions.

Input

Field	Type	Default	Description
startUrls	array	Python tutorial	Where to start. The crawl stays under each start URL's path.
maxCrawlPages	integer	100	The most pages to load in the run (1-100000).
maxCrawlDepth	integer	20	Links away from a start URL to follow (0 = start URLs only).
useSitemaps	boolean	false	Also add in-scope URLs from the site's sitemaps.
includeUrlGlobs	array	[]	Crawl only links matching these globs (replaces the path rule).
excludeUrlGlobs	array	[]	Never crawl links matching these globs.
keepUrlFragments	boolean	false	Treat #fragment URLs as separate pages.
ignoreCanonicalUrl	boolean	false	Do not skip pages whose canonical URL was already crawled.
outputFormats	array	["markdown", "text"]	Any of markdown, text, html (cleaned content HTML).
htmlTransformer	string	readableText	readableText = main content only, none = whole page.
removeElementsCssSelector	string	nav, footer, scripts, dialogs ...	Elements removed before extraction.
keepElementsCssSelector	string	-	Keep only the elements matching this selector.
removeCookieWarnings	boolean	true	Remove cookie consent banners.
waitForSelector	string	-	Wait for this CSS selector before capturing a page (JavaScript-loaded content).
proxyCountry	string	-	Exit-IP country (ISO-2). Leave empty for the most reliable exit.
maxConcurrency	integer	5	Pages loaded at the same time (1-10).
requestTimeoutSecs	integer	150	Timeout for one page (30-200 s).
maxRequestRetries	integer	2	Retries after a server error or timeout.

Globs: ** matches anything, * anything within one path segment, {a,b} either. Example: https://example.com/blog/**.

Example input

Crawl a Cloudflare-protected resource center, at most 200 pages, skipping tag pages:

{
  "startUrls": [{ "url": "https://www.upwork.com/resources/" }],
  "maxCrawlPages": 200,
  "excludeUrlGlobs": [{ "glob": "https://www.upwork.com/resources/tag/**" }],
  "outputFormats": ["markdown"]
}

Output

One dataset item per crawled page:

{
  "url": "https://docs.python.org/3/tutorial/appetite.html",
  "title": "1. Whetting Your Appetite - Python 3.15.0 documentation",
  "description": "If you do much work on computers, eventually you find that there's some task you'd like to automate...",
  "languageCode": "en",
  "markdown": "# 1. Whetting Your Appetite\n\nIf you do much work on computers, eventually you find that there's some task you'd like to automate...",
  "text": "1. Whetting Your Appetite\n\nIf you do much work on computers, eventually you find that there's some task you'd like to automate...",
  "crawl": {
    "loadedUrl": "https://docs.python.org/3/tutorial/appetite.html",
    "loadedTime": "2026-10-12T17:36:01.942018+00:00",
    "referrerUrl": "https://docs.python.org/3/tutorial/",
    "depth": 1,
    "httpStatusCode": 200
  },
  "metadata": {
    "title": "1. Whetting Your Appetite - Python 3.15.0 documentation",
    "description": "If you do much work on computers...",
    "author": null,
    "keywords": null,
    "languageCode": "en",
    "canonicalUrl": "https://docs.python.org/3/tutorial/appetite.html",
    "openGraph": { "og:title": "1. Whetting Your Appetite", "og:type": "website" },
    "jsonLd": null
  },
  "crawledAt": "2026-10-12T17:36:01.942018+00:00"
}

The key-value store also holds:

• SUMMARY - pages fetched, delivered, failed, duplicates, skipped file links, and why the crawl stopped (page limit, charge limit, run timeout).
• ERRORS - pages that failed, were blocked, did not exist or had no readable text, with the reason. These are not billed.

Pricing

$1.20 per 1000 pages. You pay only for pages delivered to the dataset. Failed, blocked, empty, not-found and duplicate pages are free. Set a maximum total charge on the run to cap the cost; the crawl stops when it is reached.

Tips

• Start from the most specific URL you can: https://example.com/docs/ instead of https://example.com/.
• Use maxCrawlPages to cap a first run, look at the result, then raise it.
• Pages that need a moment to load their content with JavaScript: set waitForSelector to an element of the content, e.g. .article-body.
• A site that sets the same canonical URL on every page: turn on ignoreCanonicalUrl.

Notes

• Only HTML pages are crawled. Links to PDFs, images, archives, office documents and other files are skipped.
• robots.txt crawl rules are not applied; crawl responsibly and respect the sites' terms.
• Each page is loaded once (no infinite scrolling or clicking); content that only appears after user interaction is not captured.