Website Content Crawler
Pricing
from $2.49 / 1,000 results
Website Content Crawler
Website Content Crawler SD - Website Content Crawler is a web crawling tool that extracts the main content of every page of a website as clean Markdown and text, with title, language and metadata, for LLMs, RAG, vector databases and search indexes.
Pricing
from $2.49 / 1,000 results
Rating
0.0
(0)
Developer
Neuro Scraper
Maintained by CommunityActor stats
0
Bookmarked
3
Total users
2
Monthly active users
5 days ago
Last modified
Categories
Share
Website Content Crawler — clean Markdown and text from whole websites
Website Content Crawler crawls a website from one or more start URLs and saves every page's main content as Markdown and plain text, with title, description, language, canonical URL, author, dates and metadata. Menus, footers and cookie banners are removed, so the output is ready for LLMs, RAG pipelines, vector databases and search indexes.
It reads pages over HTTP (no browser), follows links inside the scope you choose, can seed the crawl from the site's sitemaps, respects robots.txt by default and fetches one page at a time per site with a pause between requests.
Input
| Field | Type | Default | Meaning |
|---|---|---|---|
startUrls | URL list | crawlee.dev/python/docs/guides (prefill) | Where the crawl starts |
maxPages | integer | 20 | Pages saved (one row each) |
maxCrawlDepth | integer | 5 | Links away from a start URL; 0 = start URLs only |
crawlScope | select | startUrlPath | startUrlPath (URLs starting with the start URL), sameDomain, sameDomainAndSubdomains |
includeUrlGlobs / excludeUrlGlobs | glob lists | – | e.g. https://site.com/docs/**; include globs replace the scope |
useSitemaps | boolean | false | Also queue in-scope URLs from robots.txt sitemaps / /sitemap.xml |
respectRobotsTxt | boolean | true | Skip disallowed pages, honour Crawl-delay (max 10 s) |
htmlTransformer | select | trafilatura | trafilatura, readability, or none (whole cleaned page) |
saveMarkdown / saveText / saveHtml | booleans | true / true / false | Output formats |
keepMarkdownLinks | boolean | false | Keep [text](url) links in the Markdown |
requestDelayMs | integer | 1000 | Pause between requests to one host |
maxConcurrency | integer | 2 | Pages processed at once across hosts |
proxyConfiguration | proxy | Apify datacenter | Use RESIDENTIAL for sites that block datacenter IPs |
Output (one row per page)
| Field | Meaning |
|---|---|
url, requestUrl, startUrl, referrerUrl, depth, statusCode | Where the page came from |
title, description, language, canonicalUrl, author, publishedAt | Page metadata |
markdown, text, html | Main content in the formats you chose |
wordCount, extractor | Words in the main content; which extractor produced it |
likelyJsRendered | True when the server sent almost no text (page builds itself with JavaScript) |
truncated | True if a field was cut at 1.5 M characters |
metadata | keywords, robots, modifiedAt, Open Graph, Twitter card, JSON-LD types, hreflang, content type, response time, size |
crawledAt | ISO time (UTC) |
Measured (2026-10-03, see TEST_REPORT.md)
- crawlee.dev guides: 20 pages in 22 s; with sitemaps + globs: 60 pages in 67 s (59 found through the sitemap).
- books.toscrape.com: 12 pages in 12 s;
/and/index.htmlsaved once (content de-duplication). - About 1 page per second per site with the default delay.
Limits
- No browser: pages that render their text with JavaScript come out (nearly) empty and are flagged
likelyJsRendered. - PDFs, images and other non-HTML files are skipped. No login or cookies.
- Default scope follows only URLs that start with the start URL — start from a section or the homepage, or pick
sameDomain. - Sites behind anti-bot challenges (Cloudflare, DataDome, ...) are logged as blocked and skipped; try a RESIDENTIAL proxy.
Example
