Website Content Crawler — Clean Markdown for AI
Pricing
$2.00 / 1,000 delivered results
Website Content Crawler — Clean Markdown for AI
Crawl public HTML websites into clean Markdown and text for AI, RAG and content research. Export titles, metadata, canonical URLs and links with depth and page limits.
Pricing
$2.00 / 1,000 delivered results
Rating
0.0
(0)
Developer
GUIR Dallel
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
18 hours ago
Last modified
Categories
Share
Crawl public HTML websites into clean Markdown and text for AI, RAG and content research. Export titles, metadata, canonical URLs and links with depth and page limits.
Quick start
Paste your targets into the input form and run. The example below fetches a small sample:
The dataset supports JSON, CSV and Excel export. Use the API, schedules, Make, n8n or other Apify integrations to repeat the same collection. No external API key, browser cookies or login is required by this Actor.
Input and limits
| Field | Meaning |
|---|---|
targets | 1–50 source URLs. See README for supported URL formats. |
maxResults | Stops the run after this many unique matching results. Always set a sensible cap. |
maxItemsPerTarget | Per source limit after filters. Apple search uses this per query/country, capped at 200. |
maxPages | Caps pagination or sitemap traversal. Ignored for sources returning a single document (Greenhouse, Ashby, RSS, Apple search). |
proxyConfiguration | Optional Apify proxy settings. On Apify, Google Search uses Google SERP proxies; Google News, LinkedIn and Apple reviews use residential proxies. Other collectors connect directly unless configured here. Proxy usage is included in the Actor price when the listing uses pure pay-per-event pricing. |
maxDepth | 0 collects only start URLs; 1 also follows their internal links. Maximum 3. |
includePath | Optional substring required in discovered URL paths, such as /docs/. Start URLs are always processed. |
Duplicate targets are processed once. Filters apply before result billing. Runs are bounded snapshots; limits can stop collection before the source is exhausted.
Output
Each result includes source, id, title, url and fetchedAt where applicable, plus source-specific fields. The Results view highlights title, url, wordCount, language, depth, markdown. JSON export retains every field, including nested arrays. Missing source values are null rather than estimated.
Example result
An actual test result, with long text shortened for readability:
The free SUMMARY record in the default key-value store reports each target's result count and errors, HTTP requests, and whether the overall result or spending limit was reached. Errors are not inserted into the paid dataset. A run in which every source fails is marked failed; mixed success is reported explicitly in the summary.
Pricing
$2 per 1,000 delivered results ($0.002 each). No Actor start fee. Filtered rows, duplicates within the same context and source errors are not charged. Results are deduplicated within each run. Google Search preserves a URL appearing in distinct queries. Source requests and the default proxy costs are included in paid per-result pricing. Set Apify's maximum charge per run as an additional budget control.
Source coverage and limitations
HTTP-only crawler: does not render JavaScript, log in, solve CAPTCHAs or read PDFs. Obeys robots.txt for the DallelData user agent. Follows same-host links without query strings to avoid crawl traps; supply a final canonical start URL. Extraction removes navigation, headers, footers and forms, then selects main/article/body; complex layouts may still include boilerplate. maxDepth controls internal-link traversal (0–3); maxPages caps fetched page attempts. Does not store or compare historical page versions.
Public sources can change, throttle requests or return no matches. The Actor retries transient network failures up to twice, follows a bounded number of redirects, and rejects private-network targets. It does not bypass login or CAPTCHA challenges. Use data within the source's applicable terms and your intended permissions.
Support
Open an issue on this Actor's Issues tab with the failing public target, input, and run link. Never include passwords, tokens or personal account cookies.
