Website Content Crawler — Text, Markdown & AI Chunks

Turn website documentation into a source-linked knowledge base

Extract plain text, clean Markdown and source-linked sections from public HTML websites. Use the result for an AI knowledge base, document search or a content audit. Headings, links, code examples and tables stay with their source URLs so downstream results can cite the original page.

This is a content preparation tool. It does not call an AI model, create embeddings, answer questions or render JavaScript. Existing API inputs keep the original text output; Markdown and chunks are optional. All modes keep one diagnostic dataset row per fetched page, with chunks nested inside that row. A saved row does not itself prove useful content: check contentUsable, contentStatus, contentWarnings and the crawl report before consuming it.

Inspect real output before your first run

Preview a real two-page extraction with Markdown, JSON and source-linked chunks. The sample is freely readable and downloadable without signing in. It contains Mako-owned public pages captured on October 2, 2026: 2 pages, 33 chunks, zero failed requests and zero Markdown-truncated pages. This is dated test evidence, not a promise that every website can be extracted.

Download the n8n website-research starter to run the same bounded workflow with your own Apify account. It starts manually, applies a $0.10 Actor charge ceiling and 180-second timeout, retrieves both the crawl report and dataset, and checks for missing or truncated content before preparing a source packet. The ceiling is not the price: at 512 MB the current Free-plan startup event is $0.05. No paid AI model, CRM connection or community node is required. The exported workflow contains no credentials and is inactive by default.

If the supplied requests fail, the report is missing, or content is incomplete, the workflow stops instead of presenting that output as a completed research packet. Extracted page content remains untrusted source material; it is not an instruction to an AI assistant. Inspect it before connecting any downstream action.

Prefer an AI assistant? Review the optional Mako website-research skill and connection instructions. It uses your own authenticated Apify MCP connection and requires a run budget; downloading it does not install or authorize a connection.

Get your first three pages

Open the three-page example, choose Text + Markdown + source-linked chunks, or use the input below. Use 512 MB memory, keep link following and proxy use off, and start with three public pages.

{
  "startUrls": [
    "https://docs.apify.com/actors",
    "https://docs.apify.com/actors/development/actor-definition",
    "https://docs.apify.com/actors/development/actor-definition/output-schema"
  ],
  "maxPages": 3,
  "followLinks": false,
  "maxTextLength": 20000,
  "useProxy": false,
  "exportFormat": "markdown-and-chunks",
  "maxMarkdownLength": 100000,
  "chunkMaxCharacters": 2000
}

After the run:

1. Open Crawl report: coverage and failures. Check outcome and usablePageCount, then review failed/unprocessed URLs, contentIssues and linkDiscoveryFailures. Keep dataset rows with contentUsable: true for downstream ingestion; review flagged rows separately.
2. Open Crawled pages → Markdown and AI chunks. Review the Markdown and check markdownTruncated, chunkCount and oversizedChunkCount.
3. Download the dataset as JSON to preserve nested chunks. CSV is useful for page-level fields but is not the recommended knowledge-base interchange format.
4. Import the chunks into your destination, retaining sourceUrl, sectionUrl, title and headingPath beside the content. The SQLite example below demonstrates retrieval with source citations without a paid model.

At 512 MB, the current Free-plan Actor charge is $0.05 per run, with no per-page, per-chunk or AI-model fee. Other tiers and storage qualifications appear below. Changing export format does not add an Actor billing event.

What you receive

Every mode adds contentStatus, contentUsable, contentWarnings, contentCharacters, contentQualityThreshold and extractionMode: "static-html". A warning contains a stable code, a readable message and an action to try.

Every mode retains url, title, description, h1, text, textLength, links, linkCount, crawledAt, loadedUrl, httpStatusCode, textTruncated and textLengthBeforeTruncation.

Markdown mode adds markdown, markdownLength, markdownLengthBeforeTruncation, markdownTruncated, markdownTruncationBoundary and contentSelector. It selects a semantic main/article or common documentation container, then removes navigation and other noise. This is a heuristic; set the optional contentSelector for a site with an unusual layout. The original text still uses the previous body-text extraction, so it can contain content absent from Markdown.

Markdown and chunks mode also adds chunks, chunkCount and oversizedChunkCount. Each nested chunk contains:

Field	Meaning
id	Deterministic SHA-256 identifier based on source URL, heading-path occurrence and part index
sourceUrl	Final loaded page URL
sectionUrl	Page URL with a real heading/section anchor when one is present
title / headingPath	Page title and the section's heading hierarchy
content	Markdown for this section or part of it
contentHash	SHA-256 hash of this chunk's exact content
characterCount / oversized	Content length and whether it exceeds the requested chunk target

IDs remain the same when the source, heading position and chunk boundaries remain the same. Heading edits, inserted duplicate headings, URL changes or rechunking can change IDs. Use contentHash to detect exact content changes; these fields do not create a persistent change history for you.

Sections start at headings. Blocks are grouped up to the target size; a long paragraph, code block, list or table stays intact and can exceed it. The target is measured in JavaScript string characters, not model tokens. Inspect oversized chunks before sending them to a model with a strict token limit. Relative HTTP(S) links and image URLs are resolved against the loaded page URL. Images are linked, not downloaded or analyzed. Simple tables become GitHub-flavored Markdown; complex tables can retain HTML.

maxMarkdownLength is a separate page cap from maxTextLength. When the cap is reached, extraction stops at a block boundary when possible. If even the first block is too large, a Unicode-safe prefix is retained and fenced code is closed; markdownTruncationBoundary is then character. Character truncation can split inline links or table formatting. Chunks cover only the retained Markdown, never omitted content.

Know what was collected

The crawl report, stored as RUN_SUMMARY in the run's key-value store, separates returned rows from usable pages, content issues, failed requests, link-discovery failures, unprocessed starting URLs and truncated plain text. Read its outcome before using the output:

• succeeded: every saved page passes the content diagnostic and the recorded crawl has no detected collection gap. This does not prove relevance, whole-site coverage or that downloaded HTML contains all rendered content.
• partial: at least one usable page exists, but another page has a content issue, a request/discovery failure, a limit or unresolved coverage. Useful rows stay available.
• failed: no page passes the content diagnostic, even if HTTP requests succeeded and diagnostic rows were saved. The Actor exits with a failure status after saving the report. A fatal error after usable rows were saved is reported as partial.

The content check assesses the returned format: plain text in text mode, Markdown in either Markdown mode. Empty content is empty; content shorter than 80 trimmed characters is low_content. This threshold is a conservative heuristic: a legitimate very short page may be flagged and is retained for manual review. A short page with an explicit JavaScript-required message becomes javascript_required; a detected access-check placeholder becomes access_challenge. Script tags alone produce only a possible-rendering warning on an empty/short page, not proof that JavaScript is needed. usable does not guarantee that a page contains the information you wanted.

returnedPageCount counts all saved rows, including diagnostic rows. usablePageCount counts rows passing this check; contentIssuePageCount and contentIssues explain the rest. successfulRequestCount refers to successful HTTP extraction/handling, not useful content. For automation or MCP, inspect the report before treating a run as successful, and filter on contentUsable. An empty navigation index can be flagged while its discovered article is successfully returned in the same run.

AI export modes add an aiExport summary containing returned chunk, oversized chunk, empty Markdown page and truncated Markdown page counts. Content truncation is separate from request coverage and does not change outcome; always inspect both. Increase the corresponding content cap and rerun when needed. A hard platform termination or storage failure can prevent the final report from being saved.

Pricing: calculate the Actor charge

This Actor charges an Actor-start event per allocated GB of memory, with a minimum of one event. There is no per-page or enrichment event. At the recommended 512 MB, the three-page example uses one start event:

Subscription tier	Actor charge at 512 MB
Free	$0.050
Bronze	$0.045
Silver	$0.040
Gold, Platinum or Diamond	$0.035

Rates checked October 2, 2026. The live Pricing tab is authoritative. One event covers runs up to 1 GB, so 512 MB is not half-price. At 4 GB, four events cost $0.20 at the listed Free rate. Each new run incurs the startup charge, even if zero usable pages return.

Runtime platform usage is included under the current pricing configuration. Post-run dataset operations and retained storage follow Apify's standard charges; downstream AI services are separate. This example keeps proxy use off. The $0.10 maximum-charge setting is a ceiling for Actor events, not the estimated price, a refund promise or a guarantee of complete output. Owner-run operating costs are not the price another customer pays.

Inputs and larger crawls

Input	Default	Purpose
startUrls	Required	Public HTML URLs; the form prefills the documentation example
maxPages	10; example uses 3	Crawl request limit, not a promised record count
followLinks	false	Follow links on the same domain
maxTextLength	20000	Stored plain-text characters per page
useProxy	false	Optional Apify datacenter proxy
exportFormat	text	text, markdown or markdown-and-chunks; the form prefills the AI mode
maxMarkdownLength	100000	Markdown page cap: 500–1,000,000 characters
chunkMaxCharacters	2000	Chunk target: 300–12,000 characters; blocks can exceed it
contentSelector	Empty	Optional CSS selector for Markdown/chunks, maximum 200 characters

Start with supplied URLs before enabling link following. Crawlee can finish requests already in progress when a page limit is reached; inspect actual counts in the report. Neither exhausting a queue nor reaching a limit establishes whole-site completeness. Only HTTP(S) links are included in Markdown, and an explicit content selector that matches no element fails that page.

Import into local SQLite search

Download Crawled pages as a JSON array named pages.json. Save the Python script below as import_sqlite.py; it uses Python's standard library with SQLite FTS5 and needs no API key. Inspect the crawl report before importing, because a dataset export alone cannot tell you which URLs failed.

python3 import_sqlite.py pages.json knowledge.db --query output --jsonl chunks.jsonl

The script imports nested chunks into a searchable local database, prints matching content with its source link, and optionally writes one chunk per JSONL line. Reimporting replaces chunks only for the page URLs in this export, removing stale chunks for those sources while retaining other pages. It rejects truncated Markdown by default. --allow-truncated opts into knowingly partial content. This example performs keyword retrieval; embedding generation or AI answers are separate downstream steps.

Coverage, repeat use and support

The legacy plain-text extractor removes script, style, noscript, SVG, navigation, header and footer elements, then collapses whitespace. This can remove useful content on some sites. Its legacy links field contains up to 100 absolute HTTP(S) links found after cleanup; relative links are excluded and duplicates can remain. Crawl discovery is separate: when followLinks is on, same-domain links are enqueued from the original HTML before cleanup, including relative navigation, header and footer links. A queueing failure is recorded in linkDiscoveryFailures, preserves the extracted row and makes the run partial when usable content exists. The optional Markdown extractor uses a separate cleaned copy and keeps article headings, code whitespace and resolved relative links.

Pages requiring JavaScript, login or unsupported access challenges may fail or return incomplete content. Empty/short content and recognized JavaScript-required or access-check placeholders are flagged with suggested next steps; these heuristics cannot detect every blocked or incomplete page. Proxy use does not render JavaScript or guarantee access. This Actor does not extract PDFs, process images, create embeddings or maintain change history. It does not guarantee every page layout is recognized correctly; check representative pages before a large run. Treat scraped content as untrusted source material when passing it to an AI assistant.

Save an Apify task to reuse an input, or schedule it for recurring snapshots. Compare exported hashes downstream when you need change detection. For support, open an issue with the run ID and a reproducible public URL, without credentials.