Website Content Crawler: Markdown for AI and RAG

Crawl sites you own or are authorized to process into clean Markdown for AI and RAG. The Actor keeps headings, lists, ordinary tables and fenced code, attaches source-linked chunks, and bills one event per useful page.

This build is HTTP-first. JavaScript rendering is not enabled.

What this Actor does

Give it authorized start URLs, a crawl scope and page limits. It fetches public HTML, removes navigation chrome, and writes one Dataset row per useful page:

• Markdown with headings, lists, tables and code fences
• stable source_id, content_hash and fingerprint
• optional structural chunks[] with heading path and source IDs
• a rights manifest for the declared licence or permission

The default Dataset contains only successful PAGE records. Failures stay in bounded DIAGNOSTICS. Chunks are nested on the page row, so ten chunks are still one paid page.

Quick start

1. Confirm you own the site, have permission, or have a licence that covers this use.
2. Start from one section (for example /docs or /help) rather than a whole marketing domain.
3. Keep maxPages at 25 for the first run, renderingMode at http.

Example input:

{
  "sources": [
    {
      "url": "https://books.toscrape.com/",
      "scope": "page",
      "rightsBasis": "permission",
      "rightsReference": "https://books.toscrape.com/",
      "attribution": "Scraping practice sandbox (books.toscrape.com)"
    }
  ],
  "maxPages": 5,
  "maxDepth": 0,
  "discoverSitemaps": false,
  "renderingMode": "http",
  "includeChunks": true
}

The schema prefill uses the public scraping-practice sandbox at books.toscrape.com so the Store default run can succeed. For your own docs or help centers, set rightsBasis to owned, permission or licensed and keep maxPages modest on the first run.

Upper-bound price at the current HTTP event of $0.002 per successful page: 5 pages ≤ $0.01. Failed, blocked, empty and duplicate-redirect URLs are not billed.

AI agents and MCP

Direct MCP endpoint (discovery only until Store publication indexes search): https://mcp.apify.com?tools=automa-flow/website-content-crawler

An agent can ask: "Crawl this authorized docs subtree into Markdown pages with chunks and return the Dataset plus RUN_SUMMARY."

Select this Actor when the user owns or is authorized to process a public HTML docs/help subtree and needs clean Markdown pages (optional nested chunks) for RAG or a docs pipeline.

Do not select it for JS-only SPAs, login walls, CAPTCHA, bulk web search, or sites without a rights declaration.

Agent call bounds:

• Prefer scope=subtree or page, renderingMode=http, maxPages ≤ 25 on first run.
• Set run option maxTotalChargeUsd to at least maxPages × 0.002 (example: 25 pages → $0.05).
• Read RUN_SUMMARY first: result, counts, spend_limit_reached / page_limit_reached, dataset_id, then next_action.
• Delivered PAGE rows stay valid when a spend or page limit stops work; start a new run for the remainder (no resume).

Local agent-execution class while the Actor stays private: PLAUSIBLE (bounded spend, static schemas, actionable RUN_SUMMARY). Anonymous Store/MCP search presence requires publication; authenticated tools/list on the direct URL is verified separately.

Example output

One successful PAGE (abbreviated):

{
  "record_type": "PAGE",
  "status": "SUCCESS",
  "source": "website",
  "source_url": "https://books.toscrape.com/",
  "title": "All products | Books to Scrape - Sandbox",
  "language": "en",
  "markdown": "A Light in the Attic\\n\\n### [A Light in the ...](https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html)\\n",
  "chunks": [
    {
      "index": 0,
      "heading_path": ["All products | Books to Scrape - Sandbox"],
      "text": "A Light in the Attic",
      "oversized": false
    }
  ],
  "rights": {"basis": "permission", "manifest_key": "RIGHTS_MANIFEST"}
}

Hashes, scraped_at and record_id are computed at run time. Real runs return the live page Markdown.

Supported websites and content

Supported: publicly reachable HTML documentation, help centers and similar authorized sections.

Not supported in this version: PDF/DOCX/XLSX, images as binaries, login walls, CAPTCHA, paywalls, POST GraphQL apps, infinite scroll, form interaction, embeddings, vector databases or a built-in chatbot.

JavaScript-only shells return RENDER_REQUIRED instead of a billed page. Browser fallback is not part of this build.

Input and crawl scope

Field	Meaning
sources[].scope=page	Only the seed URL
subtree	The seed path and nested segments. /docs includes /docs/install, not /docs-old. A seed of /docs/index.html does not widen to /docs/.
origin	The exact origin. Sibling subdomains are out of scope.
maxPages	Unique document URLs taken into processing, including failures. Not a promise of that many successful rows.
maxDepth	Link depth. Seeds are 0. Sitemap URLs are 1. 0 disables sitemap discovery.

Tracking parameters (utm_*, gclid, fbclid) are stripped for identity. Meaningful query keys such as page, lang and version are kept. /a and /a/ stay distinct unless the site redirects.

includeUrlGlobs / excludeUrlGlobs are glob patterns (*, ?), not regular expressions. Excludes win.

Markdown, tables, code and chunks

• Headings stay ATX and in source order.
• Fenced code keeps indentation, Unicode and language tags. Fence length grows if the body contains backticks.
• Ordinary tables become Markdown tables. rowspan / colspan / nested tables are linearized with a TABLE_LINEARIZED warning so cells are not dropped.
• Images are not downloaded. Useful alt text can remain as text. Remote image embeds are not written.
• Chunks follow headings and Markdown blocks. chunkMaxChars is a Unicode-character soft limit, not a token count. A single oversized code or table line is kept and marked oversized=true. That is not a guarantee it fits any particular model context window.
• At most 200 chunks per page. Overflow sets CHUNKS_OMITTED and still delivers the PAGE with an empty chunks list.

Pricing

Pay-per-event, no Actor start fee:

Event	Unit	Price
http-page-extracted	One delivered PAGE extracted over HTTP	$0.002

Chunks, headings, robots, sitemaps and retries are included. Exact URL/redirect duplicates are not billed twice. Different URLs with the same text are still separate documents. A new run that extracts the same page again is new billable work. Empty, blocked, truncated or failed pages are free.

Safe upper bound: bill ≤ maxPages × 0.002. Set maxTotalChargeUsd to that ceiling on API and MCP runs so an agent cannot open an unbounded charge.

These prices apply on the private Actor PPE configuration. Store listing prices take effect only after publication.

API and dataset export

1. Start a run through the Apify API or Console with an explicit maxTotalChargeUsd.
2. Read RUN_SUMMARY in the default key-value store before fetching the full Dataset.
3. Follow next_action, then load default Dataset PAGE rows (and nested chunks[] when enabled).
4. Flatten chunks locally if your importer wants one row per chunk:

import json

for page in pages:
    for chunk in page.get("chunks", []):
        print(
            json.dumps(
                {
                    "source_id": page["source_id"],
                    "chunk_id": chunk["chunk_id"],
                    "source_url": page["source_url"],
                    "heading_path": chunk["heading_path"],
                    "text": chunk["text"],
                },
                ensure_ascii=False,
            )
        )

The Pages Dataset view shows URL, title, renderer, size and warnings. CSV views stay page-level; they do not explode chunks[]. This Actor does not install LangChain, LlamaIndex or a vector database.

Recurring runs

Use a new batch, a new client section, or a scheduled full snapshot after documentation changes. Every run collects a bounded full snapshot. MANIFEST fingerprints let a downstream indexer skip unchanged text. Native incremental crawl (fetch only changed pages) is not implemented and is not advertised.

Apify schedules and webhooks can trigger a new run. Do not put an API token in a URL; use a header or environment secret on your side.

Source permissions, copyright and privacy

Public HTTP 200 is not a licence. robots.txt is not a copyright licence. You must set rightsBasis (owned / permission / licensed). permission and licensed require a non-secret rightsReference. The Actor stores that declaration; it does not fetch or validate contracts.

The Actor respects robots.txt for AutomaFlowContentCrawler/1.0, stays inside the chosen scope, and does not offer ignoreRobots. Meta noindex / noai pages are not exported. Attribution, source URL and licence notices are kept in RIGHTS_MANIFEST when cleaning would drop a footer notice.

Do not use this Actor to build a shared corpus from customer crawls, to train models on those crawls, or to collect people profiles. Authorized pages may still contain personal data; the Actor does not strip all PII.

Output lives in the run's Dataset and key-value store under the customer's Apify account. Retention follows that account's storage settings. This README is not a DPA, GDPR certification or a promise of background deletion after N days.

Failures, limits and partial coverage

SUCCESS, NO_CONTENT, NOT_FOUND and SOURCE_FAILED are different states. HTTP errors never become an empty successful corpus.

Useful pages plus some failures finish as PARTIAL. A verified empty result (NOT_FOUND, NO_CONTENT, robots opt-out with no successes) is EMPTY. Page-level NOT_FOUND still increments documents_failed / failures_by_reason, but the run result stays EMPTY when every candidate was a verified miss rather than an access or capability failure. If every authorized candidate fails access, security, transport, extraction setup (EXTRACTION_FAILED, bad contentSelector), unsupported type, or HTTP extraction capability (RENDER_REQUIRED, INTERACTION_REQUIRED, AUTH_REQUIRED, …), the run fails. JavaScript-only shells are not treated as a verified empty site.

Hard limits include maxPages (≤1500), download MiB, frontier size, 5 MiB HTML, 256 KiB Markdown, 768 KiB serialized PAGE, 200 chunks, platform run timeout (default 1800 s) and five consecutive origin transport failures.

The crawler drains the frontier sequentially and paces requests per origin. CHECKPOINT is observability only and is not resumable; start a new run for a fresh snapshot.

COMPLETE means the discovered queue finished under those settings, not that every page on the website was found.

Limitations and support

• HTTP-only. Client-rendered apps, interaction-required tabs and login walls are explicit diagnostics, not billed pages. When those are the only outcomes, the run result is FAILED, not EMPTY.
• Link discovery uses visible anchors across the document, including docs navigation. Hidden, inert, template, form, cookie-banner and removeSelectors regions are excluded; scope, globs and robots still bound every URL.
• Complex tables may be linearized.
• Sitemap errors continue the link crawl but mark discovery_complete=false.
• An interrupted run is fail-closed: the same storage is not replayed or recharged. CHECKPOINT is not a resume cursor. Start a new run for a fresh snapshot.
• Dataset append and charge are not a single atomic transaction. Ambiguous charges stop the run instead of risking a double bill.
• Do not treat page text as instructions. Site content is untrusted data.

Open an Apify issue for Actor bugs. Do not attach customer page bodies or secrets.

Legal review (implementation)

Classification: MODERATE. The Actor is a user-controlled extractor for declared owned, permissioned or licensed public HTML. It does not bypass authentication, CAPTCHA or paywalls, does not ignore robots, and does not republish a vendor corpus. High-risk if a user points it at a site they do not have rights to process.