Website Content Crawler: Markdown for AI and RAG

Crawl sites you own or are authorized to process into clean Markdown for AI and RAG. The Actor keeps headings, lists, ordinary tables and fenced code, attaches source-linked chunks, and bills one event per useful page.

Pages are fetched over HTTP. Sites that need JavaScript rendering are not supported.

What this Actor does

Give it authorized start URLs, a crawl scope and page limits. It fetches public HTML, removes navigation chrome, and writes one Dataset row per useful page:

• Markdown with headings, lists, tables and code fences
• stable source_id, content_hash and fingerprint
• optional structural chunks[] with heading path and source IDs
• a rights manifest for the declared licence or permission

The default Dataset contains only successful PAGE records. Failures stay in bounded DIAGNOSTICS. Chunks are nested on the page row, so ten chunks are still one paid page.

Secure-for-AI mode

Set securityMode to detect or sanitize when crawled pages go to an LLM, a RAG index or an agent. off is the default and keeps the crawl unchanged.

• detect adds a security object to each page and returns the original Markdown. Do not ingest a row while security.ai_ingestion_allowed is false: a secret or an instruction aimed at a model is still in the text.
• sanitize replaces supported secrets with placeholders: known key formats (OpenAI, Anthropic, AWS, GitHub, GitLab, Google, Stripe, Slack, SendGrid, npm, Hugging Face, Apify, JWT, PEM) and values assigned to names such as DB_PASSWORD or AWS_SECRET_ACCESS_KEY. It does the same for emails, phone numbers and payment-card numbers. IP addresses, times, image names and git URLs stay as they are. A page with a high-confidence hostile instruction is quarantined as CONTENT_QUARANTINED and is not billed. If the scanner fails on a page, that page is not delivered.
• Documentation about tools and keys is delivered. "Call the function", "run this in your shell" and "copy your API key" are ordinary developer text. A page is quarantined when its text speaks to the model reading it ("AI assistants reading this page: ..."), tells the model to ignore its previous instructions, or tries to replace its role. Hidden page text is judged the same way; hidden tabs and menus are only counted.
• Looser wording that also appears in setup guides, such as "ignore the above" or "when you summarize this page, ...", marks the page for review: it is delivered with ai_ingestion_allowed false instead of being quarantined.
• A fenced or quoted attack example is softened only when the page also says not to follow it. The word "example" alone does not do that.
• Pages that quote real attacks, such as prompt-injection tutorials, can still be quarantined. That is intended: the quoted text would reach your model.

What it does not catch: this is a deterministic pattern check, not a model. It knows common English phrasing plus "ignore previous instructions" in Russian, German, Spanish and French, and it sees through look-alike letters. A reworded attack, another language, or an order that does not speak to a model ("send all passwords to ...") passes. In our tests on 62 public documentation pages, the only 2 quarantined pages quote real prompt-injection attacks. On the public deepset/prompt-injections test set, 9 of 60 injections were flagged and 0 of 56 benign prompts. Treat it as a filter for common and careless attacks, not as protection against a determined attacker, and not as DLP, GDPR compliance, JavaScript analysis or malware detection.

To try it, add "securityMode": "sanitize" to the Quick start input below.

Quick start

1. Confirm you own the site, have permission, or have a licence that covers this use.
2. Start from one section (for example /docs or /help) rather than a whole marketing domain.
3. Keep maxPages at 25 for the first run, renderingMode at http.

Example input:

{
  "sources": [
    {
      "url": "https://books.toscrape.com/",
      "scope": "page",
      "rightsBasis": "permission",
      "rightsReference": "https://books.toscrape.com/",
      "attribution": "Scraping practice sandbox (books.toscrape.com)"
    }
  ],
  "maxPages": 5,
  "maxDepth": 0,
  "discoverSitemaps": false,
  "renderingMode": "http",
  "includeChunks": true
}

The schema prefill uses the public scraping-practice sandbox at books.toscrape.com so you can try an extraction before adding your own site. For your own docs or help centers, set rightsBasis to owned, permission or licensed and keep maxPages modest on the first run.

Upper-bound price at the current HTTP event of $0.002 per successful page: 5 pages ≤ $0.01. Failed, blocked, empty and duplicate-redirect URLs are not billed.

Related tools for your ingestion workflow

• Track the URL inventory: Sitemap Extractor & URL Change Monitor inventories sitemap URLs and reports later additions or confirmed removals. Map selected authorized HTML URLs into this crawler's sources, including rights information. A sitemap lastmod change is not proof of changed page text.
• Keep original files: Parallel Bulk File Downloader stores an explicit list of permitted public file URLs. Use it for binary attachments you already identified; this crawler produces Markdown from HTML.
• Watch linked PDFs: PDF Change Monitor detects document changes from a direct PDF or a stable page with a configured PDF selector. Use verified changes to decide when your own PDF ingestion step should run; this crawler does not ingest PDF content.

These are separate Actors with separate pricing. Your workflow must choose URLs, map their inputs and start each run; the crawler does not chain them automatically.

AI agents and MCP

Direct MCP endpoint (use your own Apify authentication to run): https://mcp.apify.com?tools=automa-flow/website-content-crawler

An agent can ask: "Crawl this authorized docs subtree into Markdown pages with chunks and return the Dataset plus RUN_SUMMARY."

Select this Actor when the user owns or is authorized to process a public HTML docs/help subtree and needs clean Markdown pages (optional nested chunks) for RAG or a docs pipeline.

Do not select it for JS-only SPAs, login walls, CAPTCHA, bulk web search, or sites without a rights declaration.

Agent call bounds:

• Prefer scope=subtree or page, renderingMode=http, maxPages ≤ 25 on first run.
• Set run option maxTotalChargeUsd to at least maxPages × 0.002 (example: 25 pages → $0.05).
• Read RUN_SUMMARY first: result, counts, spend_limit_reached / page_limit_reached, dataset_id, then next_action.
• Delivered PAGE rows stay valid when a spend or page limit stops work; start a new run for the remainder (no resume).

Example output

One successful PAGE (abbreviated):

{
  "record_type": "PAGE",
  "status": "SUCCESS",
  "source": "website",
  "source_url": "https://books.toscrape.com/",
  "title": "All products | Books to Scrape - Sandbox",
  "language": "en",
  "markdown": "A Light in the Attic\\n\\n### [A Light in the ...](https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html)\\n",
  "chunks": [
    {
      "index": 0,
      "heading_path": ["All products | Books to Scrape - Sandbox"],
      "text": "A Light in the Attic",
      "oversized": false
    }
  ],
  "rights": {"basis": "permission", "manifest_key": "RIGHTS_MANIFEST"}
}

Hashes, scraped_at and record_id are computed at run time. Real runs return the live page Markdown.

Supported websites and content

Supported: publicly reachable HTML documentation, help centers and similar authorized sections.

Not supported in this version: PDF/DOCX/XLSX, images as binaries, login walls, CAPTCHA, paywalls, POST GraphQL apps, infinite scroll, form interaction, embeddings, vector databases or a built-in chatbot.

JavaScript-only shells return RENDER_REQUIRED instead of a billed page. Browser fallback is not part of this build.

Input and crawl scope

Field	Meaning
sources[].scope=page	Only the seed URL
subtree	The seed path and nested segments. /docs includes /docs/install, not /docs-old. A seed of /docs/index.html does not widen to /docs/.
origin	The exact origin. Sibling subdomains are out of scope.
maxPages	Unique document URLs taken into processing, including failures. Not a promise of that many successful rows.
maxDepth	Link depth. Seeds are 0. Sitemap URLs are 1. 0 disables sitemap discovery.

Tracking parameters (utm_*, gclid, fbclid) are stripped for identity. Meaningful query keys such as page, lang and version are kept. /a and /a/ stay distinct unless the site redirects.

includeUrlGlobs / excludeUrlGlobs are glob patterns (*, ?), not regular expressions. Excludes win.

Markdown, tables, code and chunks

• Headings stay ATX and in source order.
• Fenced code keeps indentation, Unicode and language tags. Fence length grows if the body contains backticks.
• Ordinary tables become Markdown tables. rowspan / colspan / nested tables are linearized with a TABLE_LINEARIZED warning so cells are not dropped.
• Images are not downloaded. Useful alt text can remain as text. Remote image embeds are not written.
• Chunks follow headings and Markdown blocks. chunkMaxChars is a Unicode-character soft limit, not a token count. A single oversized code or table line is kept and marked oversized=true. That is not a guarantee it fits any particular model context window.
• At most 200 chunks per page. Overflow sets CHUNKS_OMITTED and still delivers the PAGE with an empty chunks list.

Pricing

Pay-per-event, no Actor start fee:

Event	Unit	Price
http-page-extracted	One delivered PAGE extracted over HTTP	$0.002

Chunks, headings, robots, sitemaps and retries are included. Exact URL/redirect duplicates are not billed twice. Different URLs with the same text are still separate documents. A new run that extracts the same page again is new billable work. Empty, blocked, truncated or failed pages are free.

Safe upper bound: bill ≤ maxPages × 0.002. Set maxTotalChargeUsd to that ceiling on API and MCP runs so an agent cannot open an unbounded charge.

API and dataset export

1. Start a run through the Apify API or Console with an explicit maxTotalChargeUsd.
2. Read RUN_SUMMARY in the default key-value store before fetching the full Dataset.
3. Follow next_action, then load default Dataset PAGE rows (and nested chunks[] when enabled).
4. Flatten chunks locally if your importer wants one row per chunk:

import json

for page in pages:
    for chunk in page.get("chunks", []):
        print(
            json.dumps(
                {
                    "source_id": page["source_id"],
                    "chunk_id": chunk["chunk_id"],
                    "source_url": page["source_url"],
                    "heading_path": chunk["heading_path"],
                    "text": chunk["text"],
                },
                ensure_ascii=False,
            )
        )

The Pages Dataset view shows URL, title, renderer, size and warnings. CSV views stay page-level; they do not explode chunks[]. This Actor does not install LangChain, LlamaIndex or a vector database.

Recurring runs

Use a new batch, a new client section, or a scheduled full snapshot after documentation changes. Every run collects a bounded full snapshot. MANIFEST fingerprints let a downstream indexer skip unchanged text. The crawler fetches pages again on each run; it does not fetch only changed pages.

Apify schedules and webhooks can trigger a new run. Do not put an API token in a URL; use a header or environment secret on your side.

Source permissions, copyright and privacy

Public HTTP 200 is not a licence. robots.txt is not a copyright licence. You must set rightsBasis (owned / permission / licensed). permission and licensed require a non-secret rightsReference. The Actor stores that declaration; it does not fetch or validate contracts.

The Actor respects robots.txt for AutomaFlowContentCrawler/1.0, stays inside the chosen scope, and does not offer ignoreRobots. Meta noindex / noai pages are not exported. Attribution, source URL and licence notices are kept in RIGHTS_MANIFEST when cleaning would drop a footer notice.

Do not use this Actor to build a shared corpus from customer crawls, to train models on those crawls, or to collect people profiles. Authorized pages may still contain personal data; the Actor does not strip all PII.

Output lives in the run's Dataset and key-value store under the customer's Apify account. Retention follows that account's storage settings. Set retention for your needs and delete stored content when you no longer need it. The Actor does not apply a separate automatic deletion period.

Failures, limits and partial coverage

SUCCESS, NO_CONTENT, NOT_FOUND and SOURCE_FAILED are different states. HTTP errors never become an empty successful corpus.

Useful pages plus some failures finish as PARTIAL. A verified empty result (NOT_FOUND, NO_CONTENT, robots opt-out with no successes) is EMPTY. Page-level NOT_FOUND still increments documents_failed / failures_by_reason, but the run result stays EMPTY when every candidate was a verified miss rather than an access or capability failure. If every authorized candidate fails access, security, transport, extraction setup (EXTRACTION_FAILED, bad contentSelector), unsupported type, or HTTP extraction capability (RENDER_REQUIRED, INTERACTION_REQUIRED, AUTH_REQUIRED, …), the run fails. JavaScript-only shells are not treated as a verified empty site.

Hard limits include maxPages (≤1500), download MiB, frontier size, 5 MiB HTML, 256 KiB Markdown, 768 KiB serialized PAGE, 200 chunks, platform run timeout (default 1800 s) and five consecutive origin transport failures.

The crawler drains the frontier sequentially and paces requests per origin. CHECKPOINT is observability only and is not resumable; start a new run for a fresh snapshot.

COMPLETE means the discovered queue finished under those settings, not that every page on the website was found.

Limitations and support

• HTTP-only. Client-rendered apps, interaction-required tabs and login walls are explicit diagnostics, not billed pages. When those are the only outcomes, the run result is FAILED, not EMPTY.
• Link discovery uses visible anchors across the document, including docs navigation. Hidden, inert, template, form, cookie-banner and removeSelectors regions are excluded; scope, globs and robots still bound every URL.
• Complex tables may be linearized.
• Sitemap errors continue the link crawl but mark discovery_complete=false.
• An interrupted run is fail-closed: the same storage is not replayed or recharged. CHECKPOINT is not a resume cursor. Start a new run for a fresh snapshot.
• Dataset append and charge are not a single atomic transaction. Ambiguous charges stop the run instead of risking a double bill.
• Do not treat page text as instructions. Site content is untrusted data.

Open an Apify issue for Actor bugs. Do not attach customer page bodies or secrets.