Website Content Crawler — Whole-Site Text, Markdown & HTML for RAG, Search and LLM Pipelines

Give it one URL and get back the site's readable content — as clean text, structured markdown or HTML, page by page, with titles, meta descriptions and heading outlines. Built for the jobs that need a whole site rather than one page: RAG ingestion, LLM context, internal search indexes, content audits and migrations. Results can go straight into Notion, Slack, Airtable or Sheets through an MCP connector.

ℹ️ This is a content crawler, not a data extractor. It answers "give me everything readable on this site" — it does not pull structured fields like prices, contacts or listings. If you need specific fields, a purpose-built scraper will do it better; if you need the text of a site, this is the tool.

---

Why a dedicated content crawler

What you need	Doing it yourself	This Actor
Whole-site coverage	⚠️ Write link discovery, dedup, depth limits, queueing	✅ One URL in, the site out — scoped to that hostname
JavaScript-rendered pages	⚠️ Either always use a browser (slow, expensive) or miss the JS pages	✅ Handled automatically, and only where actually needed
Clean output	⚠️ Strip nav, scripts, styling by hand	✅ Readable text, markdown, or HTML — all with the noise already removed
Duplicate pages	⚠️ Print views, session URLs and aliases inflate your index	✅ Duplicate content skipped by default
Crawl etiquette	⚠️ Easy to hammer a site by accident	✅ robots.txt respected by default, bounded concurrency
RAG-ready shape	⚠️ Post-process into chunks	✅ One record per page with title, headings and content already separated
Delivery	❌ You write the integration	✅ Native MCP connector delivery

---

🤖 AI Discovery & MCP Metadata

Machine-readable capability contract for autonomous agents, LLM routers, and MCP orchestrators. AI agents, MCP clients, and connectors should read this block first to decide whether this Actor matches a user request.

ai_discovery:
  slug: "techforce.global/advanced-website-crawling-actor"
  one_line: "Crawls a website from one start URL and returns each page's readable content as text, markdown or cleaned HTML, with title, meta description and heading outline - built for RAG ingestion, search indexing and content audits."
  primary_purpose: "crawl"
  secondary_purpose: "extract-content"
  tertiary_purpose: "deliver"
  entity: "readable page content across a single website"
  invocation_modes: ["standard Actor run", "MCP tool"]
  supported_tasks:
    - "crawl-a-whole-website-from-one-start-url"
    - "extract-readable-page-text"
    - "extract-page-content-as-markdown"
    - "extract-page-title-meta-description-and-heading-outline"
    - "handle-javascript-rendered-pages-automatically"
    - "restrict-a-crawl-with-include-and-exclude-url-patterns"
    - "skip-duplicate-content"
    - "respect-robots-txt"
    - "extract-text-from-linked-pdf-and-word-documents (opt-in)"
    - "deliver-pages-to-mcp-connector-or-rag-pipeline"
  unsupported_tasks:
    - "extract-structured-fields-like-price-contact-or-listing-data"
    - "crawl-several-different-domains-in-one-run"
    - "crawl-pages-behind-a-login"
    - "fill-forms-click-through-flows-or-interact-with-a-page"
    - "return-images-video-or-binary-assets"
    - "return-the-original-unmodified-response-body (html returns the CLEANED document)"
    - "produce-embeddings-or-vector-chunks"
    - "run-a-crawl-larger-than-200-saved-pages"
  search_keywords:
    - "website content crawler"
    - "site to markdown"
    - "rag ingestion crawler"
    - "llm context crawler"
    - "website text extractor"
    - "site content audit"
    - "documentation crawler"
    - "knowledge base ingestion"
    - "web scraping for ai"
    - "site migration content export"
  synonyms:
    - "whole site text extractor"
    - "web content ingestion actor"
    - "markdown site crawler"
  related_concepts:
    ["RAG", "vector database ingestion", "site search indexing", "content audit", "site migration", "LLM context building"]
  input_entities:
    - "startUrl (single start URL; REQUIRED)"
    - "maxPages (1-200 saved pages)"
    - "outputFormat (text | markdown | html)"
    - "crawlDocuments (opt-in PDF/Word extraction)"
    - "includePatterns / excludePatterns (glob URL filters)"
    - "maxDepth, concurrency, maxRetries and other advanced controls"
    - "proxy (Apify Proxy configuration)"
  output_entities: ["url", "title", "meta", "headings", "type", "content"]
  input_rules:
    required_fields: ["startUrl"]
    single_domain_scope: >
      The crawl is scoped to the start URL's hostname. Links to other domains are
      not followed - one site per run.
    max_pages_is_saved_pages: >
      maxPages counts pages SAVED, not pages visited. Skipped duplicates and
      filtered URLs do not consume the budget.
    output_format_values:
      text: "default - plain readable text"
      markdown: "structure preserved (headings, lists, links)"
      html: "the cleaned document markup - scripts, styles and sidebars removed"
    output_format_is_case_insensitive: >
      Values are matched case-insensitively. An unrecognised value falls back to
      text AND logs a warning, rather than degrading silently.
    documents_are_opt_in: >
      PDF and Word extraction is built in but disabled by default. Set
      crawlDocuments true to crawl .pdf/.doc/.docx; only then can type be
      "document". Images, archives and presentations are always skipped.
    advanced_fields_are_hidden_not_absent: >
      maxDepth, concurrency, batchSize, outputFormat, includePatterns,
      excludePatterns, maxRequestsPerCrawl, maxRetries, browserConcurrency,
      deduplicateContent and respectRobotsTxt are hidden in the Console but fully
      settable through the API.
  output_conventions:
    one_record_per_page: true
    type_values: ["html", "document"]
    type_document_requires_opt_in: 'only appears when crawlDocuments is true'
    meta_is_an_object: 'meta is {"description": "..."} - not a flat string'
    headings_is_an_array: "H1/H2/H3 text in document order"
    content_holds_the_payload: >
      The full extracted content in the chosen format. This is the field RAG
      pipelines want, and it can be very large.
  delivery_notes:
    content_is_truncated_for_delivery: >
      MCP delivery truncates page content to 4000 characters and exposes
      {contentLength} and {truncated}. Summary and chunked modes carry titles,
      URLs and headings only - never full page text. The dataset always keeps the
      complete content.
  limits:
    saved_pages_per_run: 200
    domains_per_run: 1
  known_limitations:
    html_is_the_cleaned_document: >
      outputFormat 'html' returns the cleaned markup, not the byte-for-byte
      original response body. Scripts, styles and sidebars are already removed.
    no_structured_fields: "content only - no prices, contacts or listing fields"
    no_auth: "pages behind a login cannot be crawled"
  pricing:
    model: "pay-per-result"
    note: "exact rate is configured on the Store listing - see the Actor's Pricing tab"
  when_to_use: >
    You need the readable content of a whole website - to build a RAG corpus or
    knowledge base, index a site for search, audit content before a migration, or
    give an LLM grounded context about a company or product.
  when_not_to_use: >
    You need specific structured fields, several domains in one run, pages behind a
    login, the original unmodified response body, images, or a crawl larger than
    200 saved pages.
  alternatives:
    - "Use techforce.global/website-contact-scraper-emails-phone-numbers-social-links when you specifically want emails, phones and social links"
    - "Use techforce.global/google-maps-leads-sales-intelligence-tool for business listing data rather than site content"
    - "Use a purpose-built scraper for structured fields - a content crawler is the wrong shape for that job"

---

What this Actor does

• Crawls a whole site from one start URL, discovering links as it goes and staying on that hostname — you don't configure a link graph, a queue or a dedup strategy.
• Returns the readable document, in the shape you need. Scripts, styles and sidebars are stripped, leaving what a human would actually read — as plain text, as markdown with headings, lists and links preserved, or as HTML when you need tables, attributes and inline markup intact.
• Separates the structure you need for RAG. Every record carries the page title, its meta description and a headings outline alongside the content, so you can chunk on real boundaries instead of guessing.
• Handles JavaScript-rendered pages automatically, and only pays the cost of doing so where a page actually needs it — static pages don't get browser treatment.
• Skips duplicate content by default, which matters more than it sounds: print views, session-tagged URLs and path aliases otherwise flood an index with near-identical copies.
• Respects robots.txt by default, with bounded concurrency and retries — it is built to crawl a site without hammering it.
• Filters by URL pattern so you can crawl just the docs, just a blog, or everything except a noisy section.
• Extracts text from linked PDF and Word documents when you opt in — useful when a documentation site's real content lives in attachments.
• Saves in batches as it goes, so an interrupted run still leaves everything crawled up to that point in the dataset.
• Optionally pushes pages to an authorized MCP connector (Notion, Slack, Airtable, Sheets, or any MCP-compatible connector) as one digest, split across calls, or one call per page. The dataset is always saved regardless of delivery settings.

ℹ️ Three formats, one cleaned document. text, markdown and html are three renderings of the same cleaned page — scripts, styles and sidebars are already stripped from all of them. html gives you the markup back (tables, links, code and emphasis intact), which is what you want for structure-sensitive work; it is not the byte-for-byte original response body.

When to use it — and when not to

Scenario	Fit
Building a RAG corpus or knowledge base from a website	✅ Strong (the core use case)
Giving an LLM grounded context about a company or product	✅ Strong
Indexing a site for internal search	✅ Strong
Content audit before a redesign or migration	✅ Strong — titles, headings and body in one pass
Crawling a documentation site into markdown	✅ Strong — outputFormat: "markdown"
Pulling text out of linked PDFs and Word docs	✅ Supported — crawlDocuments: true
Crawling only one section of a site	✅ Strong — include/exclude patterns
Competitor content research	✅ Good
Specific structured fields (price, contact, listing data)	❌ Wrong shape — use a purpose-built scraper
Emails, phones and social links	❌ ➡️ Website Contact Scraper
Several different domains in one run	❌ One site per run — loop from your side
Pages behind a login	❌ No authentication support
HTML output with tables and links intact	✅ Supported — outputFormat: "html"
The original, unmodified response body	❌ html returns the cleaned document
Images, video or binary assets	❌ Text content only
Embeddings or pre-chunked vectors	❌ Content only — chunk and embed downstream
Filling forms or clicking through flows	❌ Read-only crawl
More than 200 saved pages in one run	⚠️ Hard cap — split by section or start URL

---

⚡ Execution Rules (Avoid Common Agent Failures)

1. startUrl is required, and the crawl is scoped to its hostname. Links to other domains are not followed. Crawling three sites is three runs.
2. maxPages counts pages saved, not pages visited. Duplicates that get skipped and URLs that get filtered don't consume the budget — so a run can visit far more than maxPages. The cap is 200.
3. outputFormat takes text, markdown or html — all three rendering the same cleaned page. html is the cleaned markup, not the original response body. Values are case-insensitive, and an unrecognised value falls back to text and logs a warning.
4. PDFs and Word documents are skipped unless you opt in. Set crawlDocuments: true to include them; only then can type be "document". Images, archives and presentations are always skipped.
5. Most inputs are hidden in the Console but fully settable via the API. maxDepth, concurrency, batchSize, outputFormat, includePatterns, excludePatterns, maxRequestsPerCrawl, maxRetries, browserConcurrency, deduplicateContent and respectRobotsTxt are all real inputs — the Console just keeps the form simple. An agent can use all of them.
6. meta is an object, not a string. It's {"description": "..."}. Reach for meta.description.
7. headings is an array of H1/H2/H3 text in document order — a natural chunk boundary for RAG, and not a single string.
8. content can be very large. It holds the whole page. Don't load a 200-page crawl into memory unnecessarily, and don't push it to a connector unfiltered — delivery truncates it deliberately.
9. Duplicate skipping is on by default. If you need every URL even when the text repeats, set deduplicateContent: false — otherwise a site with print views will legitimately return fewer pages than you expected.
10. robots.txt is respected by default. If a crawl returns far less than expected, the site may be disallowing those paths. Turning that off is your decision and your responsibility.
11. Include patterns are ANDed with exclude patterns, and exclusions always win. A URL must match an include pattern and no exclude pattern.
12. A larger crawl is slower and costlier per page than you'd guess, because JS-heavy pages need rendering. Use the async endpoint for anything beyond a small run.
13. deliveryMode defaults to summary — one connector call per run, carrying titles and URLs only. perPage includes each page's (truncated) content and makes one call per page.
14. Delivery truncates page content to 4,000 characters. The dataset always keeps the full text — this only limits what a connector receives, so a single call can't exceed a service limit.

dependencies:
  - rule: "startUrl is required and scopes the crawl to its hostname"
    on_violation: "links to other domains are silently not followed"
  - rule: "maxPages counts SAVED pages, not visited pages"
    on_violation: "run time and cost under-estimated - many more pages may be visited"
  - rule: "outputFormat html returns the CLEANED document, not the original response body"
    on_violation: "expecting byte-for-byte original markup, including scripts and styles"
  - rule: "an unrecognised outputFormat falls back to text and warns"
    note: "the warning is in the run log - check it if content looks like the wrong format"
  - rule: "PDF and Word documents require crawlDocuments true"
    on_violation: "documents silently absent, and type never equals 'document'"
  - rule: "advanced inputs are hidden in the Console but settable via the API"
    on_violation: "agents assume the crawl cannot be tuned"
  - rule: "meta is an object with a description key"
    on_violation: "string operations on meta throw"
  - rule: "headings is an array, not a string"
    on_violation: "chunking logic built on a string breaks"
  - rule: "content holds a whole page and can be very large"
    on_violation: "memory pressure downstream, or oversized connector payloads"
  - rule: "duplicate content is skipped by default"
    on_violation: "fewer pages than expected, read as a crawl failure"
  - rule: "robots.txt is respected by default"
    on_violation: "a thin crawl blamed on the Actor rather than the site's rules"
  - rule: "exclude patterns always beat include patterns"
    on_violation: "a section you intended to crawl is silently filtered out"
  - rule: "delivery requires mcpConnector AND mcpTool"
    on_violation: "delivery step is skipped with a warning; the dataset is still written in full"
  - rule: "mcpArguments must reference {message} for the rendered template to appear anywhere"
    on_violation: "message body is empty; the tool is still called"
  - rule: "mcpConnector only resolves when the Actor runs on the Apify platform"
    on_violation: "local runs log 'APIFY_MCP_PROXY_URL is not set' and skip delivery; the dataset is unaffected"
authentication:
  actor_input: none
  platform: "Apify API token required for API / SDK / MCP invocation"
  target_site: "no credentials - public pages only"

---

📥 Input Contract

.actor/input_schema.json is authoritative — if any table on this page disagrees with the schema files, the schema files win.

Main parameters

Field	Type	Required	Default	Allowed Values / Format	Example
startUrl	string	✅ Yes	https://www.apify.com	One URL — its hostname scopes the crawl	"https://docs.example.com"
maxPages	integer	⬜ No	50	1–200 saved pages	200
crawlDocuments	boolean	⬜ No	false	Include linked .pdf / .doc / .docx	true
proxy	object	⬜ No	—	Apify proxy configuration	see below

Advanced parameters (hidden in the Console, available via the API)

The Console form is deliberately minimal, but these are real inputs — an API caller or agent can set all of them.

Field	Type	Default	Purpose
outputFormat	string	"text"	text, markdown or html — case-insensitive
maxDepth	integer	5	How many link-hops from the start URL
includePatterns	array	["*"]	Glob patterns a URL must match
excludePatterns	array	[]	Glob patterns that reject a URL (added to built-in asset exclusions)
deduplicateContent	boolean	true	Skip pages whose content repeats
respectRobotsTxt	boolean	true	Honour the site's robots.txt
concurrency	integer	20	Parallel crawl workers
browserConcurrency	integer	8	Parallel renders, for JS pages only
maxRetries	integer	2	Retries per URL
maxRequestsPerCrawl	integer	1000	Hard ceiling on total requests
batchSize	integer	20	Pages per dataset write

URL patterns

Patterns are globs matched against both the path and the full URL. Exclusions always win.

Goal	Configuration
Only the documentation	{"includePatterns": ["/docs/*"]}
Only the blog	{"includePatterns": ["/blog/*"]}
Everything except a noisy section	{"excludePatterns": ["/archive/*", "*?print=*"]}
Docs, but not the changelog	{"includePatterns": ["/docs/*"], "excludePatterns": ["/docs/changelog*"]}

Images, archives and presentations are excluded automatically. PDF and Word files are excluded unless crawlDocuments is on.

Proxy

{
  "proxy": { "useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"], "apifyProxyCountry": "US" }
}

Optional. Most public sites crawl fine without one; enable Apify Proxy if a target blocks you, and note that residential groups require a paid Apify plan and are metered per GB.

Examples

RAG ingestion — a documentation site as markdown

{
  "startUrl": "https://docs.example.com",
  "maxPages": 200,
  "outputFormat": "markdown",
  "includePatterns": ["/docs/*"],
  "deliveryMode": "none"
}

Company context for an LLM — a marketing site as text

{
  "startUrl": "https://www.example.com",
  "maxPages": 50,
  "outputFormat": "text",
  "deliveryMode": "none"
}

Content audit including linked documents

{
  "startUrl": "https://www.example.com",
  "maxPages": 200,
  "crawlDocuments": true,
  "outputFormat": "text",
  "deliveryMode": "none"
}

Just the blog, excluding paginated archives

{
  "startUrl": "https://www.example.com/blog",
  "maxPages": 200,
  "includePatterns": ["/blog/*"],
  "excludePatterns": ["/blog/page/*", "*?tag=*"],
  "outputFormat": "markdown",
  "deliveryMode": "none"
}

Cheap validation run before committing

{
  "startUrl": "https://www.example.com",
  "maxPages": 5,
  "deliveryMode": "none"
}

MCP Delivery Configuration (Optional)

Delivery pushes crawled content into your own tools.

Parameter	Type	Default	Description
mcpConnector	string	""	Authorized MCP connector on your Apify account (Notion, Slack, Airtable, Sheets, …). Leave empty for dataset-only runs.
deliveryMode	enum	summary	summary (one call, titles and URLs) · chunked (split across calls) · perPage (one call per page, with content) · none.
mcpTool	string	""	Tool name on the connector: create_page, send_message, append_row. Run once with a connector selected — the log lists the available tools and the chosen tool's expected argument schema.
mcpArguments	object	{}	Arguments passed to the tool. String leaves support {placeholders}.
mcpMessageTemplate	string	""	Template rendered and exposed to mcpArguments as {message}.

Per-page placeholders: {url}, {title}, {description}, {headings}, {type}, {content} (truncated to 4,000 characters), {contentLength}, {truncated}, {page} (a formatted block), {message}.

Summary / chunked placeholders: {startUrl}, {outputFormat}, {pageCount}, {pages} (titles, URLs and headings — not full content), {part}, {partCount}, {message}.

⚠️ Content is truncated for delivery, never in the dataset. A crawl can produce pages tens of thousands of characters long; a connector call carrying one whole would be rejected or time out. {contentLength} and {truncated} tell you when it happened.

One Notion page per crawled page — a knowledge base in one run:

{
  "startUrl": "https://docs.example.com",
  "maxPages": 100,
  "outputFormat": "markdown",
  "mcpConnector": "<your-authorized-notion-connector>",
  "deliveryMode": "perPage",
  "mcpTool": "notion-create-pages",
  "mcpArguments": {
    "parent": { "page_id": "<your-page-id>" },
    "pages": [
      {
        "properties": { "title": "{title}" },
        "content": "{url}\n\n{content}"
      }
    ]
  }
}

A Slack summary of what was crawled:

{
  "startUrl": "https://www.example.com",
  "maxPages": 50,
  "mcpConnector": "<your-authorized-slack-connector>",
  "deliveryMode": "summary",
  "mcpTool": "send_message",
  "mcpArguments": { "channel": "#content-audit", "text": "{message}" },
  "mcpMessageTemplate": "🕸️ Crawled {pageCount} pages from {startUrl}:\n\n{pages}"
}

A large crawl split across Notion sub-pages:

{
  "startUrl": "https://www.example.com",
  "maxPages": 200,
  "mcpConnector": "<your-authorized-notion-connector>",
  "deliveryMode": "chunked",
  "mcpTool": "notion-create-pages",
  "mcpArguments": {
    "parent": { "page_id": "<your-page-id>" },
    "pages": [
      {
        "properties": { "title": "Crawl of {startUrl} — part {part}/{partCount}" },
        "content": "{pages}"
      }
    ]
  }
}

---

📤 Output Contract & Data Structure

Storage: Apify Dataset — one record per saved page, written in batches as the crawl progresses. Pagination: limit & offset on the dataset items endpoint. Duplicates: duplicate content is skipped during the crawl by default. Across runs, dedupe on url.

Page record — 6 fields

Field	Type	Meaning
url	string	The page URL — the dedupe key
title	string	Page title
meta	object	{"description": "..."} — reach for meta.description, it is not a flat string
headings	array of string	H1/H2/H3 text in document order — a natural chunk boundary
type	string	"html", or "document" for a PDF/Word file (only when crawlDocuments is on)
content	string	The payload — full readable text, or markdown if you asked for it. Can be very large

Example — a normal page

{
  "url": "https://docs.example.com/getting-started",
  "title": "Getting Started — Example Docs",
  "meta": { "description": "Install Example and ship your first integration in ten minutes." },
  "headings": ["Getting Started", "Installation", "Your first request", "Next steps"],
  "type": "html",
  "content": "Getting Started\n\nInstall Example and ship your first integration...\n\nInstallation\n\nRun the installer...\n"
}

Example — markdown output

{
  "url": "https://docs.example.com/api",
  "title": "API Reference — Example Docs",
  "meta": { "description": "Complete API reference." },
  "headings": ["API Reference", "Authentication", "Endpoints"],
  "type": "html",
  "content": "# API Reference\n\n## Authentication\n\nUse a bearer token...\n\n## Endpoints\n\n- `GET /v1/items`\n"
}

Example — a linked document (crawlDocuments: true)

{
  "url": "https://www.example.com/whitepaper.pdf",
  "title": "whitepaper.pdf",
  "meta": { "description": null },
  "headings": [],
  "type": "document",
  "content": "Executive summary\n\nThis paper examines..."
}

Feeding it into a RAG pipeline

headings exists precisely so you don't have to chunk blindly:

def chunks_for(page, max_chars=2000):
    """Chunk on real structure, with the page's own context attached."""
    context = f'{page["title"]} — {(page.get("meta") or {}).get("description") or ""}'.strip(" —")
    body, out, buf = page["content"], [], ""
    for para in body.split("\n\n"):
        if len(buf) + len(para) > max_chars and buf:
            out.append({"text": buf, "source": page["url"], "context": context})
            buf = para
        else:
            buf = f"{buf}\n\n{para}" if buf else para
    if buf:
        out.append({"text": buf, "source": page["url"], "context": context})
    return out

# meta is an OBJECT and headings is an ARRAY — guard both.
docs = [c for page in items for c in chunks_for(page)]
print(f"{len(items)} pages -> {len(docs)} chunks")

# `type` is only ever "document" when crawlDocuments was enabled.
pdfs = [p for p in items if p["type"] == "document"]

What a successful run looks like

{
  "status": "SUCCEEDED",
  "defaultDatasetId": "<datasetId>",
  "stats": { "itemCount": 137 }
}

itemCount below maxPages is normal and has several innocent causes: the site genuinely has fewer pages, duplicate content was skipped, URL patterns filtered pages out, robots.txt disallowed paths, or maxDepth was reached. The run log distinguishes them — dropped URLs are logged with the reason (depth, pattern mismatch, robots, duplicate).

itemCount: 0 usually means the start URL didn't load, or every discovered URL was filtered out by patterns or robots.txt.

---

▶️ Invocation & Integration

Lifecycle: Discover → Understand → Configure → Execute → Retrieve → Validate. Authenticate every call with Authorization: Bearer <APIFY_TOKEN>.

Apify Console

Open the Actor → Input tab → enter the Start URL → set Max Pages → optionally enable Crawl PDF and Word documents → (optional) configure the proxy → (optional) pick an MCP connector, tool and template → Start → export from the Dataset tab. The Content view shows the extracted text per page; Overview shows titles and structure.

💡 The Console form is intentionally short. Everything else — output format, depth, URL patterns, concurrency — is available through the API and MCP.

Model Context Protocol (MCP)

claude mcp add --transport http apify \
  "https://mcp.apify.com?tools=techforce.global/advanced-website-crawling-actor"

Python (apify-client)

import os
from apify_client import ApifyClient

client = ApifyClient(os.getenv("APIFY_TOKEN"))
ACTOR = "techforce.global/advanced-website-crawling-actor"

run = client.actor(ACTOR).call(run_input={
    "startUrl": "https://docs.example.com",
    "maxPages": 200,                    # SAVED pages; the crawl may visit more
    "outputFormat": "markdown",         # 'text', 'markdown' or 'html'
    "includePatterns": ["/docs/*"],
    "deliveryMode": "none",
})

items = client.dataset(run["defaultDatasetId"]).list_items().items
print(f"{len(items)} pages")

total = 0
for page in items:
    description = (page.get("meta") or {}).get("description") or ""   # meta is an OBJECT
    outline = " > ".join((page.get("headings") or [])[:3])            # headings is an ARRAY
    total += len(page.get("content") or "")
    print(f'{page["title"][:60]}')
    print(f'   {page["url"]}')
    if outline:
        print(f'   {outline}')

print(f"{total:,} characters of content collected")

Crawling several sites — one run per domain, since the crawl is hostname-scoped:

SITES = ["https://docs.example.com", "https://blog.example.org"]
corpus = []
for site in SITES:
    run = client.actor(ACTOR).call(run_input={
        "startUrl": site, "maxPages": 200,
        "outputFormat": "markdown", "deliveryMode": "none",
    })
    corpus += client.dataset(run["defaultDatasetId"]).list_items().items
print(f"{len(corpus)} pages across {len(SITES)} sites")

JavaScript / TypeScript (apify-client)

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const ACTOR = 'techforce.global/advanced-website-crawling-actor';

const run = await client.actor(ACTOR).call({
  startUrl: 'https://www.example.com',
  maxPages: 100,
  outputFormat: 'text',
  deliveryMode: 'none',
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();

// meta is an object; headings is an array; content can be very large.
const pages = items.map((p: any) => ({
  url: p.url,
  title: p.title,
  description: p.meta?.description ?? '',
  headingCount: (p.headings ?? []).length,
  chars: (p.content ?? '').length,
}));

const totalChars = pages.reduce((n, p) => n + p.chars, 0);
console.log(`${pages.length} pages · ${totalChars.toLocaleString()} characters`);

cURL — asynchronous with a spend cap (recommended)

A crawl visits many pages and renders the JS-heavy ones, so prefer async over the 300s synchronous endpoint.

# 1. Start execution with a hard spend limit
curl -X POST -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer <YOUR_APIFY_TOKEN>' \
  -d '{"startUrl":"https://docs.example.com","maxPages":200,"outputFormat":"markdown","deliveryMode":"none"}' \
  'https://api.apify.com/v2/acts/techforce.global~advanced-website-crawling-actor/runs?maxTotalChargeUsd=3.00'

# 2. Poll
curl -H 'Authorization: Bearer <YOUR_APIFY_TOKEN>' \
  'https://api.apify.com/v2/actor-runs/<runId>'

# 3. Retrieve when SUCCEEDED
curl -H 'Authorization: Bearer <YOUR_APIFY_TOKEN>' \
  'https://api.apify.com/v2/datasets/<DATASET_ID>/items?clean=true&format=json'

cURL — synchronous (small crawls only)

curl -X POST \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer <YOUR_APIFY_TOKEN>' \
  -d '{"startUrl":"https://www.example.com","maxPages":10,"deliveryMode":"none"}' \
  'https://api.apify.com/v2/acts/techforce.global~advanced-website-crawling-actor/run-sync-get-dataset-items'

---

⚠️ Autonomous Agent Error Handling & Resolution Matrix

Error Code	Detection Condition	Underlying Cause	Deterministic Agent Action
AUTH_INVALID	HTTP 401 from Apify	Missing or malformed APIFY_TOKEN	Abort & Prompt User: request a valid Apify API token.
RATE_LIMITED	HTTP 429 from Apify	Apify API rate / concurrency limits	Retry: exponential backoff (2s, 4s, 8s).
SYNC_TIMEOUT	HTTP 408 on the sync endpoint	Crawls routinely exceed the 300s ceiling	Reconfigure: async POST /runs + polling.
MISSING_START_URL	Input validation rejection	startUrl is required	Modify Input: supply one URL.
START_URL_UNREACHABLE	SUCCEEDED, 0 items	The start URL didn't load	Verify the URL, then retry; consider enabling a proxy.
ALL_URLS_FILTERED	SUCCEEDED, 0 items, patterns set	includePatterns matched nothing, or exclusions removed everything	Modify Input: loosen the patterns — exclusions always win.
ROBOTS_BLOCKED	Far fewer pages than expected, robots drops in the log	The site disallows those paths	Accept, or make a deliberate decision about respectRobotsTxt.
DUPLICATES_SKIPPED	Fewer pages than maxPages	Duplicate content skipping is on by default	Not an error. Set deduplicateContent: false only if you need every URL.
DEPTH_LIMITED	Crawl stopped shallow	maxDepth reached	Modify Input: raise maxDepth.
MAXPAGES_MISREAD	Run took far longer than page count suggests	maxPages counts saved pages, not visited	Fix Expectation: visits exceed saves.
FORMAT_FELL_BACK_TO_TEXT	Asked for a format, received plain text; log warns Unknown outputFormat	The value was not text, markdown or html	Modify Input: use a supported value — the log names them.
HTML_IS_CLEANED	html output lacks scripts, styles or sidebars	By design — html is the cleaned document	Fix Expectation: the original response body is not returned.
NO_DOCUMENTS	No PDFs in the output	crawlDocuments defaults to false	Modify Input: set it true.
META_NOT_A_STRING	String operations on meta throw	meta is an object	Fix Code: use meta.description.
HEADINGS_NOT_A_STRING	Join/split on headings misbehaves	It's an array	Fix Code: treat as a list.
CONTENT_TOO_LARGE	Memory pressure, or a connector rejects a payload	content holds a whole page	Fix Code: stream the dataset; delivery already truncates.
NO_STRUCTURED_FIELDS	User wanted price / contact / listing fields	This is a content crawler	Re-route: a purpose-built scraper, or Website Contact Scraper for contacts.
MULTI_DOMAIN_UNSUPPORTED	Agent passes several domains	The crawl is hostname-scoped	Loop: one run per site.
AUTH_PAGE_REQUESTED	Target needs a login	No authentication support	Abort & Prompt User: public pages only.
PAGE_CAP_REACHED	Exactly 200 pages	Hard cap on saved pages	Reconfigure: split by section with include patterns.
DELIVERY_SKIPPED	Log no tool name was provided	mcpConnector set but mcpTool empty	Modify Input: set mcpTool; the dataset is already saved.
DELIVERY_TOOL_UNKNOWN	Log Tool 'x' is not available on this connector	Wrong tool name for that connector	Modify Input: pick a name from the Available tools: list in the same log line.
DELIVERY_ARGS_INVALID	Connector rejects the arguments	mcpArguments doesn't match the tool's schema	Read the log: the line Connector [<tool>] expects arguments matching this schema: prints the exact shape.
DELIVERY_TRUNCATED	Delivered content shorter than the dataset's	Delivery caps content at 4,000 characters	By design. {truncated} flags it; the dataset keeps the full text.
DELIVERY_FLOOD	Up to 200 connector calls	deliveryMode: "perPage"	Reconfigure: use summary or chunked.
MCP_PROXY_MISSING	Log APIFY_MCP_PROXY_URL is not set	Running locally instead of on the platform	Reconfigure: apify push and run on the platform.
CHARGE_LIMIT	Run aborted mid-crawl	maxTotalChargeUsd spend threshold reached	Prompt User: confirm a higher spend cap.

⚠️ Verify before publishing: the Apify HTTP status rows (401, 408, 429) reflect standard platform behavior. Every crawl-side and field-level row is taken from this Actor's own contract. Re-confirm the platform rows if you depend on them for automated retry logic.

---

🗣️ Natural Language → Actor Mapping

User says	Intent	Constructed Actor input
"Crawl this site so I can ask questions about it"	RAG ingestion	{"startUrl":"<site>","maxPages":200,"outputFormat":"markdown"}
"Turn these docs into markdown"	Docs export	{"startUrl":"<docs>","outputFormat":"markdown","includePatterns":["/docs/*"]}
"Give the assistant context about this company"	LLM grounding	{"startUrl":"<site>","maxPages":50,"outputFormat":"text"}
"Index our site for search"	Search index	{"startUrl":"<site>","maxPages":200} then index content
"Audit all our page content before the redesign"	Content audit	{"startUrl":"<site>","maxPages":200,"crawlDocuments":true}
"Just the blog, not the archives"	Scoped crawl	{"includePatterns":["/blog/*"],"excludePatterns":["/blog/page/*"]}
"Include the PDFs they link to"	Documents	{"crawlDocuments":true}
"What content does our competitor publish?"	Research	{"startUrl":"<competitor>","maxPages":200}
"Push each page into Notion"	Delivery	{"mcpConnector":"notion","deliveryMode":"perPage","mcpTool":"notion-create-pages"}
"Why did I get fewer pages than I asked for?"	Diagnosis	Duplicates skipped, patterns, robots.txt, or maxDepth — the log gives the reason per URL
"Get their pricing and contact details"	Structured fields	❌ Wrong tool — use a purpose-built scraper
"Get emails and phone numbers"	Contacts	➡️ Website Contact Scraper
"Give me the HTML, not plain text"	Markup	{"outputFormat":"html"} — cleaned markup with tables and links intact
"I need the original page source exactly as served"	Raw body	❌ html returns the cleaned document, not the raw response
"Crawl these five different sites"	Multi-domain	❌ One site per run — loop
"Crawl the members-only area"	Authenticated	❌ No login support
"Download their images"	Assets	❌ Text content only
"Give me embeddings"	Vectors	❌ Content only — embed downstream

Should NOT route here: structured field extraction (prices, contacts, listings) · emails, phones and social links · several domains in one run · pages behind a login · the original unmodified response body · images, video or binary assets · embeddings or pre-chunked vectors · form filling or click-through flows · crawls larger than 200 saved pages.

---

🧭 Agent Execution & Routing Logic

[Input User Query]
       │
       ▼
1. Does the user want a site's READABLE CONTENT, or SPECIFIC FIELDS?
       ├── specific fields (price, contact, listing) → Route to a purpose-built scraper.
       │                                               For contacts: Website Contact Scraper
       └── the content itself                        → stay here
       ▼
2. Is the target public?                              NO → Abort: no login support
       ▼
3. One site, or several?
       ├── one       → startUrl
       └── several   → LOOP, one run per domain (the crawl is hostname-scoped)
       ▼
4. What shape does the consumer need?
       ├── RAG / LLM context / docs → outputFormat "markdown" (structure survives)
       ├── search index / audit     → outputFormat "text"
       ├── tables/links/attributes  → outputFormat "html" (cleaned markup)
       └── the ORIGINAL response body → not available; html is the CLEANED document
       ▼
5. Scope the crawl so budget goes where it matters:
       includePatterns to target a section, excludePatterns to drop noise.
       Remember EXCLUSIONS ALWAYS WIN over inclusions
       ▼
6. Does the site's real content live in PDFs?   YES → crawlDocuments true
       │                                              (off by default)
       ▼
7. Set maxPages (1-200). It counts SAVED pages — the crawl may visit many more,
       so run time and cost track visits, not saves
       ▼
8. Deliver to a connector? YES → mcpConnector + mcpTool + {message}
       │                          summary (1 call, no content) / chunked /
       │                          perPage (1 per page, content truncated to 4000 chars)
       │ NO  → set deliveryMode "none" (it defaults to summary)
       ▼
9. Use the ASYNC endpoint — crawls exceed 300s routinely
       ▼
[Execute]
       │
       ├──► SUCCEEDED, items > 0 ──► handle the field contract:
       │        meta is an OBJECT   → meta.description
       │        headings is an ARRAY → use it as your chunk boundary for RAG
       │        content is LARGE     → stream rather than loading everything
       │        type "document" only appears if crawlDocuments was on
       │        dedupe across runs on `url`
       │
       └──► fewer pages than maxPages ──► NOT a failure. In likelihood order:
                duplicate content skipped (on by default)
                include/exclude patterns filtered URLs out
                robots.txt disallowed paths
                maxDepth reached
                the site genuinely has fewer pages
           The log records a reason per dropped URL.

---

💰 Cost Control & Pricing Transparency

Pay-per-result: you are charged per page saved, with no monthly subscription. The exact rate is on the Actor's Pricing tab in the Apify Store.

Autonomous spend caps: pass maxTotalChargeUsd as a query parameter on the run endpoint for a hard per-execution ceiling.

⚠️ Verify before publishing: this repo contains no .actor/pay_per_event.json and the code makes no Actor.charge() calls, so billing is configured entirely on the Store listing. Confirm the live pricing model and rate before publishing.

Saves are billed; visits are what cost time

maxPages caps what you're charged for, but the crawl visits more than it saves — filtered URLs, duplicates and robots-disallowed paths are all visited or considered first. Platform usage tracks visits, not saves, and the JS-heavy pages among them are the expensive ones.

Configuration	Effect on cost
includePatterns scoped to one section	💚 Biggest saving — the crawler never visits the rest
maxDepth lowered	💚 Stops the crawl fanning out
deduplicateContent: true (default)	💚 Avoids paying for near-identical pages
maxPages raised toward 200	🟠 More saves, and proportionally more visits
crawlDocuments: true	🟠 Documents are larger and slower than pages
A site that's mostly JavaScript	🔴 Rendering is the dominant cost

Cost levers:

• Scope with includePatterns rather than trimming maxPages. Capping saves still pays for the visits; excluding a section avoids them entirely.
• Validate at maxPages: 5 before a full crawl, especially on an unfamiliar site.
• Leave duplicate skipping on. It is a pure saving on most real sites.
• Only enable crawlDocuments when documents matter — they're heavier than pages.
• Prefer summary/chunked delivery over perPage — one call instead of up to 200.
• Cap the spend with maxTotalChargeUsd on any agent-driven run.

---

🔍 Companion machine-readable files

File	Purpose
.actor/actor.json	Identity, title, description, version, and the input / output / dataset schema references
.actor/input_schema.json	Authoritative typed input contract — start URL, page cap, document toggle, output format, URL patterns, crawl controls, proxy, and the 5 delivery fields
.actor/dataset_schema.json	Two Console Output views — Overview (titles and structure) and Content (the extracted text)
.actor/output_schema.json	Declares where results are stored

If any table on this page disagrees with the schema files, the schema files win.

---

🛠️ Troubleshooting

#	Symptom you see	Most likely cause	Fix
1	0 pages	The start URL didn't load	Verify the URL; enable a proxy if the site blocks you.
2	0 pages with patterns set	includePatterns matched nothing, or exclusions removed everything	Loosen the patterns — exclusions always win.
3	Fewer pages than maxPages	Duplicates skipped, patterns, robots.txt, or maxDepth	Check the per-URL drop reasons in the log.
4	Far fewer pages than the site clearly has	robots.txt disallows those paths	Expected. Decide deliberately whether to change that.
5	Run took much longer than the page count suggests	maxPages counts saved pages; many more were visited	Scope with includePatterns to cut visits.
6	Asked for a format, got plain text	The value was not text/markdown/html — the log warns	Use a supported value (case-insensitive).
7	No PDFs in the output	crawlDocuments defaults to false	Set it true.
8	type is never "document"	Same cause	Enable crawlDocuments.
9	String operations on meta throw	meta is an object	Use meta.description.
10	headings behaves oddly	It's an array, not a string	Treat it as a list.
11	Memory pressure downstream	content holds whole pages	Stream the dataset instead of loading it all.
12	Connector rejected a page	Content too large for that service	Delivery already truncates to 4,000 chars — check {truncated}.
13	Content includes navigation or boilerplate	Extraction is heuristic, not perfect	Post-process, or scope the crawl more tightly.
14	Markdown looks noisy on some pages	Heavily styled pages convert imperfectly	Try text for those, or clean downstream.
15	Exactly 200 pages	Hard cap on saved pages	Split the crawl by section using include patterns.
16	Crawl fans out into irrelevant sections	maxDepth too high, or no include patterns	Lower maxDepth; add includePatterns.
17	Only one domain crawled	The crawl is hostname-scoped	Loop, one run per site.
18	HTTP 408 on run-sync-get-dataset-items	The synchronous endpoint has a hard 300-second ceiling	Use async POST /runs → poll → fetch.
19	Up to 200 connector calls	deliveryMode: "perPage"	Switch to summary or chunked.
20	Slack / Notion received nothing	Delivery needs both mcpConnector and mcpTool	Set both. The dataset is still written in full.
21	Connector fired but the body is empty	mcpArguments did not reference {message}	Map {message} and populate mcpMessageTemplate.
22	Log APIFY_MCP_PROXY_URL is not set	You ran locally; connectors only resolve on the platform	Deploy with apify push and run on the platform.
23	HTTP 401 / 403 from Apify	Missing, expired or malformed APIFY_TOKEN	Regenerate in Apify Console → Settings → API & Integrations.
24	HTTP 429 from Apify	Apify account concurrency / rate limits	Retry with exponential backoff (2s → 4s → 8s).

Known limitations

• html output is the cleaned document, not the byte-for-byte original response body — scripts, styles and sidebars are already removed.
• Content only — no structured fields, no images, no binary assets.
• One domain per run, and a hard cap of 200 saved pages.
• No authentication, so members-only areas can't be crawled.
• Extraction is heuristic. Most pages come out clean; some retain boilerplate, and heavily styled pages convert to markdown imperfectly.
• PDF/Word extraction is opt-in and heavier than ordinary pages.

Diagnostic checklist before opening an issue

1. Baseline run. {"startUrl":"<your site>","maxPages":5} with no patterns. Cheapest proof the pipeline works.
2. Pages returned? If yes, the Actor is fine and your patterns, depth or page cap shaped the result.
3. Read the drop reasons. Every skipped URL is logged with why — depth, pattern mismatch, robots, or duplicate. That single check answers most "why so few pages" questions.
4. Check the shape, not just the count. Missing PDFs on a default run is expected; type never being "document" without crawlDocuments is expected.
5. Try text versus markdown on a page that looked wrong — conversion quality varies with how the page is built.

If the issue survives all five steps, open an Issues ticket on the Actor page (or email support) with the run ID, the exact input JSON, and a url whose content came out wrong.

❓ FAQ

Scope

What does this Actor actually give me?

The readable content of a website, page by page — as text, markdown or cleaned HTML — plus each page's title, meta description and heading outline. It's built for feeding content into something else: a vector store, a search index, an LLM, a migration.

Can it extract prices, contact details or listing data?

No — that's a different job. A content crawler gives you the text; a purpose-built scraper gives you fields. For contacts specifically, use Website Contact Scraper.

Can I crawl several sites at once?

No — the crawl is scoped to the start URL's hostname. Loop it, one run per site.

Can it crawl pages behind a login?

No. Public pages only.

Can I plug it into Claude, Cursor, or a LangChain agent?

Yes — it is a native MCP tool:

claude mcp add --transport http apify "https://mcp.apify.com?tools=techforce.global/advanced-website-crawling-actor"

Configuration

Where are all the settings? The Console only shows three fields.

Deliberately — the form stays simple. Output format, crawl depth, URL patterns, concurrency, duplicate handling and robots.txt behaviour are all real inputs available through the API and MCP.

Text or markdown?

Markdown when structure matters — documentation, anything an LLM will read, anything you'll chunk on headings. Text when you just want the words, for a search index or a word-count audit.

Can I get HTML?

Yes — outputFormat: "html" returns the page markup with tables, links, code and emphasis intact. It is the cleaned document, though: scripts, styles and sidebars are stripped, exactly as they are for text and markdown. If you need the original response body byte-for-byte, that is not something this Actor returns.

How do I crawl just one section?

includePatterns, e.g. ["/docs/*"]. That's also the best cost lever, because the crawler never visits the rest of the site.

Why didn't it pick up the linked PDFs?

crawlDocuments is off by default. Set it true and PDF/Word files are crawled and saved with type: "document".

Output

Why is meta not a string?

It's an object — {"description": "..."} — so it can carry more metadata later without breaking the field's type. Use meta.description.

What is headings for?

It's the page's H1/H2/H3 outline in order, which makes it the natural place to chunk for RAG rather than splitting on an arbitrary character count.

Why did I get fewer pages than I asked for?

maxPages is a ceiling, not a target. Duplicate content is skipped by default, URL patterns filter pages out, robots.txt may disallow paths, and maxDepth bounds the fan-out. The log gives a reason for every dropped URL.

Is the content perfectly clean?

Mostly. Navigation, scripts and styling are stripped, but extraction is heuristic — some pages retain a little boilerplate, and heavily styled pages convert to markdown imperfectly.

Why is delivered content shorter than what's in the dataset?

Delivery truncates page content to 4,000 characters so a single connector call can't blow a service limit. The dataset always keeps the full text, and {truncated} tells you when truncation happened.

Pricing

How is this billed?

Pay-per-result — charged per page saved, with no monthly subscription. The current rate is on the Actor's Pricing tab. Apify platform usage is billed separately.

Why did a 50-page crawl cost more than expected?

Because maxPages caps saves, not visits — filtered, duplicate and disallowed URLs are considered first, and rendering JavaScript-heavy pages is the dominant cost. Scoping with includePatterns reduces visits; lowering maxPages does not.

How do I guarantee an agent never overspends?

Pass maxTotalChargeUsd as a query parameter on the run endpoint.

---

🔗 Related Actors

Site data, by what you actually need

Actor	Returns	Why pick it over this one
This Actor	A site's readable content as text, markdown or HTML	RAG, search indexing, content audits, LLM grounding
Website Contact Scraper	Emails, phones and social links per domain	You want to contact the business, not read its site
Visual Verification Agent	Whether a page renders correctly, graded A+ to C	QA and release verification, not content
Google Maps Business Leads & Sales Intelligence	Local business listings with contact data	You need the businesses, not their site text

ℹ️ These compose well. A crawl tells you what a company says; the contact scraper tells you how to reach it; the verification agent tells you whether its site actually works.

Suggested pipeline patterns

🧠 RAG corpus in two steps Crawl the docs as markdown → chunk on headings with title and meta.description as context → embed. The output is shaped for exactly this.

🔍 Site search index Crawl as text → index content with title and url → a working internal search without touching the CMS.

📋 Pre-migration content audit Crawl with crawlDocuments: true → inventory every page and attachment with its headings → find orphans and duplicates before you rebuild.

🏢 Company research pack Crawl a prospect's site → Website Contact Scraper for the contact route → a briefing that covers both what they do and who to call.

📚 Notion knowledge base Crawl as markdown → deliveryMode: "perPage" into Notion → a searchable internal mirror of an external documentation site.

Browse all Actors by Techforce Global at scraper.techforce.global.

---

🔐 Compliance & Data Privacy

This Actor crawls public web pages on a domain you specify. It does not log in, does not accept credentials, does not submit forms, and does not follow links off that hostname.

Content crawling for AI raises questions that field-level scraping doesn't, so four points deserve real attention:

• Crawl what you own or are authorised to crawl. It's built for your own sites, clients who've engaged you, and public research. Repeatedly crawling a third party's site without permission is a different activity with different risks — and robots.txt is respected by default for a reason. Overriding it is a deliberate decision you own.
• Site content is usually someone's copyrighted work. Ingesting it into a RAG system, a search index or model context does not transfer any rights. Whether your use is permitted depends on your jurisdiction, the site's terms, and what you do with the output — republishing crawled text as your own is the clearest way to get this wrong. Keep the url with every chunk so you can always attribute and trace provenance.
• Pages contain personal data more often than you'd expect. Team pages, author bylines, testimonials and staff directories all name real people, and crawling a whole site sweeps them up indiscriminately. If that content lands in a vector store, you have taken on GDPR/UK GDPR and CCPA obligations — including erasure requests, which are genuinely hard to honour once text is embedded. Scope your crawl with excludePatterns to avoid the sections you don't need.
• A crawl is a point-in-time snapshot. A RAG system built on a stale crawl will answer confidently from outdated content. Re-crawl on a cadence that matches how fast the source changes, and keep the crawl date alongside the text.

The Actor is built to crawl politely — robots.txt honoured by default, bounded concurrency, retries with limits, duplicate skipping. Don't defeat that by running many parallel crawls against one host. You remain responsible for each target site's terms of service and for all applicable copyright and data-protection law in the markets you operate in.

---

🆘 Support & Custom Pipeline Engineering

Need crawls beyond 200 pages, multi-domain orchestration, authenticated crawling, real HTML output, custom content cleaning for your document model, or a full RAG ingestion pipeline with embeddings and scheduled refresh?

• Email: bhavin.shah@techforceglobal.com
• Custom Enterprise Integrations: Book a 15-Minute Technical Consultation
• Maintained by: Techforce Global — Specialists in High-Performance Web Scrapers and Agentic Workflows.

---

Made with ❤️ by Techforce Global Specialists in High-Performance Web Scrapers and AI Automation.

---

🏷️ Structured data for search & AI discovery

{
  "@context": "https://schema.org",
  "@type": "SoftwareApplication",
  "name": "Website Content Crawler — Whole-Site Text, Markdown & HTML for RAG and Search",
  "applicationCategory": "DeveloperApplication",
  "operatingSystem": "Cloud (Apify platform)",
  "description": "Crawls a website from a single start URL and returns each page's readable content as clean text, structured markdown or cleaned HTML, together with the page title, meta description and heading outline — built for RAG ingestion, LLM context, search indexing, content audits and site migrations. JavaScript-rendered pages are handled automatically, duplicate content is skipped, robots.txt is respected by default, and URL glob patterns scope the crawl. Linked PDF and Word documents can be included optionally. Results can be delivered into Notion, Slack, Airtable, Google Sheets or any authorized MCP connector.",
  "url": "https://apify.com/techforce.global/advanced-website-crawling-actor",
  "offers": {
    "@type": "Offer",
    "priceCurrency": "USD",
    "description": "Pay-per-result, charged per page saved. The current rate is listed on the Actor's Pricing tab."
  }
}