SiteVerdict — Web Page to Clean Markdown for AI (LLM-ready)

Turn any public web page, PDF or Word file into clean, citation-ready Markdown and embedding-ready RAG chunks. Measured: highest main-content accuracy of the Apify defaults we tested (F1 91.9% vs 84.7% for Website Content Crawler's default), 98.9% of key facts kept, and you pay only for pages that actually changed.

• Accurate main content. Mozilla Readability plus our recovery step finds the paragraphs Readability drops and removes the "Related / More stories / Share / Newsletter" noise it keeps. See the benchmark.
• RAG-ready. Heading-aware chunks with exact OpenAI token counts and presets for OpenAI, Cohere, Claude/Voyage and 512-token open models. Every chunk carries its source URL, heading path, deep-link anchor (or PDF page) and a citation.
• Cheap to keep fresh. Monitoring mode re-checks pages with ETag / Last-Modified and sitemap <lastmod>, outputs only new and changed pages with a section-level diff, and alerts a Slack / webhook URL. Unchanged pages are free.
• Compliance you can show your legal team. robots.txt is always obeyed (on every redirect), paywalls, logins, CAPTCHAs and noai / TDM opt-outs are respected, and such pages are returned with no content and no charge.

3 things people use it for

1. Load a docs site / help centre into a vector database. mode: "sitemap", includeGlobs: ["/docs/**"], chunkPreset: "openai-text-embedding-3". Export the RAG chunks view straight to CSV / JSON, or pipe it into Pinecone, Qdrant or Weaviate. Each chunk has a stable chunkId.
2. Give an AI agent a "read this URL" tool. One URL in, clean Markdown and a citation out, plus a clear blocked / paywalled status the agent can explain instead of hallucinating. MCP-ready (fetch_clean_markdown).
3. Keep a knowledge base fresh on a schedule. Turn on monitoringMode, schedule daily or weekly runs, get only what changed (with which sections changed) and a Slack alert. Optional AI summary and Q&A pairs per page with your own Anthropic or OpenAI key.

Before / after

GOV.UK "VAT rates" (Crown copyright, Open Government Licence v3.0). A page-to-Markdown tool that only strips nav / footer (RAG Web Browser's default) starts like this:

## Cookies on GOV.UK
We use some essential cookies to make this website work.
We'd like to set additional cookies to understand how you use GOV.UK …
You have accepted additional cookies. You can [change your cookie settings](/help/cookies) at any time.
You have rejected additional cookies. …
Skip to main content
# VAT rates
…
Is this page useful? Maybe Yes this page is useful No this page is not useful …

SiteVerdict returns only the content, with absolute links and the table intact:

# VAT rates

The standard VAT rate is _20%_

## VAT rates for goods and services

|   | % of VAT | What the rate applies to |
| --- | --- | --- |
| Standard rate | 20% | Most goods and services |
| Reduced rate | 5% | Some goods and services, for example, children's car seats and home energy |
| Zero rate | 0% | Zero-rated goods and services, for example, most food and children's clothes |
The standard rate of VAT increased to 20% on 4 January 2011 (from 17.5%).
…

On a product page you also get a one-line fact summary from the page's schema.org data, e.g. **Price:** 55 GBP · **Availability:** In Stock · **Brand:** IKEA · **SKU:** 00263850 · **Rating:** 4.6 (1200 reviews), and a structured product field.

Benchmark (honest numbers)

30 real public pages (docs, news, blogs, e-commerce product pages, wikis, forums, government), hand-checked references, scored offline. Competitors are rebuilt from their documented default settings with the same open-source parts (we do not run other Actors). Full method, per-page scores, caveats and scripts: BENCHMARK.md.

	SiteVerdict v2	Website Content Crawler (default)	WCC + Defuddle	RAG Web Browser (default)
Main-content F1	91.9%	84.7%	86.9%	80.6%
Recall / precision	95.4% / 90.2%	83.7% / 89.2%	88.1% / 88.7%	96.2% / 73.1%
Key facts kept (95)	98.9%	82.0%	87.6%	95.5%
Boilerplate leaks (95 checks)	4	2	5	34
Code blocks exact (121)	112	109	111	107
Output tokens, 30 pages	79k	79k	83k	107k
Obeys robots.txt by default	always	off by default	off by default	no option

Where we don't win: RAG Web Browser keeps slightly more text (at 73% precision); WCC's default is faster per page and leaks a little less boilerplate but misses 18% of key facts; JavaScript-heavy shop pages are hard for everyone in HTTP mode. Live compliance check: 5 out of 5 blocked pages (robots.txt, unreachable robots.txt, 403, HTTP-200 bot interstitial) returned no content and no charge.

What you get per page

markdown · text · title, author, publishedAt, modifiedAt, language, siteName, canonicalUrl · heading outline with anchors · links (internal / external) · images · jsonLd · product (PDPs) · wordCount, exact tokens + tokenizer · contentHash, etag · chunks[] (chunkId, headingPath, anchorUrl, page for PDFs, tokens, citation) · citation (title, URL, site, author, dates, accessed, ready-made text) · changeStatus + changeSummary (monitoring) · ai (summary, key points, Q&A, optional).

Dataset views: Pages, Markdown, RAG chunks (one row per chunk, CSV-ready), Changes, AI summary & Q&A, Products, Blocked / skipped. The run's OUTPUT record is an agent-ready JSON summary, and monitoring runs also write a CHANGES record.

Pricing (pay per event)

Event	Price	When
page-converted	$0.0015	A web page converted over HTTP
page-converted-browser	$0.004	A page that needed a headless browser (renderingMode auto / browser)
document-converted	$0.003	A PDF or DOCX converted (any length up to 500 pages / 25 MB)
ai-enrichment	$0.002	AI summary / Q&A written for a page with your API key (your provider bills its tokens to you)

Never charged: blocked, paywalled, opt-out, duplicate, too-short, failed, unchanged (monitoring) and removed pages; robots.txt and sitemap fetches; llms.txt. There is no start fee.

Examples: 100 pages ≈ $0.15 · 1,000-page docs site, chunked ≈ $1.50 · weekly monitoring of 1,000 pages where 50 changed ≈ $0.075 a week · 20 PDFs ≈ $0.06 · 500 pages + AI Q&A ≈ $1.75 plus your own LLM cost.

Quick start

Two pages:

{ "startUrls": [{ "url": "https://en.wikipedia.org/wiki/Markdown" }, { "url": "https://docs.python.org/3/library/json.html" }] }

A docs site into OpenAI-sized chunks:

{
  "startUrls": [{ "url": "https://docs.example.com/" }],
  "mode": "sitemap",
  "includeGlobs": ["/guides/**"],
  "maxPages": 1000,
  "chunkPreset": "openai-text-embedding-3"
}

Weekly freshness check with a Slack alert (schedule this input):

{
  "startUrls": [{ "url": "https://docs.example.com/" }],
  "mode": "sitemap",
  "maxPages": 5000,
  "monitoringMode": true,
  "monitorId": "example-docs",
  "webhookUrl": "https://hooks.slack.com/services/…"
}

PDF reports with per-page citations, plus Q&A pairs for a RAG eval set:

{
  "startUrls": [{ "url": "https://www.rfc-editor.org/rfc/rfc9309.pdf" }],
  "chunkPreset": "voyage-claude-rag",
  "aiEnrichment": "summary+qa",
  "aiProvider": "anthropic",
  "aiApiKey": "sk-ant-…"
}

Crawl a small site and draft an llms.txt: "mode": "crawl", "maxDepth": 2, "generateLlmsTxt": true. llms.txt and llms-full.txt are saved to the key-value store.

Output example (shortened)

{
  "url": "https://docs.widget.dev/cli/configuration",
  "status": "ok",
  "changeStatus": "changed",
  "changeSummary": { "changedSections": ["Configuration reference > Options"], "addedSections": [], "removedSections": [], "wordDelta": 14 },
  "extractor": "readability+recovery",
  "documentType": "html",
  "title": "Configuration reference",
  "language": "en",
  "canonicalUrl": "https://docs.widget.dev/cli/configuration",
  "markdown": "# Configuration reference\n\nThe Widget CLI reads its settings from `widget.config.json` …",
  "tokens": 262,
  "tokenizer": "cl100k",
  "chunks": [
    {
      "chunkId": "8fa7c16d9555369133945cd2",
      "headingPath": ["Configuration reference", "Options"],
      "anchorUrl": "https://docs.widget.dev/cli/configuration#options",
      "text": "Configuration reference > Options\n\n## Options\n\n| Option | Type | …",
      "tokens": 148,
      "citation": { "title": "Configuration reference", "url": "https://docs.widget.dev/cli/configuration#options", "accessedAt": "2026-10-11T07:26:22Z" }
    }
  ],
  "citation": { "title": "Configuration reference", "url": "https://docs.widget.dev/cli/configuration", "text": "\"Configuration reference\". Widget Docs. https://docs.widget.dev/cli/configuration. accessed 2026-10-11" },
  "charged": "page-converted"
}

A page we do not convert (no content, not charged):

{ "url": "https://lobste.rs/s/u9ue6c/…", "status": "blocked", "reason": "robots-txt", "detail": "robots.txt disallows … (disallow: / (line 18))", "charged": false }

Statuses: ok, unchanged, removed, blocked (robots-txt, bot-protection, http-403, login-required, ai-opt-out, http-451, private-address), paywalled, duplicate, too-short, unsupported-content (no-text-layer for scanned PDFs, images, archives …), error (not-found, rate-limited, network …).

Options that matter

• Rendering: http (default, fastest, cheapest), auto (headless Chromium only for JavaScript app shells), browser (every page).
• Extraction: auto (Readability + recovery, recommended), readability, main-element, full-page; contentSelector forces a region, removeSelectors strips extra blocks, removeReferenceLists (on) strips Wikipedia-style citation noise.
• Chunking: chunkPreset (openai-text-embedding-3 512/64, openai-large-context 1024/128, cohere-embed-v3 400/40, voyage-claude-rag 800/100, small-open-model 320/32) or custom with chunkSize, chunkOverlap and tokenizer (cl100k, o200k, estimate). Chunks split at headings, keep code and tables whole (big tables repeat their header), and overlap whole sentences within a section.
• Documents: convertDocuments (on) converts PDF and DOCX; followDocumentLinks also queues in-scope document links while crawling.
• Monitoring: monitoringMode, monitorId, outputUnchanged, useSitemapLastmod, reportRemoved, webhookUrl (HTTPS, secret), webhookOnNoChanges.
• AI enrichment: aiEnrichment (summary, qa, summary+qa), aiProvider, aiApiKey (secret), aiModel (default claude-sonnet-5-5 / gpt-4o-mini; choose e.g. claude-haiku-5-5 for cheaper bulk runs), aiQaPairs, aiMaxInputChars, aiLanguage.
• Politeness: at least 1 s between requests to the same host, robots.txt Crawl-delay honoured (up to 30 s), retries only for 429 / 5xx / network errors.

FAQ

Can it get past Cloudflare, a CAPTCHA or a login? No, by design. Those pages come back as blocked and cost nothing. Use the site's API or licence, or ask the owner.

How exact are the token counts? Exact for OpenAI models (tiktoken cl100k_base / o200k_base). Claude, Cohere and Voyage tokenizers are not published for offline use, so their presets count with cl100k plus a safety margin (x1.15 to x1.25) so chunks stay under the model's limit.

What does monitoring cost? Only new and changed pages are charged. A 304 Not Modified, an identical content hash, or an unchanged sitemap <lastmod> costs nothing. Removed pages are reported for free.

Does AI enrichment use your key or mine? Yours, always. It is off by default, stored as an encrypted secret input, sent only to the provider you choose, and never logged. We charge $0.002 per page only when the model returns valid output.

PDFs? Text-layer PDFs and Word .docx files are converted (title, author and dates from the document metadata, one citation per PDF page). Scanned PDFs without a text layer are reported as no-text-layer and not charged (no OCR).

Why is a page "paywalled" when I can read part of it? The publisher marks it as not free (isAccessibleForFree: false), or the page shows a subscription wall over a short teaser. We return nothing, not even the teaser.

Can I change the user agent? No. Requests always identify as SiteVerdictBot. You can add your own robots.txt token, but it can only add restrictions; other crawlers' names (Googlebot, GPTBot …) are refused.

When should I use Website Content Crawler instead? WCC is mature and very capable: proxies, several crawler engines, cookie-dialog clicking, file downloads, huge crawls. If you need to get through hard-to-crawl sites or run proxied crawls of millions of pages, use WCC. Use this Actor when you want measured extraction quality, RAG-ready chunks with exact token counts, cheap incremental monitoring, PDF/DOCX conversion and compliance you can document.

Limits

• Max 20,000 pages per run; 8 MB HTML / 25 MB documents per URL; PDFs up to 500 pages; 30 s timeout per request by default.
• No proxies, no logins, no consent-dialog clicking, no OCR, no screenshots.
• Extraction is heuristic: unusual layouts (especially JavaScript-heavy shop pages) may need renderingMode: "auto" or a contentSelector.
• PDF text order follows the PDF's text layer; complex multi-column layouts and tables in PDFs come out as paragraphs.
• Paywall detection is conservative towards publishers: occasionally a free page with a "subscribe to continue reading" box is marked paywalled.
• Monitoring memory lives in a named key-value store in your Apify account (siteverdict-w2md-monitor-<monitorId>); delete it to start fresh.
• Language is what the page declares; it is not detected from the text.

Compliance, sources and responsible use

• Only public pages are fetched, with an honest user agent: Mozilla/5.0 (compatible; SiteVerdictBot/2.0.0; SiteVerdict/web-to-markdown; +aman.thakran@bharatfare.com). Site owners can control access with User-agent: SiteVerdictBot in robots.txt.
• robots.txt follows RFC 9309 (5xx or unreachable means "disallow all"; 4xx means "allow all"), and is checked on every redirect hop and browser navigation.
• Paywalls (schema.org isAccessibleForFree, content_tier, paywall markup), login walls, CAPTCHAs / bot challenges (including HTTP-200 "waiting room" interstitials) and AI / text-and-data-mining opt-outs (noai, tdm-reservation: 1) are respected: no content, no charge, never bypassed. After 3 blocks a host is skipped.
• Copyright stays with the publisher. Converting a page does not give you a licence to republish it or train models on it. Check each site's terms (e.g. Wikipedia text is CC BY-SA and needs attribution; GOV.UK content is OGL v3.0). Use the citation field to attribute.
• Personal data: the output contains only what the page publishes (e.g. an author byline). E-mail addresses and phone numbers are stripped from author fields. If you process pages about individuals you are the controller under UK GDPR / EU GDPR: have a lawful basis, keep only what you need, and do not use the output for unsolicited marketing (PECR).
• AI enrichment sends page text to the provider you choose under your account and its terms.
• Requests to private or internal network addresses (including webhook URLs) are refused.
• Where results are stored: the run's default dataset and key-value store in your own Apify account. These are unnamed; how long Apify keeps them is set by Apify and your plan, not by SiteVerdict (Apify data retention). You can delete them sooner in Apify Console.
• Monitoring (off by default): to skip unchanged pages, a named key-value store siteverdict-w2md-monitor-<monitor id> keeps, per page, a content hash, section hashes, the title, word count, ETag/Last-Modified and dates (not the page text). Apify keeps named stores until you delete them.
• What SiteVerdict keeps: nothing. The Actor runs in your Apify account and sends no data to SiteVerdict; our contact address appears only in the User-Agent sent to the sites it reads.
• This is not legal advice. When in doubt, ask the site owner.

Built by SiteVerdict (BharatFare Travel and Technologies Ltd, UK). Open-source components: Mozilla Readability (Apache-2.0), turndown (MIT), linkedom (ISC), unpdf / pdf.js (MIT / Apache-2.0), mammoth (BSD-2-Clause), js-tiktoken (MIT), Anthropic TypeScript SDK (MIT).