# SiteVerdict — Web Page to Clean Markdown for AI (LLM-ready) (`siteverdict/siteverdict-web-to-markdown`) Actor Turn public web pages, sitemaps or whole sites into clean Markdown, text and RAG chunks with title, author, dates, canonical URL, outline, links, tables and JSON-LD. Respects robots.txt, paywalls and AI opt-outs. Optional llms.txt. - **URL**: https://apify.com/siteverdict/siteverdict-web-to-markdown.md - **Developed by:** [Aman Thakran](https://apify.com/siteverdict) (community) - **Categories:** AI, Developer tools - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing Pay per event This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## SiteVerdict — Web Page to Clean Markdown for AI (LLM-ready) **Turn any public web page, PDF or Word file into clean, citation-ready Markdown and embedding-ready RAG chunks. Measured: highest main-content accuracy of the Apify defaults we tested (F1 91.9% vs 84.7% for Website Content Crawler's default), 98.9% of key facts kept, and you pay only for pages that actually changed.** - **Accurate main content.** Mozilla Readability plus our recovery step finds the paragraphs Readability drops and removes the "Related / More stories / Share / Newsletter" noise it keeps. See the [benchmark](#benchmark-honest-numbers). - **RAG-ready.** Heading-aware chunks with **exact OpenAI token counts** and presets for OpenAI, Cohere, Claude/Voyage and 512-token open models. Every chunk carries its source URL, heading path, deep-link anchor (or PDF page) and a citation. - **Cheap to keep fresh.** Monitoring mode re-checks pages with ETag / Last-Modified and sitemap ``, outputs only **new and changed** pages with a section-level diff, and alerts a Slack / webhook URL. **Unchanged pages are free.** - **Compliance you can show your legal team.** robots.txt is always obeyed (on every redirect), paywalls, logins, CAPTCHAs and `noai` / TDM opt-outs are respected, and such pages are returned with **no content and no charge**. ### 3 things people use it for 1. **Load a docs site / help centre into a vector database.** `mode: "sitemap"`, `includeGlobs: ["/docs/**"]`, `chunkPreset: "openai-text-embedding-3"`. Export the **RAG chunks** view straight to CSV / JSON, or pipe it into Pinecone, Qdrant or Weaviate. Each chunk has a stable `chunkId`. 2. **Give an AI agent a "read this URL" tool.** One URL in, clean Markdown and a citation out, plus a clear `blocked` / `paywalled` status the agent can explain instead of hallucinating. MCP-ready (`fetch_clean_markdown`). 3. **Keep a knowledge base fresh on a schedule.** Turn on `monitoringMode`, schedule daily or weekly runs, get only what changed (with which sections changed) and a Slack alert. Optional **AI summary and Q\&A pairs** per page with **your own** Anthropic or OpenAI key. ### Before / after GOV.UK "VAT rates" (Crown copyright, Open Government Licence v3.0). A page-to-Markdown tool that only strips `nav` / `footer` (RAG Web Browser's default) starts like this: ```markdown ### Cookies on GOV.UK We use some essential cookies to make this website work. We'd like to set additional cookies to understand how you use GOV.UK … You have accepted additional cookies. You can [change your cookie settings](/help/cookies) at any time. You have rejected additional cookies. … Skip to main content ## VAT rates … Is this page useful? Maybe Yes this page is useful No this page is not useful … ``` **SiteVerdict** returns only the content, with absolute links and the table intact: ```markdown ## VAT rates The standard VAT rate is _20%_ ### VAT rates for goods and services | | % of VAT | What the rate applies to | | --- | --- | --- | | Standard rate | 20% | Most goods and services | | Reduced rate | 5% | Some goods and services, for example, children's car seats and home energy | | Zero rate | 0% | Zero-rated goods and services, for example, most food and children's clothes | The standard rate of VAT increased to 20% on 4 January 2011 (from 17.5%). … ``` On a product page you also get a one-line fact summary from the page's schema.org data, e.g. `**Price:** 55 GBP · **Availability:** In Stock · **Brand:** IKEA · **SKU:** 00263850 · **Rating:** 4.6 (1200 reviews)`, and a structured `product` field. ### Benchmark (honest numbers) 30 real public pages (docs, news, blogs, e-commerce product pages, wikis, forums, government), hand-checked references, scored offline. Competitors are rebuilt from their **documented default settings** with the same open-source parts (we do not run other Actors). Full method, per-page scores, caveats and scripts: `BENCHMARK.md`. | | SiteVerdict v2 | Website Content Crawler (default) | WCC + Defuddle | RAG Web Browser (default) | |---|---|---|---|---| | Main-content F1 | **91.9%** | 84.7% | 86.9% | 80.6% | | Recall / precision | 95.4% / **90.2%** | 83.7% / 89.2% | 88.1% / 88.7% | **96.2%** / 73.1% | | Key facts kept (95) | **98.9%** | 82.0% | 87.6% | 95.5% | | Boilerplate leaks (95 checks) | 4 | **2** | 5 | 34 | | Code blocks exact (121) | **112** | 109 | 111 | 107 | | Output tokens, 30 pages | 79k | **79k** | 83k | 107k | | Obeys robots.txt by default | **always** | off by default | off by default | no option | Where we don't win: RAG Web Browser keeps slightly more text (at 73% precision); WCC's default is faster per page and leaks a little less boilerplate but misses 18% of key facts; JavaScript-heavy shop pages are hard for everyone in HTTP mode. Live compliance check: 5 out of 5 blocked pages (robots.txt, unreachable robots.txt, 403, HTTP-200 bot interstitial) returned no content and no charge. ### What you get per page `markdown` · `text` · `title`, `author`, `publishedAt`, `modifiedAt`, `language`, `siteName`, `canonicalUrl` · heading `outline` with anchors · `links` (internal / external) · `images` · `jsonLd` · `product` (PDPs) · `wordCount`, exact `tokens` + `tokenizer` · `contentHash`, `etag` · `chunks[]` (`chunkId`, `headingPath`, `anchorUrl`, `page` for PDFs, `tokens`, `citation`) · `citation` (title, URL, site, author, dates, accessed, ready-made `text`) · `changeStatus` + `changeSummary` (monitoring) · `ai` (summary, key points, Q\&A, optional). Dataset views: **Pages**, **Markdown**, **RAG chunks** (one row per chunk, CSV-ready), **Changes**, **AI summary & Q\&A**, **Products**, **Blocked / skipped**. The run's `OUTPUT` record is an agent-ready JSON summary, and monitoring runs also write a `CHANGES` record. ### Pricing (pay per event) | Event | Price | When | |---|---|---| | `page-converted` | $0.0015 | A web page converted over HTTP | | `page-converted-browser` | $0.004 | A page that needed a headless browser (`renderingMode` auto / browser) | | `document-converted` | $0.003 | A PDF or DOCX converted (any length up to 500 pages / 25 MB) | | `ai-enrichment` | $0.002 | AI summary / Q\&A written for a page with **your** API key (your provider bills its tokens to you) | **Never charged:** blocked, paywalled, opt-out, duplicate, too-short, failed, **unchanged** (monitoring) and **removed** pages; robots.txt and sitemap fetches; llms.txt. There is no start fee. Examples: **100 pages ≈ $0.15** · **1,000-page docs site, chunked ≈ $1.50** · **weekly monitoring of 1,000 pages where 50 changed ≈ $0.075 a week** · **20 PDFs ≈ $0.06** · **500 pages + AI Q\&A ≈ $1.75** plus your own LLM cost. ### Quick start **Two pages:** ```json { "startUrls": [{ "url": "https://en.wikipedia.org/wiki/Markdown" }, { "url": "https://docs.python.org/3/library/json.html" }] } ``` **A docs site into OpenAI-sized chunks:** ```json { "startUrls": [{ "url": "https://docs.example.com/" }], "mode": "sitemap", "includeGlobs": ["/guides/**"], "maxPages": 1000, "chunkPreset": "openai-text-embedding-3" } ``` **Weekly freshness check with a Slack alert** (schedule this input): ```json { "startUrls": [{ "url": "https://docs.example.com/" }], "mode": "sitemap", "maxPages": 5000, "monitoringMode": true, "monitorId": "example-docs", "webhookUrl": "https://hooks.slack.com/services/…" } ``` **PDF reports with per-page citations, plus Q\&A pairs for a RAG eval set:** ```json { "startUrls": [{ "url": "https://www.rfc-editor.org/rfc/rfc9309.pdf" }], "chunkPreset": "voyage-claude-rag", "aiEnrichment": "summary+qa", "aiProvider": "anthropic", "aiApiKey": "sk-ant-…" } ``` **Crawl a small site and draft an llms.txt:** `"mode": "crawl", "maxDepth": 2, "generateLlmsTxt": true`. `llms.txt` and `llms-full.txt` are saved to the key-value store. ### Output example (shortened) ```json { "url": "https://docs.widget.dev/cli/configuration", "status": "ok", "changeStatus": "changed", "changeSummary": { "changedSections": ["Configuration reference > Options"], "addedSections": [], "removedSections": [], "wordDelta": 14 }, "extractor": "readability+recovery", "documentType": "html", "title": "Configuration reference", "language": "en", "canonicalUrl": "https://docs.widget.dev/cli/configuration", "markdown": "# Configuration reference\n\nThe Widget CLI reads its settings from `widget.config.json` …", "tokens": 262, "tokenizer": "cl100k", "chunks": [ { "chunkId": "8fa7c16d9555369133945cd2", "headingPath": ["Configuration reference", "Options"], "anchorUrl": "https://docs.widget.dev/cli/configuration#options", "text": "Configuration reference > Options\n\n## Options\n\n| Option | Type | …", "tokens": 148, "citation": { "title": "Configuration reference", "url": "https://docs.widget.dev/cli/configuration#options", "accessedAt": "2026-10-11T07:26:22Z" } } ], "citation": { "title": "Configuration reference", "url": "https://docs.widget.dev/cli/configuration", "text": "\"Configuration reference\". Widget Docs. https://docs.widget.dev/cli/configuration. accessed 2026-10-11" }, "charged": "page-converted" } ``` A page we do not convert (no content, not charged): ```json { "url": "https://lobste.rs/s/u9ue6c/…", "status": "blocked", "reason": "robots-txt", "detail": "robots.txt disallows … (disallow: / (line 18))", "charged": false } ``` Statuses: `ok`, `unchanged`, `removed`, `blocked` (`robots-txt`, `bot-protection`, `http-403`, `login-required`, `ai-opt-out`, `http-451`, `private-address`), `paywalled`, `duplicate`, `too-short`, `unsupported-content` (`no-text-layer` for scanned PDFs, images, archives …), `error` (`not-found`, `rate-limited`, `network` …). ### Options that matter - **Rendering:** `http` (default, fastest, cheapest), `auto` (headless Chromium only for JavaScript app shells), `browser` (every page). - **Extraction:** `auto` (Readability + recovery, recommended), `readability`, `main-element`, `full-page`; `contentSelector` forces a region, `removeSelectors` strips extra blocks, `removeReferenceLists` (on) strips Wikipedia-style citation noise. - **Chunking:** `chunkPreset` (`openai-text-embedding-3` 512/64, `openai-large-context` 1024/128, `cohere-embed-v3` 400/40, `voyage-claude-rag` 800/100, `small-open-model` 320/32) or `custom` with `chunkSize`, `chunkOverlap` and `tokenizer` (`cl100k`, `o200k`, `estimate`). Chunks split at headings, keep code and tables whole (big tables repeat their header), and overlap whole sentences within a section. - **Documents:** `convertDocuments` (on) converts PDF and DOCX; `followDocumentLinks` also queues in-scope document links while crawling. - **Monitoring:** `monitoringMode`, `monitorId`, `outputUnchanged`, `useSitemapLastmod`, `reportRemoved`, `webhookUrl` (HTTPS, secret), `webhookOnNoChanges`. - **AI enrichment:** `aiEnrichment` (`summary`, `qa`, `summary+qa`), `aiProvider`, `aiApiKey` (secret), `aiModel` (default `claude-sonnet-5-5` / `gpt-4o-mini`; choose e.g. `claude-haiku-5-5` for cheaper bulk runs), `aiQaPairs`, `aiMaxInputChars`, `aiLanguage`. - **Politeness:** at least 1 s between requests to the same host, robots.txt `Crawl-delay` honoured (up to 30 s), retries only for 429 / 5xx / network errors. ### FAQ **Can it get past Cloudflare, a CAPTCHA or a login?** No, by design. Those pages come back as `blocked` and cost nothing. Use the site's API or licence, or ask the owner. **How exact are the token counts?** Exact for OpenAI models (tiktoken `cl100k_base` / `o200k_base`). Claude, Cohere and Voyage tokenizers are not published for offline use, so their presets count with cl100k plus a safety margin (x1.15 to x1.25) so chunks stay under the model's limit. **What does monitoring cost?** Only new and changed pages are charged. A 304 Not Modified, an identical content hash, or an unchanged sitemap `` costs nothing. Removed pages are reported for free. **Does AI enrichment use your key or mine?** Yours, always. It is off by default, stored as an encrypted secret input, sent only to the provider you choose, and never logged. We charge $0.002 per page only when the model returns valid output. **PDFs?** Text-layer PDFs and Word .docx files are converted (title, author and dates from the document metadata, one citation per PDF page). Scanned PDFs without a text layer are reported as `no-text-layer` and not charged (no OCR). **Why is a page "paywalled" when I can read part of it?** The publisher marks it as not free (`isAccessibleForFree: false`), or the page shows a subscription wall over a short teaser. We return nothing, not even the teaser. **Can I change the user agent?** No. Requests always identify as `SiteVerdictBot`. You can add your own robots.txt token, but it can only add restrictions; other crawlers' names (Googlebot, GPTBot …) are refused. **When should I use Website Content Crawler instead?** WCC is mature and very capable: proxies, several crawler engines, cookie-dialog clicking, file downloads, huge crawls. If you need to get through hard-to-crawl sites or run proxied crawls of millions of pages, use WCC. Use this Actor when you want measured extraction quality, RAG-ready chunks with exact token counts, cheap incremental monitoring, PDF/DOCX conversion and compliance you can document. ### Limits - Max 20,000 pages per run; 8 MB HTML / 25 MB documents per URL; PDFs up to 500 pages; 30 s timeout per request by default. - No proxies, no logins, no consent-dialog clicking, no OCR, no screenshots. - Extraction is heuristic: unusual layouts (especially JavaScript-heavy shop pages) may need `renderingMode: "auto"` or a `contentSelector`. - PDF text order follows the PDF's text layer; complex multi-column layouts and tables in PDFs come out as paragraphs. - Paywall detection is conservative towards publishers: occasionally a free page with a "subscribe to continue reading" box is marked `paywalled`. - Monitoring memory lives in a named key-value store in **your** Apify account (`siteverdict-w2md-monitor-`); delete it to start fresh. - Language is what the page declares; it is not detected from the text. ### Compliance, sources and responsible use - Only **public** pages are fetched, with an honest user agent: `Mozilla/5.0 (compatible; SiteVerdictBot/2.0.0; SiteVerdict/web-to-markdown; +aman.thakran@bharatfare.com)`. Site owners can control access with `User-agent: SiteVerdictBot` in robots.txt. - robots.txt follows RFC 9309 (5xx or unreachable means "disallow all"; 4xx means "allow all"), and is checked on every redirect hop and browser navigation. - Paywalls (schema.org `isAccessibleForFree`, `content_tier`, paywall markup), login walls, CAPTCHAs / bot challenges (including HTTP-200 "waiting room" interstitials) and AI / text-and-data-mining opt-outs (`noai`, `tdm-reservation: 1`) are respected: no content, no charge, never bypassed. After 3 blocks a host is skipped. - **Copyright stays with the publisher.** Converting a page does not give you a licence to republish it or train models on it. Check each site's terms (e.g. Wikipedia text is CC BY-SA and needs attribution; GOV.UK content is OGL v3.0). Use the `citation` field to attribute. - **Personal data:** the output contains only what the page publishes (e.g. an author byline). E-mail addresses and phone numbers are stripped from author fields. If you process pages about individuals you are the controller under UK GDPR / EU GDPR: have a lawful basis, keep only what you need, and do not use the output for unsolicited marketing (PECR). - AI enrichment sends page text to the provider **you** choose under **your** account and its terms. - Requests to private or internal network addresses (including webhook URLs) are refused. - **Where results are stored:** the run's default dataset and key-value store in your own Apify account. These are unnamed; how long Apify keeps them is set by Apify and your plan, not by SiteVerdict ([Apify data retention](https://docs.apify.com/platform/storage/usage#data-retention)). You can delete them sooner in Apify Console. - **Monitoring (off by default):** to skip unchanged pages, a **named** key-value store `siteverdict-w2md-monitor-` keeps, per page, a content hash, section hashes, the title, word count, ETag/Last-Modified and dates (not the page text). Apify keeps named stores until you delete them. - **What SiteVerdict keeps:** nothing. The Actor runs in your Apify account and sends no data to SiteVerdict; our contact address appears only in the User-Agent sent to the sites it reads. - This is not legal advice. When in doubt, ask the site owner. Built by SiteVerdict (BharatFare Travel and Technologies Ltd, UK). Open-source components: [Mozilla Readability](https://github.com/mozilla/readability) (Apache-2.0), [turndown](https://github.com/mixmark-io/turndown) (MIT), [linkedom](https://github.com/WebReflection/linkedom) (ISC), [unpdf](https://github.com/unjs/unpdf) / pdf.js (MIT / Apache-2.0), [mammoth](https://github.com/mwilliamson/mammoth.js) (BSD-2-Clause), [js-tiktoken](https://github.com/dqbd/tiktoken) (MIT), [Anthropic TypeScript SDK](https://github.com/anthropics/anthropic-sdk-typescript) (MIT). # Changelog This Actor's version history is a separate document: https://apify.com/siteverdict/siteverdict-web-to-markdown/changelog.md # Actor input Schema ## `startUrls` (type: `array`): Pages to convert (mode 'urls'), sites or sitemap.xml files (mode 'sitemap'), or crawl seeds (mode 'crawl'). Bare domains like example.com are fine. ## `mode` (type: `string`): 'urls' converts exactly the URLs you list. 'sitemap' reads each site's sitemap (from robots.txt or /sitemap.xml; or give a sitemap URL directly). 'crawl' follows links within the same site up to Max depth. ## `maxPages` (type: `integer`): Hard cap on pages fetched in this run (all statuses). You are only charged for pages successfully converted. ## `maxDepth` (type: `integer`): Crawl mode only. 0 = start URLs only, 1 = plus pages they link to, and so on. ## `includeGlobs` (type: `array`): Optional. Globs starting with '/' match the path (e.g. /docs/\*\*); others match the full URL (e.g. https://example.com/blog/\*). Start URLs are always processed. ## `excludeGlobs` (type: `array`): Optional. e.g. /tag/\*\*, /search\*, \*\*/*?page=* ## `allowSubdomains` (type: `boolean`): If on, crawling example.com also follows docs.example.com. Off = same host only (www. is ignored). ## `renderingMode` (type: `string`): 'http' is fast and cheap (no JavaScript). 'auto' uses a headless browser only when a page looks like a JavaScript app shell (charged as page-converted-browser). 'browser' renders every page. Bot challenges are never 'solved' in any mode. ## `extractionMode` (type: `string`): 'auto' (v2 engine) runs Mozilla Readability, then maps its picks back onto the page to recover paragraphs it dropped (after images, forum post headers) and trims related-content rails. Highest measured F1 in BENCHMARK.md. 'full-page' keeps everything except navigation, header, footer and sidebars. ## `contentSelector` (type: `string`): If set and found on the page, only these elements are converted, e.g. 'article .post-body' or '#docs-content'. ## `removeSelectors` (type: `array`): Extra elements to strip before extraction, e.g. '.newsletter-signup', '#related'. Scripts, cookie banners, dialogs and hidden elements are always removed. ## `removeReferenceLists` (type: `boolean`): Removes MediaWiki-style footnote markers, reference lists, navboxes, hatnotes and maintenance banners. Saves tokens on wiki pages; turn off if you need the bibliography. ## `includeLinksInMarkdown` (type: `boolean`): Off = link text only (fewer tokens). Links are always listed separately in 'links'. ## `includeImagesInMarkdown` (type: `boolean`): Off = no ![alt](src) lines. Decorative images (empty alt), tracking pixels and repeated gallery images are always left out of the Markdown; every image is still listed in 'images'. ## `includePlainText` (type: `boolean`): Adds a 'text' field (Markdown syntax stripped). ## `includeJsonLd` (type: `boolean`): Adds parsed schema.org JSON-LD blocks (@graph flattened). ## `convertDocuments` (type: `boolean`): PDF (text layer, per-page citations) and Word .docx URLs are converted to Markdown and charged as 'document-converted'. Scanned PDFs without text are reported, not charged (no OCR). Max 25 MB / 500 pages. ## `followDocumentLinks` (type: `boolean`): Crawl and sitemap modes: also queue in-scope .pdf / .docx links (they count toward Max pages). ## `minWords` (type: `integer`): Pages with fewer extracted words get status 'too-short' and are not charged (content is still included). ## `dedupe` (type: `string`): Skip pages whose canonical URL or normalised text hash was already converted in this run (status 'duplicate', not charged). ## `enableChunking` (type: `boolean`): Heading-aware chunks with source URL, heading path and deep-link anchor. Code blocks and tables are kept whole when they fit; oversized tables repeat their header row. ## `chunkPreset` (type: `string`): Sets chunk size, overlap and token counter for your embedding model. OpenAI presets count tokens EXACTLY (tiktoken cl100k_base). Cohere / Claude-RAG / 512-token open models use cl100k with a safety margin. 'custom' uses the fields below. ## `chunkSize` (type: `integer`): Target maximum tokens per chunk (custom preset), counted with the token counter below. ## `chunkOverlap` (type: `integer`): Tokens repeated from the end of the previous chunk (whole sentences) when a section is split. Not applied across headings. Max half the chunk size. ## `tokenizer` (type: `string`): cl100k = exact OpenAI text-embedding-3 / GPT-4 counts; o200k = GPT-4o family; estimate = fast heuristic. ## `prependHeadingPath` (type: `boolean`): Adds a breadcrumb line like 'Guide > Install > Linux' to each chunk's text. Improves retrieval; the path is also in 'headingPath'. ## `generateLlmsTxt` (type: `boolean`): Builds an llms.txt (llmstxt.org format) and llms-full.txt from the converted pages and saves them to the key-value store. Best with mode 'sitemap' or 'crawl'. ## `monitoringMode` (type: `boolean`): Remembers every page between runs (named store per Monitor ID). Next runs send If-None-Match / If-Modified-Since, skip sitemap URLs whose lastmod did not move, and output only new and changed pages with a section-level change summary. Unchanged pages are NEVER charged. Schedule the Actor to keep a RAG index fresh. ## `monitorId` (type: `string`): Name of this monitor's memory (e.g. 'docs-site'). Use a different ID per site or URL list. ## `outputUnchanged` (type: `boolean`): Adds a free row with changeStatus 'unchanged' (no content). ## `useSitemapLastmod` (type: `boolean`): Sitemap mode: a known URL whose is identical to last run is not downloaded at all. ## `reportRemoved` (type: `boolean`): URLs and sitemap mode: pages known from earlier runs that are gone get a free 'removed' row (only when this run covered the full list). ## `webhookUrl` (type: `string`): Optional. POSTs a JSON summary with a Slack-compatible 'text' field when pages are new / changed / removed. HTTPS only, public hosts only. ## `webhookOnNoChanges` (type: `boolean`): Heartbeat mode. ## `aiEnrichment` (type: `string`): Off by default. Uses YOUR Anthropic or OpenAI key to add a summary, key points and grounded question/answer pairs to each converted page (field 'ai'). Charged as 'ai-enrichment' only when the model returns valid output; your provider bills its tokens to you. ## `aiProvider` (type: `string`): Which API your key belongs to. ## `aiApiKey` (type: `string`): Stored encrypted by Apify as a secret input. Sent only to the chosen provider; never logged or written to the dataset. ## `aiModel` (type: `string`): Default: claude-sonnet-5-5 (Anthropic) or gpt-4o-mini (OpenAI). For cheaper bulk runs choose e.g. claude-haiku-5-5. ## `aiQaPairs` (type: `integer`): Upper limit; short pages may yield fewer. ## `aiMaxInputChars` (type: `integer`): Longer pages are truncated and flagged ai.inputTruncated. ## `aiLanguage` (type: `string`): Default: the page's language. ## `robotsUserAgent` (type: `string`): We always obey robots.txt rules for 'SiteVerdictBot' (and '\*'). You may add your own bot name: its rules then apply too, so it can only add restrictions. Other crawlers' names (Googlebot, GPTBot, ...) are refused. ## `perHostDelayMs` (type: `integer`): Minimum spacing per host. A longer robots.txt Crawl-delay (up to 30 s) is honoured automatically. ## `maxConcurrency` (type: `integer`): Pages processed in parallel across all sites (browser mode is capped at 8). ## `requestTimeoutSecs` (type: `integer`): Per request / page render. ## `maxRetries` (type: `integer`): Retries for network errors, 429 and 5xx (with backoff and Retry-After). 401/403 are never retried. ## `acceptLanguage` (type: `string`): e.g. 'en-GB,en;q=0.9' to request a language version where sites support it. ## `outputSkippedPages` (type: `boolean`): Adds a row with status and reason (robots-txt, bot-protection, paywalled, login-required, ai-opt-out, duplicate, error...) and no content. Never charged. ## Actor input object example ```json { "startUrls": [ { "url": "https://en.wikipedia.org/wiki/Markdown" } ], "mode": "urls", "maxPages": 50, "maxDepth": 2, "includeGlobs": [], "excludeGlobs": [], "allowSubdomains": false, "renderingMode": "http", "extractionMode": "auto", "removeSelectors": [], "removeReferenceLists": true, "includeLinksInMarkdown": true, "includeImagesInMarkdown": true, "includePlainText": true, "includeJsonLd": true, "convertDocuments": true, "followDocumentLinks": false, "minWords": 20, "dedupe": "canonical-and-hash", "enableChunking": true, "chunkPreset": "custom", "chunkSize": 512, "chunkOverlap": 64, "tokenizer": "cl100k", "prependHeadingPath": true, "generateLlmsTxt": false, "monitoringMode": false, "monitorId": "default", "outputUnchanged": false, "useSitemapLastmod": true, "reportRemoved": true, "webhookOnNoChanges": false, "aiEnrichment": "none", "aiProvider": "anthropic", "aiQaPairs": 5, "aiMaxInputChars": 60000, "robotsUserAgent": "SiteVerdictBot", "perHostDelayMs": 1000, "maxConcurrency": 5, "requestTimeoutSecs": 30, "maxRetries": 2, "outputSkippedPages": true } ``` # Actor output Schema ## `results` (type: `string`): No description ## `output` (type: `string`): No description ## `files` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ { "url": "https://en.wikipedia.org/wiki/Markdown" } ] }; // Run the Actor and wait for it to finish const run = await client.actor("siteverdict/siteverdict-web-to-markdown").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": [{ "url": "https://en.wikipedia.org/wiki/Markdown" }] } # Run the Actor and wait for it to finish run = client.actor("siteverdict/siteverdict-web-to-markdown").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ { "url": "https://en.wikipedia.org/wiki/Markdown" } ] }' | apify call siteverdict/siteverdict-web-to-markdown --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,siteverdict/siteverdict-web-to-markdown" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/RVqUjgt1Piae9dkQ0/builds/W1dOBJTn1zfeYx1Ed/openapi.json