Turn public web pages, sitemaps or whole sites into clean Markdown, text and RAG chunks with title, author, dates, canonical URL, outline, links, tables and JSON-LD. Respects robots.txt, paywalls and AI opt-outs. Optional llms.txt.
The existing host check (name filter + DNS lookup) is replaced by the shared SiteVerdict guard: internal names (localhost, *.local, *.internal, single-label names, cloud-metadata names), and private / loopback / link-local / CGNAT / multicast / reserved / metadata addresses in every IPv4/IPv6 encoding (decimal/octal/hex, IPv4-mapped, NAT64, 6to4, Teredo, unicode dots) are refused for start URLs, every redirect hop, robots.txt and the webhook.
HTTP fetching now uses a DNS-pinned transport: the address that was checked is the address that is connected to (no rebinding / TTL-0 window). Refused pages keep the existing wording ("Host resolves to a private or reserved address; refused.") and are not charged.
Browser rendering: Chromium sends every request (navigations, redirects, meta refresh, JS navigations, frames, scripts, XHR) through a local checking proxy; blocked requests are counted in SUMMARY.privateNetworkBlocked.
No change to inputs, outputs (one additive SUMMARY field) or pricing.
Memory default (15 Oct 2026)
15 Oct 2026 (build after this entry): default run memory is now 1024 MB via defaultMemoryMbytes (was 4096 MB (no key set); peak 138-190 MB). Measured on the example input: same output, see the public-15 remediation log. You can still choose more memory per run. No change to inputs, outputs or pricing.
Extraction v2: Readability-guided recovery (gap filling, adjacent-prose extension, stop at related / popular / newsletter rails, junk trimming), widget pre-clean (share bars, newsletter and back-in-stock sign-ups, ad slots, cookie notices), MediaWiki citation-noise removal (removeReferenceLists, default on), schema.org product facts line + product field, tracking parameters stripped from links, decorative / duplicate images dropped from Markdown, HTTP-200 waiting-room / bot-failover interstitials detected as bot-protection.
PDF and DOCX conversion (unpdf / pdf.js, mammoth): document metadata, page markers, per-page chunks with page and #page=N citation URLs; scanned PDFs reported as no-text-layer (not charged). followDocumentLinks for crawl / sitemap modes. New event document-converted.
Embedding presets and exact token counts: js-tiktoken cl100k / o200k; presets for OpenAI, Cohere v3, Claude/Voyage and 512-token open models; tokens + tokenizer on pages and chunks.
Monitoring / incremental re-crawl: named state store per monitorId, conditional GET (ETag / Last-Modified, 304 = unchanged), sitemap <lastmod> delta, content-hash change detection with a section-level changeSummary, removed pages, CHANGES record, HTTPS webhook alerts (Slack-compatible). Unchanged / removed rows are never charged.
AI enrichment (bring your own key): summary, key points and grounded Q&A pairs via Anthropic (official SDK, structured output, server-side refusal fallback) or OpenAI; secret input, off by default; new event ai-enrichment charged only on valid output.
Citations: every chunk now carries a citation object (anchor or page URL); the page citation gains modifiedAt and a ready-made text.
New dataset views: Changes, AI summary & Q&A, Products; the chunk view is CSV / vector-DB ready.
Benchmark harness in benchmark/ (not shipped in the image): snapshot fetcher, references, scorer, compliance check.
1.0.0 — 2026-10-11
First release.
Modes: list of URLs, sitemap (robots.txt Sitemap: / /sitemap.xml, sitemap indexes, gzip, text sitemaps), same-site crawl with depth, include/exclude globs and max pages.
Fast HTTP mode with Mozilla Readability + semantic <main>/<article> fallback; optional headless Chromium (auto / browser).
Markdown (GFM tables, fenced code with language, images with alt), plain text, title/author/dates/language/canonical/site name, heading outline with anchors, internal/external links, images, JSON-LD, word count, token estimate, SHA-256 content hash, citation object.
Heading-aware RAG chunking (token size, sentence overlap within sections, tables split by rows with header repeated, code fences kept balanced, heading path + anchor per chunk).
De-duplication by canonical URL and content hash.
llms.txt + llms-full.txt generation.
Compliance: robots.txt (RFC 9309) on every redirect hop and browser navigation, Crawl-delay, honest SiteVerdictBot UA, paywall / login / bot-protection / noai / TDM-reservation detection with no content returned, SSRF guard, per-host politeness, host skipped after 3 bot-protection blocks.
Pay-per-event: page-converted, page-converted-browser; max total charge respected; never charged for blocked, paywalled, duplicate, too-short or failed pages.
