# Changelog of Website & Page to Markdown for RAG and LLMs (`humble-echidna/page-to-markdown`) Actor - **URL**: https://apify.com/humble-echidna/page-to-markdown/changelog.md - **Full Actor documentation**: https://apify.com/humble-echidna/page-to-markdown.md ## Changelog Versions follow MAJOR.MINOR.PATCH (`src/version.py`); Apify shows MAJOR.MINOR from `.actor/actor.json`. Every run logs its version and records it in the `RUN_STATS` key-value record. > **About future changes in output:** main-content detection is heuristic. When a site redesigns, what counts > as its main content can shift. The run log and `RUN_STATS` name every URL that failed and why; one broken > site never affects the other URLs in a run. ### 1.1.2 (2026-09-25) - Input field descriptions rewritten for AI agents (Apify's MCP server shows agents only the description, not the form): each now states its default, its allowed range, the value of each option and how it combines with other fields. No change to field names, types, defaults or behaviour. ### 1.1.1 (2026-09-25) - Whole-site mode: a start URL that redirects to another section of the same site (`/platform/storage` -> `/storage`) now reads the sitemap for the section where it lives, instead of falling back to following links, which could miss pages that no link reaches. `RUN_STATS` records the new section as `scopeMovedTo`. ### 1.1.0 (2026-09-24) - **Whole-site mode** (`crawlWholeSite`): each URL is a site, a section of one (only pages under its path) or a sitemap URL. Pages are found sitemap first, through the shared `mms_sitemap` package (robots.txt `Sitemap:` lines, else /sitemap.xml and /sitemap\_index.xml, indexes followed); a site without a usable sitemap is crawled by following its same-site links, with no depth limit (a start URL that redirects is scoped to where it lands). `maxPagesPerSite` caps the pages returned per site (default 100, up to 10,000), and at most 5 pages are fetched per page returned (plus 100). - **URL patterns** (`includeUrlPatterns`, `excludeUrlPatterns`) for pages found by crawling, in both crawl modes; excluded pages are never fetched. The URLs you list are always converted. - **Only changed pages** (`onlyChangedPages`): a sitemap page whose `lastmod` is unchanged since this search's last run is skipped without being fetched; any other page is fetched and skipped, not charged, when its Markdown hashes the same. The memory lives in the `page-to-markdown-changes` key-value store in the user's account, one record per input URL and set of options, and only covers pages that were actually delivered. - **Chunking for RAG** (`chunkMarkdown`, `maxChunkChars`, `chunkOutput`): each page's Markdown split by headings, with oversized sections split at paragraph, then line breaks; each chunk has its heading path, URL, title and index. Either a `chunks` array on the page row, or one row per chunk in the run's second dataset, `chunks` (the default dataset keeps one row per page). - **Charging is unchanged: per page.** Chunks never add charges; chunk rows go to a separate dataset, which the per-item event doesn't bill. Unchanged pages skipped by "only changed pages" aren't charged. - New output fields: `discoveredVia` (input, link or sitemap), `lastmod`, `contentHash`, `chunkCount`, `chunks`; `depth` is null for pages found in a sitemap. - AI-crawler opt-outs and robots.txt apply to sitemap downloads too (they go through the same page client). ### 1.0.1 (2026-09-24) - Internal: extraction, Markdown conversion and page fetching now live in the shared `mms_extract` package, shared with the new article-extractor actor. No change in behaviour or output. ### 1.0.0 (2026-09-24) First release. - Converts the web pages the user lists into Markdown: main content only (readability-style extraction; navigation, headers, footers, cookie banners, sidebars and link farms removed), with headings, lists, GitHub tables, fenced code blocks with their language, blockquotes and absolute links. Images optional (off by default). - Per page: title, meta description, language, canonical URL, links in the content, word count, depth, stable id. - Optional same-site crawl: depth 0-2, max pages per URL (default 10); other sites and file links are never followed; `www.` is the same site; each page is converted and charged once per run. - Honest identity: User-Agent `HumbleEchidnaApify/1.0 (+https://apify.com/humble-echidna)`, the same as every actor in this portfolio. - robots.txt checked before every request (redirect hops and crawled pages included), read once per origin per run and capped at 500 KiB while streaming (decompressed bytes included, so a small gzip file can't expand in memory), matched on the product token `HumbleEchidnaApify` (never the full User-Agent string), with a byte-order mark handled and paths percent-encoding-normalised (RFC 9309); unreachable robots.txt (5xx/429/network) = disallowed, per RFC 9309. Opt-outs from AI use honoured too: a page that robots.txt disallows for any well-known AI crawler (GPTBot, CCBot, ClaudeBot, Google-Extended, ...) is skipped and reported as "site opts out of AI use" (`optedOutOfAI` in `RUN_STATS`). `Crawl-delay` honoured, retries included (a site asking for more than 30 s is reported instead of fetched), `Retry-After` honoured (seconds or a date; more than 30 s is reported), at most 2 concurrent requests per host, and a total time limit per download. - No headless browser in this version: JavaScript-only pages are detected and reported as "needs JavaScript", never returned as empty results. Non-HTML responses (PDF, images, ...), pages over 5 MB, 401/403/451 refusals, 404s and private-network addresses are reported per URL and not charged. - Public addresses only: every hostname (start URL, crawled page, redirect hop, robots.txt) is resolved and refused if any address it resolves to isn't globally routable (loopback, private, link-local incl. cloud metadata, CGNAT, multicast, reserved, and the IPv6 forms that carry those: IPv4-mapped, IPv4-compatible, NAT64; 6to4, Teredo and site-local are refused outright). The connection then goes to the address that was checked, so DNS rebinding can't slip an internal address in between. Only the standard web ports (80 and 443) are fetched. These refusals are reported as "not a public web address", never as robots.txt blocks; a domain that doesn't resolve is reported as such. - A start URL that finds every result slot taken by pages still in flight waits for them instead of being skipped (a failing page gives its slot back). - The safety code (robots.txt, the private-network guard, pacing, capped downloads) is the shared `mms_common` implementation every actor in this portfolio uses. - Markdown capped at 500,000 characters per page (`markdownTruncated`). - Charged per page converted, through Apify's standard `apify-default-dataset-item` event. - "Max pages per run" input, and the maximum cost per run is honoured: each page is counted against the limit before it's fetched (given back if it fails), so no page is fetched that the run can't return; once the limit is covered, crawls stop, URLs still being fetched are cancelled and the rest are skipped (`queriesSkipped` in `RUN_STATS`). - Failure isolation: a failing URL, or a failing page found while crawling, only affects itself. - A malformed address (`http://[bad`, bad IDNA), as a start URL or a redirect target, is reported for that URL; a malformed link on a page is skipped and the page is still converted. - A page dropped as invalid (e.g. no Markdown) is never charged and gives its result slot back. - `RUN_STATS` has the same keys on every exit path (`dropped_invalid`, `failure` and `error` included). - One overall time limit per page (redirects, retries, waits and robots.txt included), on top of the per-download one; robots.txt rules are matched in linear time, so a hostile file can't stall the run.