Versions follow MAJOR.MINOR.PATCH (src/version.py); Apify shows MAJOR.MINOR from .actor/actor.json.
Every run logs its version and records it in the RUN_STATS key-value record.
About future changes in output: main-content detection is heuristic. When a site redesigns, what counts
as its main content can shift. The run log and RUN_STATS name every URL that failed and why; one broken
site never affects the other URLs in a run.
- Input field descriptions rewritten for AI agents (Apify's MCP server shows agents only the description, not the
form): each now states its default, its allowed range, the value of each option and how it combines with other
fields. No change to field names, types, defaults or behaviour.
- Whole-site mode: a start URL that redirects to another section of the same site (
/platform/storage ->
/storage) now reads the sitemap for the section where it lives, instead of falling back to following links,
which could miss pages that no link reaches. RUN_STATS records the new section as scopeMovedTo.
- Whole-site mode (
crawlWholeSite): each URL is a site, a section of one (only pages under its path) or a
sitemap URL. Pages are found sitemap first, through the shared mms_sitemap package (robots.txt Sitemap: lines,
else /sitemap.xml and /sitemap_index.xml, indexes followed); a site without a usable sitemap is crawled by
following its same-site links, with no depth limit (a start URL that redirects is scoped to where it lands). maxPagesPerSite caps the pages returned per site (default
100, up to 10,000), and at most 5 pages are fetched per page returned (plus 100).
- URL patterns (
includeUrlPatterns, excludeUrlPatterns) for pages found by crawling, in both crawl modes;
excluded pages are never fetched. The URLs you list are always converted.
- Only changed pages (
onlyChangedPages): a sitemap page whose lastmod is unchanged since this search's last
run is skipped without being fetched; any other page is fetched and skipped, not charged, when its Markdown hashes
the same. The memory lives in the page-to-markdown-changes key-value store in the user's account, one record per
input URL and set of options, and only covers pages that were actually delivered.
- Chunking for RAG (
chunkMarkdown, maxChunkChars, chunkOutput): each page's Markdown split by headings,
with oversized sections split at paragraph, then line breaks; each chunk has its heading path, URL, title and
index. Either a chunks array on the page row, or one row per chunk in the run's second dataset, chunks (the
default dataset keeps one row per page).
- Charging is unchanged: per page. Chunks never add charges; chunk rows go to a separate dataset, which the
per-item event doesn't bill. Unchanged pages skipped by "only changed pages" aren't charged.
- New output fields:
discoveredVia (input, link or sitemap), lastmod, contentHash, chunkCount, chunks;
depth is null for pages found in a sitemap.
- AI-crawler opt-outs and robots.txt apply to sitemap downloads too (they go through the same page client).
- Internal: extraction, Markdown conversion and page fetching now live in the shared
mms_extract package, shared with the new article-extractor actor. No change in behaviour or output.
First release.
- Converts the web pages the user lists into Markdown: main content only (readability-style extraction; navigation,
headers, footers, cookie banners, sidebars and link farms removed), with headings, lists, GitHub tables, fenced
code blocks with their language, blockquotes and absolute links. Images optional (off by default).
- Per page: title, meta description, language, canonical URL, links in the content, word count, depth, stable id.
- Optional same-site crawl: depth 0-2, max pages per URL (default 10); other sites and file links are never
followed;
www. is the same site; each page is converted and charged once per run.
- Honest identity: User-Agent
HumbleEchidnaApify/1.0 (+https://apify.com/humble-echidna), the same as every
actor in this portfolio.
- robots.txt checked before every request (redirect hops and crawled pages included), read once per origin per run
and capped at 500 KiB while streaming (decompressed bytes included, so a small gzip file can't expand in memory),
matched on the product token
HumbleEchidnaApify (never the full User-Agent string), with a byte-order mark
handled and paths percent-encoding-normalised (RFC 9309); unreachable robots.txt (5xx/429/network) = disallowed,
per RFC 9309. Opt-outs from AI use honoured too: a page that robots.txt disallows for any well-known AI crawler
(GPTBot, CCBot, ClaudeBot, Google-Extended, ...) is skipped and reported as "site opts out of AI use"
(optedOutOfAI in RUN_STATS). Crawl-delay honoured, retries included (a site asking for more than 30 s is
reported instead of fetched), Retry-After honoured (seconds or a date; more than 30 s is reported), at most 2
concurrent requests per host, and a total time limit per download.
- No headless browser in this version: JavaScript-only pages are detected and reported as "needs JavaScript",
never returned as empty results. Non-HTML responses (PDF, images, ...), pages over 5 MB, 401/403/451 refusals,
404s and private-network addresses are reported per URL and not charged.
- Public addresses only: every hostname (start URL, crawled page, redirect hop, robots.txt) is resolved and refused
if any address it resolves to isn't globally routable (loopback, private, link-local incl. cloud metadata,
CGNAT, multicast, reserved, and the IPv6 forms that carry those: IPv4-mapped, IPv4-compatible, NAT64; 6to4,
Teredo and site-local are refused outright).
The connection then goes to the address that was checked, so DNS rebinding can't slip an internal address in
between. Only the standard web ports (80 and 443) are fetched. These refusals are reported as "not a public web
address", never as robots.txt blocks; a domain that doesn't resolve is reported as such.
- A start URL that finds every result slot taken by pages still in flight waits for them instead of being skipped
(a failing page gives its slot back).
- The safety code (robots.txt, the private-network guard, pacing, capped downloads) is the shared
mms_common
implementation every actor in this portfolio uses.
- Markdown capped at 500,000 characters per page (
markdownTruncated).
- Charged per page converted, through Apify's standard
apify-default-dataset-item event.
- "Max pages per run" input, and the maximum cost per run is honoured: each page is counted against the limit
before it's fetched (given back if it fails), so no page is fetched that the run can't return; once the limit is
covered, crawls stop, URLs still being fetched are cancelled and the rest are skipped (
queriesSkipped in RUN_STATS).
- Failure isolation: a failing URL, or a failing page found while crawling, only affects itself.
- A malformed address (
http://[bad, bad IDNA), as a start URL or a redirect target, is reported for that URL; a
malformed link on a page is skipped and the page is still converted.
- A page dropped as invalid (e.g. no Markdown) is never charged and gives its result slot back.
RUN_STATS has the same keys on every exit path (dropped_invalid, failure and error included).- One overall time limit per page (redirects, retries, waits and robots.txt included), on top of the per-download
one; robots.txt rules are matched in linear time, so a hostile file can't stall the run.
