# Changelog of Website to Markdown Crawler (`entranced_gelato/website-to-markdown-crawler`) Actor

- **URL**: https://apify.com/entranced_gelato/website-to-markdown-crawler/changelog.md
- **Full Actor documentation**: https://apify.com/entranced_gelato/website-to-markdown-crawler.md

## Changelog

### 2026-11-01

#### Added

- **Several start URLs in one run** (`startUrls`, up to 20 together with `startUrl`). Each site is crawled in its own section, **Max pages** and the maximum total charge apply to the whole run, and a page reached from two start URLs is returned and charged once. With several start URLs, `RUN_SUMMARY` lists the stats per site.

### 2026-10-17: Compliance update

#### Added

- **TDM reservation flags** on every page item: `tdmReserved` (`true`/`false`) and `tdmPolicy` (URL or `null`), from the TDMRep `tdm-reservation` / `tdm-policy` response headers and `<meta>` tags; a valid `<meta>` value beats the header. `/.well-known/tdmrep.json` is not fetched, so request counts don't change. The overview and Markdown dataset views show the flag.
- `LICENSE` with an all-rights-reserved notice.

#### Changed

- **Honest identification:** every request (pages, robots.txt, sitemaps) sends `Mozilla/5.0 (compatible; AIDevsBot/1.0; +https://apify.com/entranced_gelato/website-to-markdown-crawler)` instead of a Chrome 126 browser User-Agent. Sites that only admit browsers may now block the crawl; it stops and says so.
- **robots.txt:**
  - The `AIDevsBot` group applies when robots.txt has one (case-insensitive), otherwise `User-agent: *`; longest match wins.
  - HTTP 401/403 for robots.txt now counts as disallow-all (before: allow-all). Other 4xx still mean no restrictions. 429, 5xx, network errors and timeouts disallow the site for the run.
  - A page that redirects to a disallowed URL is dropped (not saved, not charged). With `respectRobotsTxt` on, disallowed sitemap files are no longer fetched.
- **Entry point:** the `CRAWLER_TEST` environment variable no longer exists. Before, any value made the Actor exit without doing anything. `src/main.js` now runs `main()` only when started as the program, never on import.
- `package.json`: `"license": "UNLICENSED"`, `"private": true` (was ISC).
- README:
  - New FAQ answers: bot identity and opt-out, robots.txt status handling, `tdmReserved`, copyright, wrong charges (Issues tab, 14 days), data protection.
  - "Is it legal?" rewritten.
  - Not-affiliated line for the integrations named.
- `respectRobotsTxt` description names the `AIDevsBot` rules.

### 2026-10-11

#### Changed

- **Apify's standard proxies only.** Residential proxy groups and the `GOOGLE_SERP` group are dropped from the proxy settings and custom proxy URLs are ignored. If nothing usable is left, the run uses Apify Proxy in automatic mode (datacenter proxies). When settings are dropped the run logs a warning and sets a status message; it never fails because of it. "No proxy" still means no proxy.

### 2026-10-10

#### Added

- **Sitemap discovery** (`useSitemap`, on by default): reads robots.txt `Sitemap:` lines and `/sitemap.xml`, follows sitemap indexes and `.xml.gz` files, and queues only URLs inside the start URL's section.
- **Include/exclude URL patterns** (`includeUrlPatterns`, `excludeUrlPatterns`) with glob syntax (`*`, `**`, `{a,b}`), applied to links and sitemap URLs.
- **robots.txt support** (`respectRobotsTxt`, on by default, RFC 9309): disallowed pages are skipped; if robots.txt is unreachable the site is treated as disallowed.
- `source` field (`start`, `link`, `sitemap`) on each page, `errorCode` on error rows, and a `RUN_SUMMARY` key-value record (totals and stop reason).
- `.actor/store-listing.json` (with a "Docs site to Markdown" task), dataset views, output schema links, CI workflow and offline tests (unit + local end-to-end).

#### Changed

- "Page crawled" ($0.001, as low as $0.0007) is charged through `pushData` after each page is saved. The crawl stops when the run's maximum total charge is reached and pushes nothing more.
- No charge for duplicate pages (redirect targets or identical content), empty pages (`EMPTY_CONTENT`), login pages, errors or non-HTML files.
- CAPTCHA and anti-bot pages, a refused start page or three refusals in a row stop the crawl with a clear status message; nothing is bypassed. HTTP 401/403 are no longer retried with a new proxy session. Downloads are capped at 30 MB.
- The same-domain check now compares origins exactly (it used to accept look-alike hosts such as `example.com.evil.example.net`) and also accepts the domain the start URL redirects to. Relative links resolve against the final URL after redirects.
- README rewritten around the AIDevs LLM data toolkit; `maxPages` prefill lowered to 10 for a cheaper demo run (default stays 25).