# Changelog of Website Metadata Extractor: Meta Tags, Open Graph, JSON-LD (`fecklesspleb/website-metadata-extractor`) Actor

- **URL**: https://apify.com/fecklesspleb/website-metadata-extractor/changelog.md
- **Full Actor documentation**: https://apify.com/fecklesspleb/website-metadata-extractor.md

## Changelog

Version stays "1.0" in .actor/actor.json: Apify accepts only MAJOR.MINOR there and assigns the patch (build) number at the next build, which is the patch bump for this release.

### Unreleased

- The sample page is now `https://www.python.org/` (was `https://example.com`): `SAMPLE_URL` in `my_actor/main.py`, the `urls` input schema `default` and `prefill`, and a new actor.json `exampleRunInput` all hold this one value, so `{}`, the prefill and the example input are all sample runs (15 s deadline, free warning row if the page cannot be read). A sample run reads 1 page ($0.001 + start fee). Only the exact URL list `["https://www.python.org/"]` is a sample run; other spellings (for example without the trailing slash) are normal runs. README Default column updated. New test `tests/test_sample_1102v.py`.

### 2026-11-02 (actor.json version unchanged: 1.0): private-address (SSRF) guard

- On the Apify platform, every request (the URL itself, each redirect hop, and robots.txt) is refused when the host
  resolves to a private, loopback, link-local, carrier-grade NAT, reserved, multicast or unspecified address
  (including IPv4-mapped IPv6), or when any of the host's addresses is one. Same rules as Bulk HTTP Request.
- Such a URL gets a free error row (it is dropped like other error rows when `includeFailedUrls` is off):
  `error: "not fetched: private, loopback and link-local addresses are not allowed"`. It counts as `failed` in
  `RUN_SUMMARY`. When `includeFailedUrls` is off and no row is output at all, the dataset gets one free
  status row instead: `recordType: "status"`, `message: "Not charged: N URL(s) were not fetched (...)"`.
- A host the guard cannot resolve fails as a DNS error, as before (`error` now reads
  "DNS or connection error: DNS lookup failed for <host>"); the row stays free.
- A robots.txt that redirects to a private address counts as unavailable (no robots.txt rules), like a robots.txt
  that cannot be fetched.
- Local runs (APIFY_IS_AT_HOME not set) are unchanged.
- Files: `my_actor/common.py` (`ip_blocked`, `host_problem`, `check_public`, `guard_request` httpx request hook,
  `guard_site`, `Blocked`, `DnsLookupFailed`, `blocked_status_row`; `Http._robots_for` checks the site before fetching robots.txt),
  `my_actor/main.py` (hook on the httpx client, `Blocked` handled), `tests/test_ssrf_guard_1102t.py` (new),
  README Limits (one line).

### Unreleased (next 1.0.x build): error rows are free (round 1030b)

- **Charges (lower):** rows with an error are no longer charged: HTTP 400 or higher, not an HTML page, DNS or connection errors, timeouts, and URLs robots.txt disallows (directly or through a redirect). Before, only rows for a redirect that robots.txt disallows were free. Rows for URLs that returned metadata are charged as before.
- Unchanged: the rows output, their order and content, the **Include failed URLs** filter, and **Max results** (free rows still count toward it). Tested by comparing the rows of the previous code with the new code on the same inputs.
- Under a **Maximum cost per run**, error rows no longer use up the budget, so a run can save more paid rows before it stops.

### Unreleased (next 1.0.x build): plain URL input, per-URL timestamps, robots.txt on redirects (round 1030)

- **BREAKING (input editor):** the `urls` field uses the plain URL list editor (`stringList`); the default and prefill are now `["https://example.com"]`. Plain strings and `{"url": ...}` objects both work, so existing API inputs keep working. Before, Apify Console and apify-cli rejected plain strings. `startUrls` is accepted as an alias for `urls`.
- **BREAKING (output values):** `extractedAt` is now the time each URL was fetched, not the run start time.
- **BREAKING (fewer requests and charges):** redirects are followed one hop at a time (at most 10) and every target is checked against its own robots.txt. A URL that redirects to a disallowed page gets a free error row when **Include failed URLs** is on.
- The **Include failed URLs** description now says failed rows are charged (except the redirect case above) and lists the failure kinds.
- Empty input and the prefilled sample: 15 second limit; if the sample page is down or slow the run succeeds with one free row with a `warning` field.
- Unchanged: `jsonLdErrors` is an integer count (the README already said so).