This changelog highlights releases that materially changed Link Extractor for users. Dates use UTC; small internal maintenance is intentionally omitted.
Reliability
- Large crawls no longer end as timed-out runs: up to 10 sites are crawled at once, each one page at a time, the default timeout is now 1 hour, and a run nearing its timeout stops with its rows saved and a clear status message.
- One unusable
urls item no longer fails the whole run. Plain strings and { "url": "..." } objects are both accepted.
- Link-heavy pages no longer fail the run: a row is shortened to the 9 MB dataset limit and
error says so.
- Large files and very large pages no longer exhaust memory: non-HTML responses are skipped from their headers and only the first 5 MB of a page is read.
- A run with a very short timeout keeps a third of its time as a reserve, so it can still fetch its pages.
start is now charged with the first saved row, so a run that saves nothing costs nothing. Charging stops at your spending limit, and no pages are fetched after it.
Reliability
- Added
statusCode, finalUrl, and contentType fetch diagnostics to every page result.
- Non-2xx responses and explicitly non-HTML responses now return an actionable error instead of looking like successful empty extractions.
Features
- Added opt-in, breadth-first same-host crawling with
maxDepth and maxPagesPerStartUrl limits.
- Added
sourceStartUrl and depth provenance to every page summary.
- Existing inputs remain independent-page extractions because
maxDepth defaults to 0.
Features
- Added ready-to-run task examples for extracting all links, finding external links, and checking page link structure.
Features
- Initial public release.
- Extract all links from web pages with anchor text, rel attributes, internal/external classification, and page location.