# Hacker News Link Rot: Dead Links & Archive Recovery (`aiqlabs/hacker-news-link-rot`) Actor

Hacker News recommends links forever and never rechecks them. Point this at the front page, or at every story above 300 points from 2013, and it reports the dead links, the redirects that answer 200 OK while the article is gone, and the domains nobody has registered - with an archived copy of each.

- **URL**: https://apify.com/aiqlabs/hacker-news-link-rot.md
- **Developed by:** [Ai-Q Labs](https://apify.com/aiqlabs) (community)
- **Categories:** News, Developer tools, SEO tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Hacker News Link Rot: Dead Links & Archive Recovery

Hacker News never rechecks a link. A story submitted in 2012 keeps its points, its comment thread and its URL forever, and the site will happily send you to that URL in 2026 whether or not anything is still there.

Point this Actor at a slice of Hacker News and it tells you which links are dead, which ones **answer 200 OK while the article is gone**, and which ones now point at a domain **nobody has registered** — and for each dead link it fetches the Internet Archive's copy from around the date the story was posted.

Measured on 5,000 stories: **16.5%** of the outbound links from 2012–2014 stories above 300 points no longer work, rising to **24.5%** for 2007–2011. On today's front page it is **0%**.

### What it is for

- **Checking a reading list or a bibliography.** "Best of HN" lists, awesome-lists, internal wikis and course reading lists are mostly built from old HN stories. A quarter of the links in a 2010-era list are dead, and there is no way to tell by looking.
- **Finding the domains that dropped.** A link with 600 points that now points at an unregistered domain is an address anyone can buy and fill with whatever they like — while Hacker News keeps vouching for it.
- **Recovering what was actually discussed.** For every dead link, the archived copy from within a few days of the submission date, so the thread still makes sense.
- **Measuring rot.** The dead-link rate per year, per domain, and weighted by points rather than by URL count.

### What it reports

| Finding | Severity | What it means |
|---|---|---|
| `link_domain_unregistered` | **critical** | The domain is not registered with its registry. Anyone can take the address and decide what a decade of accumulated links now points at. |
| `link_domain_unregistered_restricted` | high | The domain is not registered, but its TLD is gated — `.gov`, `.mil`, `.edu`, `.int`, `.arpa`. A dead address, not one a stranger can move into. |
| `link_dns_missing` | high | The hostname does not resolve. The domain may still be held by its owner. |
| `link_gone` | medium | The server is up and answers 404 or 410 for this URL. |
| `link_server_error` | medium | The server answers 5xx. It was checked once; it may be temporary. |
| `link_unreachable` | medium | Nothing answered at all. |
| `link_redirect_to_root` | medium | **The quiet one.** The link redirects to the site's front page and drops the path. It reports 200 OK, so nothing on Hacker News — and nothing in an ordinary link checker — shows that the article is gone. |
| `link_landed_offsite` | low | Now lands on a different domain from the one submitted. Could be a rebrand, a shortener, or the domain changing hands. |
| `link_blocked` | low | The server answered 401, 403 or 429 to an automated request. It is up and declining to talk to a program; whether a person clicking through reaches a page was not established. |
| `story_not_found` | low | No item with that id. |

Ask HN, Show HN text posts, polls and job posts carry no outbound link. They come back with `linkKind: "text_post"` and no findings, rather than being counted as healthy.

### How this differs from the other Hacker News Actors

The Store already has ten or so Actors for Hacker News, and every one of them **fetches**: the front page, a search, a thread, the comments, a user's history. They hand you the data as HN holds it, which means they hand you the URL as HN holds it — unchecked.

None of them ask whether the link still works. This one does nothing else.

The second difference is the recovery. Reporting that a link is dead leaves the reader with a hole; reporting that it is dead **and here is the copy from three days after it was posted** leaves them with the article.

### Input

Three ways in, in priority order.

```jsonc
// 1. The front page right now (the default)
{ "list": "topstories", "maxStories": 100 }

// 2. The archive - where the rot actually is
{ "minPoints": 300, "fromYear": 2012, "toYear": 2014, "maxStories": 500 }

// 3. Specific stories
{ "storyIds": ["3479558", "https://news.ycombinator.com/item?id=4493984"] }
```

Setting `minPoints`, `minComments`, `query`, `fromYear` or `toYear` switches the run to the search index, which reaches back to 2007. A points floor is the useful knob: a story that got 2,000 points and now leads nowhere is a different problem from one that got three.

### Output

One row per story, plus five key-value records:

| Record | What is in it |
|---|---|
| `SUMMARY` | Counts, the dead-link rate, the share of *points* sitting on dead links, and every caveat below stated in the record itself. |
| `ACTION_LIST` | Findings ranked by severity, then by points. |
| `RECOVERY_MAP` | Every dead link with its archived replacement — and the ones with no replacement, kept in rather than dropped. |
| `DOMAIN_PROFILES` | Rot per domain. A handful of dead domains usually account for most of it. |
| `TAKEABLE_DOMAINS` | The addresses anyone could register today, grouped by domain, with the stories that point at them. |

### What it will not do

- **It does not scrape Hacker News.** Both data sources are official, documented and keyless: the Firebase API for live lists and items, and the Algolia search index HN itself links to. `news.ycombinator.com` is never fetched.
- **It does not pretend to be a browser.** Every request carries a User-Agent naming this Actor. A server that refuses it is recorded as having refused; nothing is retried behind a browser identity.
- **It does not judge page content.** Only the status line is used. Deciding from the wording that a live page is "parked" or "a squatter" is the kind of guess that produces confident nonsense, so registration questions go to the registry instead.
- **It does not mirror anything.** Archived copies are linked, not copied.
- **It does not follow a link more than five redirects deep.**

### How common is each of these? (measured, not guessed)

Severities were set from a sample, not from taste. Every figure below came out of this Actor's own modules.

**Outbound links, by era** (`test/base-rates.mjs`):

| | 2007–2011, ≥150 pts (n=192) | 2012–2014, ≥300 pts (n=115) | Front page today (n=100) |
|---|---|---|---|
| dead | **24.5%** | **16.5%** | **0.0%** |
| blocked (401/403/429) | 10.4% | 15.7% | 5.0% |
| redirected at all | 74.0% | 72.2% | 0.0% |
| landed off-site, still 200 | 9.4% | 11.3% | 0.0% |
| `link_redirect_to_root` | 1.0% | 3.5% | 0.0% |
| submitted as `http://` | 84.4% | 80.9% | 0.0% |

Two of those numbers decided a design:

**`blocked` is 10–16% of old links.** Folding refusals into "dead" would have doubled the reported rot and half of it would have been false. A 403 to a program says nothing about a person with a browser, so it is `low` and worded as such.

**`http://` is 80%+ of old submissions** and nearly all of it now upgrades to https. A warning on that would fire on four out of five healthy links, so there is no finding for it at all.

**Dropped domains** (`test/find-takeable.mjs`, 5,000 stories from 2008–2012 above 30 points, 2,127 distinct domains):

| | |
|---|---|
| domains that no longer resolve | 133 |
| …of those, **not registered at all** | **34 (25.6%)** |
| …still registered by someone | 84 |
| …on a TLD with no RDAP service — unknowable | 15 |
| unregistered as a share of every domain linked | **1.60%** |

So the critical finding is rare, which is what makes it worth reading. Examples it found, each confirmed against the authoritative registry with a live control:

| Domain | Story | Points |
|---|---|---|
| `ndy.gd` | "PayPal CEO writes personal response" (2012) | 619 |
| `danielhope.org` | "9 year old's DIY Cardboard Arcade gets Flash mobbed" (2012) | 603 |
| `stopsoap.com` | "Stop SOAP" (2012) | 598 |
| `paulasmuth.com` | "Dear Github, please fix the Watch-Button" (2012) | 477 |

### Where a "free" domain is not free

Sweeping that sample turned up `change.gov` — the Obama transition site, 86 points in November 2008 — as unregistered. It is, and the registry says so. But `.gov` registration requires a sponsoring US government organisation, so calling it an address "anyone can take" would have been a confident falsehood at the top severity.

Registrations on `.gov`, `.mil`, `.edu`, `.int` and `.arpa` are therefore reported as `link_domain_unregistered_restricted` at `high`, worded as a dead address rather than an open one, and they never appear in `TAKEABLE_DOMAINS`.

### Where the registry cannot answer

Registration is read over RDAP through the IANA bootstrap, which covers about 1,200 TLDs — and several common ones are not among them, including **`.io`, `.co`, `.jp` and `.ro`**. A dead link on one of those is reported as dead with its registration recorded as `unknown`.

It is never assumed to be safely held, and it is never called free. Their absence from `TAKEABLE_DOMAINS` is not evidence that somebody holds them.

The lookup goes to the authoritative registry server every time. Public RDAP proxies are not used: one of them carries no records at all for `.io` and answers 404 for every name in it, which reads exactly like "this domain is free".

### Two ceilings in the search API, both silent

**1,000 results per query.** Page 20 of a 50-per-page query returns HTTP 200 with an empty list — nothing distinguishes it from running out of matches. Runs that hit the ceiling say so in `SUMMARY.notes`. The way past it is a narrower year range, not more pages; `test/find-takeable.mjs` queries one year at a time for exactly this reason.

**Approximate counts.** Unless the result set is small enough, the API reports a hit count the size of the *whole index* — 44 million for a query that matches five thousand stories. An inexact count is recorded as `null` rather than passed on. A narrow window usually makes it exact; a wide one does not, even with a date bound.

### Fact, inference, and the difference

Three kinds of statement appear in these rows and they are worded differently on purpose.

- **"Nobody has registered this domain"** is a registry's answer, read over RDAP from the authoritative server.
- **"This URL answers 404"** is a server's answer, read once, today.
- **"This redirect drops the path"** is this Actor reading two URLs against each other. It is the one inference here, and it is the finding most likely to be interesting, so it is named for what it is rather than reported as "broken".

An archived copy never softens any of them. The link is still broken; there is simply a copy.

# Actor input Schema

## `list` (type: `string`):

Used when no search and no story ids are given. These are the live lists from the official Firebase API: topstories and newstories hold 500, beststories 200, the rest fewer. Checking a live list mostly reports healthy links - the rot is in the archive, which you reach with the search fields below.

## `minPoints` (type: `integer`):

Setting this switches the run to the Hacker News search index, which reaches back to 2007. A points floor is how you find the links that mattered: a story that got 2,000 points and now leads nowhere is a different problem from one that got 3.

## `fromYear` (type: `integer`):

Optional, and worth setting: a narrow window usually gets an exact match count out of the search API, where a broad one gets an estimate the size of the whole index. It also keeps each query inside the API's 1,000-result ceiling. Hacker News began in 2007.

## `toYear` (type: `integer`):

Optional. Inclusive - 2014 means up to the end of 2014. Pair it with fromYear to keep each query inside the search API's 1,000-result ceiling.

## `query` (type: `string`):

Optional free text, matched against story titles and URLs by the Hacker News search index. Leave it empty to take everything that fits the other filters.

## `minComments` (type: `integer`):

Optional alternative to a points floor. Like minPoints, setting it switches the run to the search index.

## `storyIds` (type: `array`):

Optional, and takes priority over everything else. Accepts bare ids and news.ycombinator.com/item?id=... URLs. Comments and polls are refused with a reason rather than audited, because they carry no outbound link.

## `checkDomainRegistration` (type: `boolean`):

Only for links that already failed. This is what separates 'the page is gone' from 'the address is free for anyone to take' - the second means a stranger can register the domain and decide what a decade of accumulated links now points at.

## `findArchivedCopies` (type: `boolean`):

Asks the Internet Archive for the capture nearest the date the story was posted, rather than the most recent one - the last capture of a dead domain is usually a parking page. Only captures that were themselves successful are offered.

## `onlyIssues` (type: `boolean`):

Leave this off to get a row per story including the healthy ones, which is what you want when the output feeds a spreadsheet.

## `maxStories` (type: `integer`):

A ceiling on the run. Anything dropped is stated in SUMMARY rather than silently truncated. The search index itself will not return more than 1,000 results for one query, whatever you put here.

## `requestTimeoutSecs` (type: `integer`):

How long to wait for any one request - the Hacker News API, somebody's website, the Internet Archive, or a registry - before giving up on it and recording the reason.

## Actor input object example

```json
{
  "list": "topstories",
  "minPoints": 300,
  "fromYear": 2012,
  "toYear": 2014,
  "query": "",
  "minComments": 0,
  "storyIds": [],
  "checkDomainRegistration": true,
  "findArchivedCopies": true,
  "onlyIssues": false,
  "maxStories": 200,
  "requestTimeoutSecs": 30
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `csv` (type: `string`):

No description

## `actionList` (type: `string`):

No description

## `takeableDomains` (type: `string`):

No description

## `recoveryMap` (type: `string`):

No description

## `domainProfiles` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "minPoints": 300,
    "fromYear": 2012,
    "toYear": 2014
};

// Run the Actor and wait for it to finish
const run = await client.actor("aiqlabs/hacker-news-link-rot").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "minPoints": 300,
    "fromYear": 2012,
    "toYear": 2014,
}

# Run the Actor and wait for it to finish
run = client.actor("aiqlabs/hacker-news-link-rot").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "minPoints": 300,
  "fromYear": 2012,
  "toYear": 2014
}' |
apify call aiqlabs/hacker-news-link-rot --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=aiqlabs/hacker-news-link-rot",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/3yDOaVJuVUYaWTUhY/builds/OPkd6QqQyjXB6tZ3p/openapi.json