Hacker News Link Rot: Dead Links & Archive Recovery

Hacker News never rechecks a link. A story submitted in 2012 keeps its points, its comment thread and its URL forever, and the site will happily send you to that URL in 2026 whether or not anything is still there.

Point this Actor at a slice of Hacker News and it tells you which links are dead, which ones answer 200 OK while the article is gone, and which ones now point at a domain nobody has registered — and for each dead link it fetches the Internet Archive's copy from around the date the story was posted.

Measured on 5,000 stories: 16.5% of the outbound links from 2012–2014 stories above 300 points no longer work, rising to 24.5% for 2007–2011. On today's front page it is 0%.

What it is for

• Checking a reading list or a bibliography. "Best of HN" lists, awesome-lists, internal wikis and course reading lists are mostly built from old HN stories. A quarter of the links in a 2010-era list are dead, and there is no way to tell by looking.
• Finding the domains that dropped. A link with 600 points that now points at an unregistered domain is an address anyone can buy and fill with whatever they like — while Hacker News keeps vouching for it.
• Recovering what was actually discussed. For every dead link, the archived copy from within a few days of the submission date, so the thread still makes sense.
• Measuring rot. The dead-link rate per year, per domain, and weighted by points rather than by URL count.

What it reports

Finding	Severity	What it means
link_domain_unregistered	critical	The domain is not registered with its registry. Anyone can take the address and decide what a decade of accumulated links now points at.
link_domain_unregistered_restricted	high	The domain is not registered, but its TLD is gated — .gov, .mil, .edu, .int, .arpa. A dead address, not one a stranger can move into.
link_dns_missing	high	The hostname does not resolve. The domain may still be held by its owner.
link_gone	medium	The server is up and answers 404 or 410 for this URL.
link_server_error	medium	The server answers 5xx. It was checked once; it may be temporary.
link_unreachable	medium	Nothing answered at all.
link_redirect_to_root	medium	The quiet one. The link redirects to the site's front page and drops the path. It reports 200 OK, so nothing on Hacker News — and nothing in an ordinary link checker — shows that the article is gone.
link_landed_offsite	low	Now lands on a different domain from the one submitted. Could be a rebrand, a shortener, or the domain changing hands.
link_blocked	low	The server answered 401, 403 or 429 to an automated request. It is up and declining to talk to a program; whether a person clicking through reaches a page was not established.
story_not_found	low	No item with that id.

Ask HN, Show HN text posts, polls and job posts carry no outbound link. They come back with linkKind: "text_post" and no findings, rather than being counted as healthy.

How this differs from the other Hacker News Actors

The Store already has ten or so Actors for Hacker News, and every one of them fetches: the front page, a search, a thread, the comments, a user's history. They hand you the data as HN holds it, which means they hand you the URL as HN holds it — unchecked.

None of them ask whether the link still works. This one does nothing else.

The second difference is the recovery. Reporting that a link is dead leaves the reader with a hole; reporting that it is dead and here is the copy from three days after it was posted leaves them with the article.

Input

Three ways in, in priority order.

// 1. The front page right now (the default)
{ "list": "topstories", "maxStories": 100 }

// 2. The archive - where the rot actually is
{ "minPoints": 300, "fromYear": 2012, "toYear": 2014, "maxStories": 500 }

// 3. Specific stories
{ "storyIds": ["3479558", "https://news.ycombinator.com/item?id=4493984"] }

Setting minPoints, minComments, query, fromYear or toYear switches the run to the search index, which reaches back to 2007. A points floor is the useful knob: a story that got 2,000 points and now leads nowhere is a different problem from one that got three.

Output

One row per story, plus five key-value records:

Record	What is in it
SUMMARY	Counts, the dead-link rate, the share of points sitting on dead links, and every caveat below stated in the record itself.
ACTION_LIST	Findings ranked by severity, then by points.
RECOVERY_MAP	Every dead link with its archived replacement — and the ones with no replacement, kept in rather than dropped.
DOMAIN_PROFILES	Rot per domain. A handful of dead domains usually account for most of it.
TAKEABLE_DOMAINS	The addresses anyone could register today, grouped by domain, with the stories that point at them.

What it will not do

• It does not scrape Hacker News. Both data sources are official, documented and keyless: the Firebase API for live lists and items, and the Algolia search index HN itself links to. news.ycombinator.com is never fetched.
• It does not pretend to be a browser. Every request carries a User-Agent naming this Actor. A server that refuses it is recorded as having refused; nothing is retried behind a browser identity.
• It does not judge page content. Only the status line is used. Deciding from the wording that a live page is "parked" or "a squatter" is the kind of guess that produces confident nonsense, so registration questions go to the registry instead.
• It does not mirror anything. Archived copies are linked, not copied.
• It does not follow a link more than five redirects deep.

How common is each of these? (measured, not guessed)

Severities were set from a sample, not from taste. Every figure below came out of this Actor's own modules.

Outbound links, by era (test/base-rates.mjs):

	2007–2011, ≥150 pts (n=192)	2012–2014, ≥300 pts (n=115)	Front page today (n=100)
dead	24.5%	16.5%	0.0%
blocked (401/403/429)	10.4%	15.7%	5.0%
redirected at all	74.0%	72.2%	0.0%
landed off-site, still 200	9.4%	11.3%	0.0%
link_redirect_to_root	1.0%	3.5%	0.0%
submitted as http://	84.4%	80.9%	0.0%

Two of those numbers decided a design:

blocked is 10–16% of old links. Folding refusals into "dead" would have doubled the reported rot and half of it would have been false. A 403 to a program says nothing about a person with a browser, so it is low and worded as such.

http:// is 80%+ of old submissions and nearly all of it now upgrades to https. A warning on that would fire on four out of five healthy links, so there is no finding for it at all.

Dropped domains (test/find-takeable.mjs, 5,000 stories from 2008–2012 above 30 points, 2,127 distinct domains):

domains that no longer resolve	133
…of those, not registered at all	34 (25.6%)
…still registered by someone	84
…on a TLD with no RDAP service — unknowable	15
unregistered as a share of every domain linked	1.60%

So the critical finding is rare, which is what makes it worth reading. Examples it found, each confirmed against the authoritative registry with a live control:

Domain	Story	Points
ndy.gd	"PayPal CEO writes personal response" (2012)	619
danielhope.org	"9 year old's DIY Cardboard Arcade gets Flash mobbed" (2012)	603
stopsoap.com	"Stop SOAP" (2012)	598
paulasmuth.com	"Dear Github, please fix the Watch-Button" (2012)	477

Where a "free" domain is not free

Sweeping that sample turned up change.gov — the Obama transition site, 86 points in November 2008 — as unregistered. It is, and the registry says so. But .gov registration requires a sponsoring US government organisation, so calling it an address "anyone can take" would have been a confident falsehood at the top severity.

Registrations on .gov, .mil, .edu, .int and .arpa are therefore reported as link_domain_unregistered_restricted at high, worded as a dead address rather than an open one, and they never appear in TAKEABLE_DOMAINS.

Where the registry cannot answer

Registration is read over RDAP through the IANA bootstrap, which covers about 1,200 TLDs — and several common ones are not among them, including .io, .co, .jp and .ro. A dead link on one of those is reported as dead with its registration recorded as unknown.

It is never assumed to be safely held, and it is never called free. Their absence from TAKEABLE_DOMAINS is not evidence that somebody holds them.

The lookup goes to the authoritative registry server every time. Public RDAP proxies are not used: one of them carries no records at all for .io and answers 404 for every name in it, which reads exactly like "this domain is free".

Two ceilings in the search API, both silent

1,000 results per query. Page 20 of a 50-per-page query returns HTTP 200 with an empty list — nothing distinguishes it from running out of matches. Runs that hit the ceiling say so in SUMMARY.notes. The way past it is a narrower year range, not more pages; test/find-takeable.mjs queries one year at a time for exactly this reason.

Approximate counts. Unless the result set is small enough, the API reports a hit count the size of the whole index — 44 million for a query that matches five thousand stories. An inexact count is recorded as null rather than passed on. A narrow window usually makes it exact; a wide one does not, even with a date bound.

Fact, inference, and the difference

Three kinds of statement appear in these rows and they are worded differently on purpose.

• "Nobody has registered this domain" is a registry's answer, read over RDAP from the authoritative server.
• "This URL answers 404" is a server's answer, read once, today.
• "This redirect drops the path" is this Actor reading two URLs against each other. It is the one inference here, and it is the finding most likely to be interesting, so it is named for what it is rather than reported as "broken".

An archived copy never softens any of them. The link is still broken; there is simply a copy.