Wayback Machine URL Extractor & Lost Page Finder

Find every page a website ever had, see which ones are broken today, and get a ready-to-use 301 redirect map, all from the Internet Archive's Wayback Machine.

This actor lists all archived URLs of any domain through the official Internet Archive CDX API. It then checks whether each page still works on the live site and flags the ones that are lost (404/410), silently redirected to the homepage, or unreachable. For lost pages it suggests the closest live replacement URL and can recover the archived title, meta description, and text. Use it for SEO site migrations, broken backlink recovery, expired domain research, and content restoration.

What does Wayback Machine Lost Page Finder do?

• Extracts archived URLs for a host, a whole domain with subdomains, or a URL prefix. You get the first-seen date and links to the first, latest, and all snapshots.
• Checks live HTTP status of every archived URL (tries HTTPS first, then follows redirects) and sorts it into one of these categories: ok, redirected, redirects-to-homepage, lost, server-error, blocked-or-client-error, or unreachable.
• Builds a 301 redirect map. For each lost URL it suggests the most similar live URL on the same site. You can download it as a REDIRECT_MAP CSV (from,to,score,status,category) and paste it into Nginx, Apache, Cloudflare, Netlify, or a WordPress redirect plugin.
• Recovers lost content from the latest archived snapshot: title, meta description, H1, word count, and clean text.
• Cleans the noise by default. It skips CSS/JS/images/fonts, wp-json, cdn-cgi, tracking parameters, login/activation links, and token URLs, and it removes duplicate ?ref= query-string variants.
• Handles the Wayback Machine's flakiness. The CDX API often answers with 503/504 or "temporarily offline", so the actor retries with exponential backoff and resumes paging where it stopped.

Who is it for? Use cases

Use case	How this actor helps
SEO site migration audit	List every URL the old site ever had and make sure each one now returns 200 or a proper 301.
Broken backlink / link equity recovery	Find old URLs that other sites still link to but that return 404 now, then redirect them to the best live page.
"Soft 404" detection	Find pages that redirect everything to the homepage (Google treats these as soft 404s).
Expired & aged domain research	See what a domain hosted in the past and how many pages it had before you buy it.
Content recovery	Restore deleted blog posts, docs, or product pages from archived snapshots.
Competitor research	See how a competitor's site structure, product lines, or landing pages changed over time.

How to use it

1. Enter one or more domains (e.g. example.com) or URL prefixes.
2. Choose a match type: host (exactly that host), domain (including all subdomains), or prefix (e.g. example.com/blog/).
3. Set Max URLs per domain to control the cost.
4. Keep Check live status on to find lost pages. Optionally set Recover archived content to lost.
5. Run it, then download the dataset (JSON, CSV, Excel) or the REDIRECT_MAP CSV from the key-value store.

Example input

{
  "domains": ["paulgraham.com", "djangoproject.com"],
  "matchType": "host",
  "maxUrlsPerDomain": 150,
  "checkLiveStatus": true,
  "outputOnlyLost": false,
  "extractArchivedContent": "lost",
  "maxContentExtractions": 5,
  "maxTextChars": 400
}

Example output (one dataset item)

{
  "domain": "djangoproject.com",
  "url": "http://www.djangoproject.com/community2/",
  "firstArchivedAt": "2007-02-08T04:12:05Z",
  "firstCaptureStatus": 200,
  "mimeType": "text/html",
  "firstSnapshotUrl": "https://web.archive.org/web/20070208041205/http://www.djangoproject.com/community2/",
  "allSnapshotsUrl": "https://web.archive.org/web/*/http://www.djangoproject.com/community2/",
  "liveStatus": 404,
  "liveFinalUrl": "https://www.djangoproject.com/community2/",
  "liveCategory": "lost",
  "suggestedRedirect": "https://www.djangoproject.com/community/dsf/",
  "suggestedRedirectScore": 0.8,
  "archivedTitle": "Django | Community",
  "archivedWordCount": 988,
  "archivedText": "Django community ..."
}

Example REDIRECT_MAP (CSV in the key-value store)

from,to,score,status,category
http://www.djangoproject.com/community2/,https://www.djangoproject.com/community/dsf/,0.8,404,lost
http://www.paulgraham.com/ac.html,http://paulgraham.com/arc.html,0.86,200,redirects-to-homepage

The key-value store also contains a SUMMARY record with per-domain counts by live-status category.

Input reference

Field	Default	Description
domains	–	Domains, hosts, or URL prefixes to look up (required).
matchType	host	host, domain (with subdomains), or prefix.
maxUrlsPerDomain	50	Maximum number of unique archived URLs per domain.
checkLiveStatus	true	Check each URL on the live site.
outputOnlyLost	false	Only save URLs that are not OK on the live site.
extractArchivedContent	none	none, lost (only broken pages), or all.
maxContentExtractions	50	Maximum number of pages to recover content for, per domain.
maxTextChars	3000	Maximum length of recovered text (0 = metadata only).
onlyHtml / onlySuccessfulCaptures	true	Only HTML pages that were archived with status 200.
ignoreQueryStrings	true	Merge page?ref=x into page.
fromYear / toYear	–	Limit to URLs first archived in a year range.
includeUrlPatterns / excludeUrlPatterns	–	Regex filters (custom excludes replace the defaults).
liveCheckConcurrency	4	Parallel live requests (kept low to be polite).

How much does it cost?

The actor uses pay-per-event pricing, so you only pay for what you get:

Event	Price	1,000 URLs
Actor start (per run)	$0.005	–
Archived URL saved	$0.0015	$1.50
Live status check	$0.0005	$0.50
Archived content recovered	$0.004	$4.00

A typical audit of 1,000 URLs with live checks costs about $2.00. Set Maximum cost per run in the run options, and the actor stops cleanly when it reaches your limit. URL extraction alone (live check off) costs $1.50 per 1,000 URLs.

FAQ

Is it legal to use data from the Wayback Machine? The actor uses the Internet Archive's official public CDX API and public snapshot URLs. It collects no personal data and never logs in. The live check sends a single HEAD/GET request per URL to the public website. Use the data in line with the Internet Archive's terms and copyright law, especially when you republish recovered content.

Why are some runs slow? The Wayback Machine CDX API can take 10–30 seconds per page and sometimes returns 503 errors. The actor retries automatically, so large domains just take longer. Use maxUrlsPerDomain, prefix match, or a year range to speed things up.

How are redirect suggestions chosen? Each lost URL's path is compared with the paths of live, working URLs on the same host using fuzzy string similarity. A score of 1.0 means a near-identical path. When nothing scores 0.6 or higher, the homepage is suggested with a score of 0. Always review the suggestions before you deploy them.

Can I get only broken pages? Yes. Set outputOnlyLost: true, and you pay only for broken URLs (live checks are still charged).

Does it work with subdomains? Yes. Use matchType: "domain".

Related actors by pandabear

• App Store Scraper - Search, Top Charts & Keyword Ranks: iOS apps by keyword, top chart, category or developer, with full details and ASO keyword rank tracking.
• Sitemap Extractor & SEO Index Audit: extracts every URL from a site's XML sitemaps and finds 404, redirect, noindex and canonical problems.
• Bulk WHOIS & Domain Availability Checker: checks domain age, registrar, expiry date and availability in bulk. Useful before buying an expired domain.
• App Store Reviews Scraper - All Countries + Analytics: Apple App Store reviews from 76 countries, with app details, rating trends by version and complaint keywords.
• Website Contact Scraper - Emails, Phones, Socials & Tech Stack: emails, phones, social links, company info and tech stack from any list of websites.
• Trending Topics Scraper - Google Trends, HN, Wikipedia & More: trending topics from Google Trends, Hacker News, Wikipedia, GitHub, Bluesky, Baidu and more, with heat scores and cross-platform clusters.

Related keywords

Wayback Machine scraper, Wayback Machine URL extractor, archive.org CDX API, find deleted pages, broken link checker, 404 finder, soft 404, SEO redirect map, 301 redirects generator, site migration audit, expired domain history, recover deleted website content, waybackurls alternative.

中文说明

这个工具能做什么？

Wayback Machine 网址提取 & 失效页面查找器通过互联网档案馆（Internet Archive）官方公开的 CDX API，列出任意网站历史上被存档过的全部网址。然后它会逐一检查这些网址在线上网站的当前状态，找出以下几类问题页面：

• 已失效：返回 404/410；
• 被统一跳转到首页：Google 视为"软 404"；
• 无法访问：连接失败或超时。

对失效页面，它会在同一网站的正常页面中自动匹配最相近的网址，生成可以直接使用的 301 跳转表，还可以从存档快照中恢复页面的标题、描述和正文。

适用场景：

• 网站改版或迁移的 SEO 审计
• 找回失效外链的权重
• 检测"软 404"
• 调研过期域名的历史
• 恢复被删除的文章或文档
• 研究竞品网站结构的变化

使用方法

1. 在 domains 中填入一个或多个域名（如 example.com）或网址前缀；
2. 选择匹配方式：host（仅该主机）、domain（包含所有子域名）或 prefix（如 example.com/blog/）；
3. 用"每个域名最大网址数"控制费用；
4. 保持"检测线上状态"开启；如需恢复内容，把"恢复存档内容"设为 lost；
5. 运行完成后，可以下载数据集（JSON/CSV/Excel），或从键值存储中下载 REDIRECT_MAP 跳转表（CSV）。

主要输入参数

参数	默认值	说明
domains	–	要查询的域名、主机或网址前缀（必填）
matchType	host	host / domain（含子域名）/ prefix
maxUrlsPerDomain	50	每个域名最多提取多少个不重复网址
checkLiveStatus	true	是否检测每个网址的线上状态
outputOnlyLost	false	只输出有问题的网址
extractArchivedContent	none	none / lost（仅失效页面）/ all
maxContentExtractions	50	每个域名最多恢复多少个页面的内容
maxTextChars	3000	恢复正文的最大字符数（0 表示只要元数据）
onlyHtml / onlySuccessfulCaptures	true	只保留状态为 200 的 HTML 存档
ignoreQueryStrings	true	把 page?ref=x 合并为 page
fromYear / toYear	–	按首次存档年份筛选
includeUrlPatterns / excludeUrlPatterns	–	正则过滤（自定义排除规则会替换默认规则）
liveCheckConcurrency	4	线上检测的并发数（保持较低，礼貌访问）

输出内容

数据集中每条记录对应一个网址，主要字段如下：

字段	含义
url	网址
firstArchivedAt	首次存档时间
firstSnapshotUrl / latestSnapshotUrl / allSnapshotsUrl	首个、最新、全部快照的链接
liveStatus	线上 HTTP 状态码
liveFinalUrl	跳转后的最终网址
liveCategory	分类：ok、redirected、redirects-to-homepage、lost、server-error、blocked-or-client-error、unreachable
suggestedRedirect / suggestedRedirectScore	建议跳转目标及相似度，1.0 表示几乎相同
archivedTitle / archivedDescription / archivedH1 / archivedWordCount / archivedText	从存档恢复的内容

键值存储中还有两项：

• SUMMARY：每个域名按状态分类的统计；
• REDIRECT_MAP：跳转表 CSV，列为 from,to,score,status,category，可以导入 Nginx、Apache、Cloudflare、Netlify 或 WordPress 跳转插件。

价格（按事件计费）

事件	单价	每 1,000 个网址
每次运行启动	$0.005	–
保存一个存档网址	$0.0015	$1.50
一次线上状态检测	$0.0005	$0.50
恢复一个页面内容	$0.004	$4.00

典型用法（1,000 个网址 + 线上检测）约 $2.00。你可以在运行选项中设置"每次运行最高费用"，达到上限时工具会正常停止，已获取的结果都会保留。

常见问题

使用 Wayback Machine 的数据合法吗？ 本工具只使用互联网档案馆官方公开的 CDX API 和公开快照链接，不登录，不收集个人数据；线上检测对每个网址只发一次 HEAD/GET 请求。转载恢复的内容时，请遵守互联网档案馆的条款和版权法。

为什么有时运行较慢？ CDX API 每页可能需要 10–30 秒，偶尔会返回 503。本工具会自动重试，所以大型网站只是耗时更长。可以通过限制网址数量、使用 prefix 匹配或年份范围来加快速度。

跳转建议是怎么生成的？ 把失效网址的路径与同一网站正常页面的路径做模糊相似度比较。如果相似度都低于 0.6，就建议跳转到首页（分数为 0）。部署前请人工检查。

能只要失效页面吗？ 可以。设置 outputOnlyLost: true，只为有问题的网址付费（线上检测仍然计费）。

支持子域名吗？ 支持，使用 matchType: "domain" 即可。

pandabear 的其他工具

• App Store Scraper - Search, Top Charts & Keyword Ranks：按关键词、排行榜、分类或开发者抓取苹果应用数据，并跟踪 ASO 关键词排名
• Sitemap Extractor & SEO Index Audit：提取网站全部 sitemap 网址，找出 404、重定向、noindex、canonical 等问题
• Bulk WHOIS & Domain Availability Checker：批量查询域名年龄、注册商、到期日期和是否可注册，购买过期域名前很有用
• App Store Reviews Scraper：抓取苹果 App Store 76 个国家的应用评论，附带应用详情、按版本的评分趋势和差评关键词
• Website Contact Scraper：从网站列表中提取邮箱、电话、社交媒体链接、公司信息和技术栈
• Trending Topics Scraper：汇总 Google Trends、Hacker News、维基百科、GitHub、Bluesky、百度热搜等平台的热门话题，附热度分和跨平台聚类