# Wayback Machine Scraper - Archived URLs & Lost 404 Page Finder (`pandabear/wayback-lost-url-finder`) Actor

Wayback Machine scraper: extract every archived URL of a website from the Internet Archive CDX API, check which still work today, find lost 404 pages, get 301 redirect suggestions and recover old page titles and text. No API key needed.

- **URL**: https://apify.com/pandabear/wayback-lost-url-finder.md
- **Developed by:** [Andy zeng](https://apify.com/pandabear) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.50 / 1,000 archived url saveds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Wayback Machine URL Extractor & Lost Page Finder

**Find every page a website ever had, see which ones are broken today, and get a ready-to-use 301 redirect map, all from the Internet Archive's Wayback Machine.**

This actor lists all archived URLs of any domain through the official [Internet Archive CDX API](https://archive.org/developers/wayback-cdx-server.html). It then checks whether each page still works on the live site and flags the ones that are **lost** (404/410), **silently redirected to the homepage**, or **unreachable**. For lost pages it suggests the closest live replacement URL and can recover the archived title, meta description, and text. Use it for SEO site migrations, broken backlink recovery, expired domain research, and content restoration.

### What does Wayback Machine Lost Page Finder do?

- **Extracts archived URLs** for a host, a whole domain with subdomains, or a URL prefix. You get the first-seen date and links to the first, latest, and all snapshots.
- **Checks live HTTP status** of every archived URL (tries HTTPS first, then follows redirects) and sorts it into one of these categories: `ok`, `redirected`, `redirects-to-homepage`, `lost`, `server-error`, `blocked-or-client-error`, or `unreachable`.
- **Builds a 301 redirect map.** For each lost URL it suggests the most similar *live* URL on the same site. You can download it as a `REDIRECT_MAP` CSV (`from,to,score,status,category`) and paste it into Nginx, Apache, Cloudflare, Netlify, or a WordPress redirect plugin.
- **Recovers lost content** from the latest archived snapshot: title, meta description, H1, word count, and clean text.
- **Cleans the noise by default.** It skips CSS/JS/images/fonts, `wp-json`, `cdn-cgi`, tracking parameters, login/activation links, and token URLs, and it removes duplicate `?ref=` query-string variants.
- **Handles the Wayback Machine's flakiness.** The CDX API often answers with 503/504 or "temporarily offline", so the actor retries with exponential backoff and resumes paging where it stopped.

### Who is it for? Use cases

| Use case | How this actor helps |
|---|---|
| **SEO site migration audit** | List every URL the old site ever had and make sure each one now returns 200 or a proper 301. |
| **Broken backlink / link equity recovery** | Find old URLs that other sites still link to but that return 404 now, then redirect them to the best live page. |
| **"Soft 404" detection** | Find pages that redirect everything to the homepage (Google treats these as soft 404s). |
| **Expired & aged domain research** | See what a domain hosted in the past and how many pages it had before you buy it. |
| **Content recovery** | Restore deleted blog posts, docs, or product pages from archived snapshots. |
| **Competitor research** | See how a competitor's site structure, product lines, or landing pages changed over time. |

### How to use it

1. Enter one or more **domains** (e.g. `example.com`) or URL prefixes.
2. Choose a **match type**: `host` (exactly that host), `domain` (including all subdomains), or `prefix` (e.g. `example.com/blog/`).
3. Set **Max URLs per domain** to control the cost.
4. Keep **Check live status** on to find lost pages. Optionally set **Recover archived content** to `lost`.
5. Run it, then download the dataset (JSON, CSV, Excel) or the `REDIRECT_MAP` CSV from the key-value store.

#### Example input

```json
{
  "domains": ["paulgraham.com", "djangoproject.com"],
  "matchType": "host",
  "maxUrlsPerDomain": 150,
  "checkLiveStatus": true,
  "outputOnlyLost": false,
  "extractArchivedContent": "lost",
  "maxContentExtractions": 5,
  "maxTextChars": 400
}
```

#### Example output (one dataset item)

```json
{
  "domain": "djangoproject.com",
  "url": "http://www.djangoproject.com/community2/",
  "firstArchivedAt": "2007-02-08T04:12:05Z",
  "firstCaptureStatus": 200,
  "mimeType": "text/html",
  "firstSnapshotUrl": "https://web.archive.org/web/20070208041205/http://www.djangoproject.com/community2/",
  "allSnapshotsUrl": "https://web.archive.org/web/*/http://www.djangoproject.com/community2/",
  "liveStatus": 404,
  "liveFinalUrl": "https://www.djangoproject.com/community2/",
  "liveCategory": "lost",
  "suggestedRedirect": "https://www.djangoproject.com/community/dsf/",
  "suggestedRedirectScore": 0.8,
  "archivedTitle": "Django | Community",
  "archivedWordCount": 988,
  "archivedText": "Django community ..."
}
```

#### Example `REDIRECT_MAP` (CSV in the key-value store)

```csv
from,to,score,status,category
http://www.djangoproject.com/community2/,https://www.djangoproject.com/community/dsf/,0.8,404,lost
http://www.paulgraham.com/ac.html,http://paulgraham.com/arc.html,0.86,200,redirects-to-homepage
```

The key-value store also contains a `SUMMARY` record with per-domain counts by live-status category.

### Input reference

| Field | Default | Description |
|---|---|---|
| `domains` | – | Domains, hosts, or URL prefixes to look up (required). |
| `matchType` | `host` | `host`, `domain` (with subdomains), or `prefix`. |
| `maxUrlsPerDomain` | 50 | Maximum number of unique archived URLs per domain. |
| `checkLiveStatus` | true | Check each URL on the live site. |
| `outputOnlyLost` | false | Only save URLs that are not OK on the live site. |
| `extractArchivedContent` | `none` | `none`, `lost` (only broken pages), or `all`. |
| `maxContentExtractions` | 50 | Maximum number of pages to recover content for, per domain. |
| `maxTextChars` | 3000 | Maximum length of recovered text (0 = metadata only). |
| `onlyHtml` / `onlySuccessfulCaptures` | true | Only HTML pages that were archived with status 200. |
| `ignoreQueryStrings` | true | Merge `page?ref=x` into `page`. |
| `fromYear` / `toYear` | – | Limit to URLs first archived in a year range. |
| `includeUrlPatterns` / `excludeUrlPatterns` | – | Regex filters (custom excludes replace the defaults). |
| `liveCheckConcurrency` | 4 | Parallel live requests (kept low to be polite). |

### How much does it cost?

The actor uses **pay-per-event** pricing, so you only pay for what you get:

| Event | Price | 1,000 URLs |
|---|---|---|
| Actor start (per run) | $0.005 | – |
| Archived URL saved | $0.0015 | $1.50 |
| Live status check | $0.0005 | $0.50 |
| Archived content recovered | $0.004 | $4.00 |

A typical audit of **1,000 URLs with live checks costs about $2.00**. Set *Maximum cost per run* in the run options, and the actor stops cleanly when it reaches your limit. URL extraction alone (live check off) costs $1.50 per 1,000 URLs.

### FAQ

**Is it legal to use data from the Wayback Machine?**
The actor uses the Internet Archive's official public CDX API and public snapshot URLs. It collects no personal data and never logs in. The live check sends a single HEAD/GET request per URL to the public website. Use the data in line with the Internet Archive's terms and copyright law, especially when you republish recovered content.

**Why are some runs slow?**
The Wayback Machine CDX API can take 10–30 seconds per page and sometimes returns 503 errors. The actor retries automatically, so large domains just take longer. Use `maxUrlsPerDomain`, `prefix` match, or a year range to speed things up.

**How are redirect suggestions chosen?**
Each lost URL's path is compared with the paths of live, working URLs on the same host using fuzzy string similarity. A score of 1.0 means a near-identical path. When nothing scores 0.6 or higher, the homepage is suggested with a score of 0. Always review the suggestions before you deploy them.

**Can I get only broken pages?**
Yes. Set `outputOnlyLost: true`, and you pay only for broken URLs (live checks are still charged).

**Does it work with subdomains?**
Yes. Use `matchType: "domain"`.

### Related actors by pandabear

- [App Store Scraper - Search, Top Charts & Keyword Ranks](https://apify.com/pandabear/app-store-search-charts-scraper): iOS apps by keyword, top chart, category or developer, with full details and ASO keyword rank tracking.
- [Sitemap Extractor & SEO Index Audit](https://apify.com/pandabear/sitemap-seo-auditor): extracts every URL from a site's XML sitemaps and finds 404, redirect, noindex and canonical problems.
- [Bulk WHOIS & Domain Availability Checker](https://apify.com/pandabear/bulk-whois-domain-checker): checks domain age, registrar, expiry date and availability in bulk. Useful before buying an expired domain.
- [App Store Reviews Scraper - All Countries + Analytics](https://apify.com/pandabear/app-store-reviews-scraper): Apple App Store reviews from 76 countries, with app details, rating trends by version and complaint keywords.
- [Website Contact Scraper - Emails, Phones, Socials & Tech Stack](https://apify.com/pandabear/website-contact-enrichment): emails, phones, social links, company info and tech stack from any list of websites.
- [Trending Topics Scraper - Google Trends, HN, Wikipedia & More](https://apify.com/pandabear/trending-topics-aggregator): trending topics from Google Trends, Hacker News, Wikipedia, GitHub, Bluesky, Baidu and more, with heat scores and cross-platform clusters.

### Related keywords

Wayback Machine scraper, Wayback Machine URL extractor, archive.org CDX API, find deleted pages, broken link checker, 404 finder, soft 404, SEO redirect map, 301 redirects generator, site migration audit, expired domain history, recover deleted website content, waybackurls alternative.

### 中文说明

#### 这个工具能做什么？

**Wayback Machine 网址提取 & 失效页面查找器**通过互联网档案馆（Internet Archive）官方公开的 CDX API，列出任意网站历史上被存档过的全部网址。然后它会逐一检查这些网址在线上网站的当前状态，找出以下几类问题页面：

- **已失效**：返回 404/410；
- **被统一跳转到首页**：Google 视为"软 404"；
- **无法访问**：连接失败或超时。

对失效页面，它会在同一网站的正常页面中自动匹配最相近的网址，生成可以直接使用的 **301 跳转表**，还可以从存档快照中恢复页面的标题、描述和正文。

**适用场景：**

- 网站改版或迁移的 SEO 审计
- 找回失效外链的权重
- 检测"软 404"
- 调研过期域名的历史
- 恢复被删除的文章或文档
- 研究竞品网站结构的变化

#### 使用方法

1. 在 `domains` 中填入一个或多个域名（如 `example.com`）或网址前缀；
2. 选择匹配方式：`host`（仅该主机）、`domain`（包含所有子域名）或 `prefix`（如 `example.com/blog/`）；
3. 用"每个域名最大网址数"控制费用；
4. 保持"检测线上状态"开启；如需恢复内容，把"恢复存档内容"设为 `lost`；
5. 运行完成后，可以下载数据集（JSON/CSV/Excel），或从键值存储中下载 `REDIRECT_MAP` 跳转表（CSV）。

#### 主要输入参数

| 参数 | 默认值 | 说明 |
|---|---|---|
| `domains` | – | 要查询的域名、主机或网址前缀（必填） |
| `matchType` | `host` | `host` / `domain`（含子域名）/ `prefix` |
| `maxUrlsPerDomain` | 50 | 每个域名最多提取多少个不重复网址 |
| `checkLiveStatus` | true | 是否检测每个网址的线上状态 |
| `outputOnlyLost` | false | 只输出有问题的网址 |
| `extractArchivedContent` | `none` | `none` / `lost`（仅失效页面）/ `all` |
| `maxContentExtractions` | 50 | 每个域名最多恢复多少个页面的内容 |
| `maxTextChars` | 3000 | 恢复正文的最大字符数（0 表示只要元数据） |
| `onlyHtml` / `onlySuccessfulCaptures` | true | 只保留状态为 200 的 HTML 存档 |
| `ignoreQueryStrings` | true | 把 `page?ref=x` 合并为 `page` |
| `fromYear` / `toYear` | – | 按首次存档年份筛选 |
| `includeUrlPatterns` / `excludeUrlPatterns` | – | 正则过滤（自定义排除规则会替换默认规则） |
| `liveCheckConcurrency` | 4 | 线上检测的并发数（保持较低，礼貌访问） |

#### 输出内容

数据集中每条记录对应一个网址，主要字段如下：

| 字段 | 含义 |
|---|---|
| `url` | 网址 |
| `firstArchivedAt` | 首次存档时间 |
| `firstSnapshotUrl` / `latestSnapshotUrl` / `allSnapshotsUrl` | 首个、最新、全部快照的链接 |
| `liveStatus` | 线上 HTTP 状态码 |
| `liveFinalUrl` | 跳转后的最终网址 |
| `liveCategory` | 分类：`ok`、`redirected`、`redirects-to-homepage`、`lost`、`server-error`、`blocked-or-client-error`、`unreachable` |
| `suggestedRedirect` / `suggestedRedirectScore` | 建议跳转目标及相似度，1.0 表示几乎相同 |
| `archivedTitle` / `archivedDescription` / `archivedH1` / `archivedWordCount` / `archivedText` | 从存档恢复的内容 |

键值存储中还有两项：

- `SUMMARY`：每个域名按状态分类的统计；
- `REDIRECT_MAP`：跳转表 CSV，列为 `from,to,score,status,category`，可以导入 Nginx、Apache、Cloudflare、Netlify 或 WordPress 跳转插件。

#### 价格（按事件计费）

| 事件 | 单价 | 每 1,000 个网址 |
|---|---|---|
| 每次运行启动 | $0.005 | – |
| 保存一个存档网址 | $0.0015 | $1.50 |
| 一次线上状态检测 | $0.0005 | $0.50 |
| 恢复一个页面内容 | $0.004 | $4.00 |

典型用法（1,000 个网址 + 线上检测）约 **$2.00**。你可以在运行选项中设置"每次运行最高费用"，达到上限时工具会正常停止，已获取的结果都会保留。

#### 常见问题

**使用 Wayback Machine 的数据合法吗？**
本工具只使用互联网档案馆官方公开的 CDX API 和公开快照链接，不登录，不收集个人数据；线上检测对每个网址只发一次 HEAD/GET 请求。转载恢复的内容时，请遵守互联网档案馆的条款和版权法。

**为什么有时运行较慢？**
CDX API 每页可能需要 10–30 秒，偶尔会返回 503。本工具会自动重试，所以大型网站只是耗时更长。可以通过限制网址数量、使用 `prefix` 匹配或年份范围来加快速度。

**跳转建议是怎么生成的？**
把失效网址的路径与同一网站正常页面的路径做模糊相似度比较。如果相似度都低于 0.6，就建议跳转到首页（分数为 0）。部署前请人工检查。

**能只要失效页面吗？**
可以。设置 `outputOnlyLost: true`，只为有问题的网址付费（线上检测仍然计费）。

**支持子域名吗？**
支持，使用 `matchType: "domain"` 即可。

#### pandabear 的其他工具

- [App Store Scraper - Search, Top Charts & Keyword Ranks](https://apify.com/pandabear/app-store-search-charts-scraper)：按关键词、排行榜、分类或开发者抓取苹果应用数据，并跟踪 ASO 关键词排名
- [Sitemap Extractor & SEO Index Audit](https://apify.com/pandabear/sitemap-seo-auditor)：提取网站全部 sitemap 网址，找出 404、重定向、noindex、canonical 等问题
- [Bulk WHOIS & Domain Availability Checker](https://apify.com/pandabear/bulk-whois-domain-checker)：批量查询域名年龄、注册商、到期日期和是否可注册，购买过期域名前很有用
- [App Store Reviews Scraper](https://apify.com/pandabear/app-store-reviews-scraper)：抓取苹果 App Store 76 个国家的应用评论，附带应用详情、按版本的评分趋势和差评关键词
- [Website Contact Scraper](https://apify.com/pandabear/website-contact-enrichment)：从网站列表中提取邮箱、电话、社交媒体链接、公司信息和技术栈
- [Trending Topics Scraper](https://apify.com/pandabear/trending-topics-aggregator)：汇总 Google Trends、Hacker News、维基百科、GitHub、Bluesky、百度热搜等平台的热门话题，附热度分和跨平台聚类

# Actor input Schema

## `domains` (type: `array`):

Websites to analyze, e.g. `example.com` or `example.com/blog`. One per line.

## `matchType` (type: `string`):

`host` = all pages on this exact host (www is treated the same). `domain` = also include all subdomains (slower on big sites). `prefix` = only URLs starting with the given path, e.g. `example.com/blog/`.

## `maxUrlsPerDomain` (type: `integer`):

Stop after this many unique archived URLs per domain. Controls cost and run time.

## `checkLiveStatus` (type: `boolean`):

Request each URL on the live site (HEAD, falling back to GET) to find pages that are now 404/410, redirect to the homepage, or no longer resolve. Needed for the redirect map.

## `outputOnlyLost` (type: `boolean`):

Save only URLs that are lost (404/410), redirect to the homepage, return 5xx, or are unreachable. You are only charged for saved results.

## `extractArchivedContent` (type: `string`):

Fetch the latest archived snapshot and extract title, meta description, H1, word count and text. Useful to rebuild lost pages.

## `maxContentExtractions` (type: `integer`):

Upper limit of archived pages to download per domain.

## `maxTextChars` (type: `integer`):

Truncate recovered page text to this many characters. 0 = don't store the text, only title/description/word count.

## `onlyHtml` (type: `boolean`):

Skip images, PDFs, CSS, JS and other non-HTML captures.

## `onlySuccessfulCaptures` (type: `boolean`):

Only list URLs the Wayback Machine captured with status 200, i.e. real pages that existed. Turn off to also include archived redirects and errors.

## `ignoreQueryStrings` (type: `boolean`):

Treat `page?ref=x` and `page` as the same URL and drop tracking/query parameters. Turn off to keep every unique query-string variant.

## `fromYear` (type: `integer`):

Only consider captures from this year onward.

## `toYear` (type: `integer`):

Only consider captures up to this year.

## `includeUrlPatterns` (type: `array`):

If set, keep only URLs matching at least one of these regular expressions, e.g. `/blog/`.

## `excludeUrlPatterns` (type: `array`):

Skip URLs matching any of these regular expressions. Leave empty to use sensible defaults (static assets, wp-json, tracking parameters...).

## `liveCheckConcurrency` (type: `integer`):

How many live-status requests run in parallel against the target site. Keep it low (2–6) to be polite and avoid HTTP 429 rate limiting.

## Actor input object example

```json
{
  "domains": [
    "example.com",
    "example.org/blog"
  ],
  "matchType": "host",
  "maxUrlsPerDomain": 50,
  "checkLiveStatus": true,
  "outputOnlyLost": false,
  "extractArchivedContent": "none",
  "maxContentExtractions": 50,
  "maxTextChars": 3000,
  "onlyHtml": true,
  "onlySuccessfulCaptures": true,
  "ignoreQueryStrings": true,
  "liveCheckConcurrency": 4
}
```

# Actor output Schema

## `results` (type: `string`):

All archived URLs with first-seen date, snapshot links, live HTTP status, category and redirect suggestion.

## `lostPages` (type: `string`):

Lost / soft-404 pages with suggested redirect and content recovered from the Wayback Machine.

## `redirectMap` (type: `string`):

CSV with columns from,to,score,status,category for every lost or homepage-redirected URL.

## `summary` (type: `string`):

Per-domain counts of archived URLs, filtered URLs and live-status categories.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "domains": [
        "paulgraham.com"
    ],
    "maxUrlsPerDomain": 50
};

// Run the Actor and wait for it to finish
const run = await client.actor("pandabear/wayback-lost-url-finder").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "domains": ["paulgraham.com"],
    "maxUrlsPerDomain": 50,
}

# Run the Actor and wait for it to finish
run = client.actor("pandabear/wayback-lost-url-finder").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "domains": [
    "paulgraham.com"
  ],
  "maxUrlsPerDomain": 50
}' |
apify call pandabear/wayback-lost-url-finder --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,pandabear/wayback-lost-url-finder"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/5m2bdNKFmYmwGAXX5/builds/yldHoIG8HJGdIFPHM/openapi.json