# Wayback Machine URL Extractor - Archived URLs (`logiover/wayback-machine-url-extractor`) Actor Extract every archived URL of any domain from the Internet Archive's Wayback Machine (CDX API). Recover lost or old pages, build redirect maps and run OSINT, with date and status filters. No API key, export to CSV or JSON. - **URL**: https://apify.com/logiover/wayback-machine-url-extractor.md - **Developed by:** [Logiover](https://apify.com/logiover) (community) - **Categories:** SEO tools, Developer tools - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $3.50 / 1,000 results This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Wayback Machine URL Extractor 🕰️ — Archived URLs from the Internet Archive **Recover every historical URL a website has ever published — straight from the Internet Archive's Wayback Machine.** This **Wayback Machine scraper** queries the public **CDX API** to extract **archived URLs** and **historical URLs** for any domain — including pages that were deleted, renamed, or lost in a migration. Feed in one domain and get back up to tens of thousands of unique URLs, each with its capture date, archived HTTP status, MIME type, and a direct Wayback snapshot link. Point it at one domain and it pulls the full historical URL inventory automatically. **No API key, no login, no rate-limit headaches — one row per archived URL.** > Looking to **recover old URLs** after a site migration, build a redirect map, find old/deleted pages, do **OSINT** on a domain's history, or pull a list of **Internet Archive URLs** without writing CDX queries by hand? This is the **Internet Archive URL extractor** that does it at scale. --- ### ✨ Key features - 🕰️ **Full historical URL inventory** — pulls every unique URL the Wayback Machine has on record for a domain, going back to 1996. - 🔑 **No API key required** — uses the open Internet Archive **CDX API**; no auth, no token, no login. - 🌐 **Subdomain & path matching** — capture the host plus all subdomains and paths, or narrow down to a single host or path prefix. - 📅 **Date-range filtering** — restrict to snapshots captured between two dates (`fromDate` / `toDate`). - ✅ **Status-code filtering** — keep only `200 OK` captures and drop dead/redirected ones. - 🔗 **Direct snapshot links** — every row includes a ready-to-open `web.archive.org/web/...` URL. - 🌊 **Streamed pagination** — pages through massive result sets with the CDX `resumeKey` mechanism, so memory stays flat even on 100k+ URL domains. - 🔢 **Result caps** — set `maxResults` per domain, or `0` for unlimited. - 📋 **Multiple domains per run** — process a whole list in one go. - 📤 **Export-ready** — JSON, CSV, and Excel output via the Apify Dataset or REST API. ### 💡 Use cases - **SEO migration & redirect maps** — recover lost/old URLs after a site move and rebuild a complete **301 redirect map** so you don't lose link equity. - **Content recovery** — find and restore blog posts, product pages, or docs that were deleted but still live in the archive. - **OSINT & research** — enumerate a target domain's historical footprint, old endpoints, removed pages, and forgotten subdomains. - **Link reclamation** — find old URLs that still earn backlinks so you can redirect them and reclaim the link value. - **Finding old endpoints** — surface admin paths, legacy APIs, and orphaned pages that no longer appear on the live site. - **Competitive & web-archaeology research** — reconstruct how a competitor's URL structure and content changed across years of snapshots. - **Datasets** — build a domain's URL/MIME/capture-history dataset for analysis. ### 📦 What you get One row per unique archived URL, including: | Field | Description | |-------|-------------| | `domain` | The normalized domain this URL belongs to | | `url` | The original archived URL | | `timestamp` | Raw 14-digit Wayback capture timestamp (`YYYYMMDDhhmmss`) | | `capturedAt` | ISO 8601 form of the capture timestamp | | `statusCode` | HTTP status the archive recorded for that capture (e.g. `200`, `301`, `404`, or `-`) | | `mimeType` | Content type recorded at capture time (e.g. `text/html`) | | `digest` | Wayback content digest (used internally for de-duplication) | | `snapshotUrl` | Direct link to the archived snapshot on `web.archive.org` | #### Example output ```json { "domain": "nasa.gov", "url": "http://www.nasa.gov/mission_pages/station/main/index.html", "timestamp": "20120114043915", "capturedAt": "2012-01-14T04:39:15.000Z", "statusCode": "200", "mimeType": "text/html", "digest": "AB23CD45EF67GH89IJ01KL23MN45OP67", "snapshotUrl": "https://web.archive.org/web/20120114043915/http://www.nasa.gov/mission_pages/station/main/index.html" } ```` ### 🚀 How to use it 1. Click **Try for free / Start**. 2. Add one or more domains to **Domains** (e.g. `nasa.gov`, `bbc.com`). URLs and `www.` are normalized automatically. 3. (Optional) Pick a **matchType**, set a **date range**, filter by **status code**, or raise **maxResults** (`0` = unlimited). 4. Click **Save & Start**. 5. Export the archived URL list as **JSON, CSV, Excel or via API**, and open any row's `snapshotUrl` to view the archived page. ### ⚙️ Input | Field | Type | Description | Default | |-------|------|-------------|---------| | `domains` | array | **Required.** One or more domains or URLs (e.g. `nasa.gov`, `bbc.com`). Wildcards added automatically | – | | `matchType` | enum | `subdomains` (host + all subdomains + paths), `host` (exact host only), `domain` (host + subdomains), `prefix` (path prefix) | `subdomains` | | `fromDate` | string | Optional `YYYYMMDD` lower bound on capture date | – | | `toDate` | string | Optional `YYYYMMDD` upper bound on capture date | – | | `filterStatus` | string | Optional — only return captures with this HTTP status (e.g. `200`) | – | | `maxResults` | integer | Max unique URLs per domain. `0` = unlimited | `5000` | | `proxyConfiguration` | object | Proxy settings. Defaults to Apify Proxy | Apify Proxy | #### Example input ```json { "domains": ["nasa.gov"], "matchType": "subdomains", "fromDate": "20100101", "toDate": "20201231", "filterStatus": "200", "maxResults": 5000, "proxyConfiguration": { "useApifyProxy": true } } ``` ### 🔍 How it works 1. Each domain you provide is normalized — scheme, `www.`, paths and wildcards are stripped down to a bare host. 2. A **CDX API** query is built from your `matchType`, date range, and status filter, requesting the `original`, `timestamp`, `statuscode`, `mimetype` and `digest` fields with `collapse=urlkey` so each URL appears **only once** instead of returning every capture of it. 3. Results are paged using the CDX `showResumeKey` / `resumeKey` mechanism, and each page is pushed to the dataset in a batch — so even domains with hundreds of thousands of archived URLs stream out without exhausting memory. 4. For every row, a direct `snapshotUrl` is constructed in the `https://web.archive.org/web//` form, so you can open the exact archived page. 5. Slow responses, `5xx`, and `429` errors are retried with exponential backoff on a fresh proxy IP — the CDX index can be slow, so retries keep large runs reliable. ### 🧰 Tips & best practices - Big domains (news sites, government sites) can have **hundreds of thousands** of archived URLs. Start with the default `maxResults` of `5000` to gauge volume, then raise it or set `0` for everything. - Use `filterStatus: "200"` to skip dead and redirected captures and keep only pages that actually resolved — ideal for building redirect maps. - Narrow with `fromDate` / `toDate` (both `YYYYMMDD`) when you only care about a specific era of the site. - Use `matchType: "subdomains"` to sweep every subdomain at once, or `host` for a single host without its subdomains. - Sort or filter the dataset by `mimeType` to isolate just HTML pages, images, PDFs, etc. ### ❓ FAQ #### How do I get all URLs of a website from the Wayback Machine? Add the domain to **Domains**, leave `matchType` on `subdomains`, set `maxResults` to `0` for everything, and run it. The actor queries the Internet Archive CDX API and returns one row per unique archived URL. #### Can I find old or deleted pages of a domain? Yes — that's the core use case. The Wayback Machine keeps URLs even after they're removed from the live site, so deleted blog posts, retired product pages, and old endpoints all show up in the results with a `snapshotUrl` to view them. #### How do I export archived URLs to CSV or JSON? Run the actor, then download the dataset as **CSV, JSON or Excel** (or pull it via the REST API). Every archived URL is one row, so it drops straight into a spreadsheet or pipeline. #### Is this free and without an API key? The Internet Archive CDX API is public and requires **no API key and no login**. You only pay for the Apify platform usage of the run itself. #### Can I filter by date or status code? Yes — set `fromDate` / `toDate` (`YYYYMMDD`) to restrict to a capture window, and `filterStatus` (e.g. `200`) to keep only captures with a specific HTTP status. #### How many URLs can it return? Up to tens of thousands per domain — set `maxResults` to `0` for unlimited. Results stream to the dataset in pages via the CDX `resumeKey`, so even 100k+ URL domains run without memory issues. #### Why are some `statusCode` values `-`? The Wayback index sometimes records captures without a stored status code (e.g. revisit records). Those rows are still valid archived URLs. ### 🔗 Related actors by the same author - **[Sitemap to URL Crawler](https://apify.com/logiover/sitemap-to-url-crawler)** — extract all URLs from any sitemap.xml. - **[Website SEO Audit Crawler](https://apify.com/logiover/website-seo-audit-crawler)** — run a full on-page SEO audit across a whole site. - **[Bulk URL Status Checker](https://apify.com/logiover/bulk-url-status-checker)** — check HTTP status codes for a list of URLs in bulk. - **[Broken Link Checker](https://apify.com/logiover/broken-link-checker)** — crawl a site and find dead links with HTTP status codes. ### 📝 Changelog #### 2026-06-15 - Initial release — extract archived URLs from the Wayback Machine CDX API with date/status filters, CSV/JSON export, no API key. # Actor input Schema ## `domains` (type: `array`): One or more domains (or full URLs) to extract archived URLs for. Examples: nasa.gov, bbc.com, https://example.com. Do not include trailing wildcards. ## `matchType` (type: `string`): How the domain is matched against the archive. 'subdomains' = the host and all of its subdomains and paths (broadest, default). 'host' = only the exact host (no subdomains). 'domain' = exact host only. 'prefix' = URLs that start with the given path prefix. ## `fromDate` (type: `string`): Optional. Only return snapshots captured on or after this date. Format: YYYYMMDD (e.g. 20100101). Leave empty for no lower bound. ## `toDate` (type: `string`): Optional. Only return snapshots captured on or before this date. Format: YYYYMMDD (e.g. 20201231). Leave empty for no upper bound. ## `filterStatus` (type: `string`): Optional. Only return URLs whose archived snapshot returned this HTTP status code (e.g. 200 to exclude dead/redirected captures). Leave empty for all. ## `maxResults` (type: `integer`): Maximum number of unique URLs to extract per domain. Use 0 for unlimited (warning: large sites can yield hundreds of thousands of URLs). ## `proxyConfiguration` (type: `object`): Recommended to avoid rate limiting from the Internet Archive. ## Actor input object example ```json { "domains": [ "nasa.gov" ], "matchType": "subdomains", "maxResults": 5000, "proxyConfiguration": { "useApifyProxy": true } } ``` # Actor output Schema ## `results` (type: `string`): The dataset containing every archived URL with its capture timestamp, HTTP status, MIME type and Wayback snapshot link. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "domains": [ "nasa.gov" ] }; // Run the Actor and wait for it to finish const run = await client.actor("logiover/wayback-machine-url-extractor").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "domains": ["nasa.gov"] } # Run the Actor and wait for it to finish run = client.actor("logiover/wayback-machine-url-extractor").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "domains": [ "nasa.gov" ] }' | apify call logiover/wayback-machine-url-extractor --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=logiover/wayback-machine-url-extractor", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Wayback Machine URL Extractor - Archived URLs", "description": "Extract every archived URL of any domain from the Internet Archive's Wayback Machine (CDX API). Recover lost or old pages, build redirect maps and run OSINT, with date and status filters. No API key, export to CSV or JSON.", "version": "1.0", "x-build-id": "NOT2n5L3EBwYQc2UU" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/logiover~wayback-machine-url-extractor/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-logiover-wayback-machine-url-extractor", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/logiover~wayback-machine-url-extractor/runs": { "post": { "operationId": "runs-sync-logiover-wayback-machine-url-extractor", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/logiover~wayback-machine-url-extractor/run-sync": { "post": { "operationId": "run-sync-logiover-wayback-machine-url-extractor", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "domains" ], "properties": { "domains": { "title": "Domains", "type": "array", "description": "One or more domains (or full URLs) to extract archived URLs for. Examples: nasa.gov, bbc.com, https://example.com. Do not include trailing wildcards.", "items": { "type": "string" } }, "matchType": { "title": "Match Type", "enum": [ "subdomains", "host", "domain", "prefix" ], "type": "string", "description": "How the domain is matched against the archive. 'subdomains' = the host and all of its subdomains and paths (broadest, default). 'host' = only the exact host (no subdomains). 'domain' = exact host only. 'prefix' = URLs that start with the given path prefix.", "default": "subdomains" }, "fromDate": { "title": "From Date (YYYYMMDD)", "type": "string", "description": "Optional. Only return snapshots captured on or after this date. Format: YYYYMMDD (e.g. 20100101). Leave empty for no lower bound." }, "toDate": { "title": "To Date (YYYYMMDD)", "type": "string", "description": "Optional. Only return snapshots captured on or before this date. Format: YYYYMMDD (e.g. 20201231). Leave empty for no upper bound." }, "filterStatus": { "title": "HTTP Status Filter", "type": "string", "description": "Optional. Only return URLs whose archived snapshot returned this HTTP status code (e.g. 200 to exclude dead/redirected captures). Leave empty for all." }, "maxResults": { "title": "Max Results Per Domain", "minimum": 0, "type": "integer", "description": "Maximum number of unique URLs to extract per domain. Use 0 for unlimited (warning: large sites can yield hundreds of thousands of URLs).", "default": 5000 }, "proxyConfiguration": { "title": "Proxy Configuration", "type": "object", "description": "Recommended to avoid rate limiting from the Internet Archive.", "default": { "useApifyProxy": true } } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```