Wayback Machine URL Exporter - CDX API

Export every archived URL of a domain, subdomain or path from the Internet Archive Wayback Machine CDX API. Deduplicate by URL or content, filter by HTTP status, MIME type, date range and substring. Great for SEO migrations, broken-link recovery and OSINT.

What does this Actor do?

The Internet Archive's Wayback Machine has captured hundreds of billions of web pages. Its CDX API can list every URL it ever saw for a domain - including pages that no longer exist. This Actor turns that API into a clean, deduplicated dataset you can filter by HTTP status, MIME type, date range and substring, with direct links to each archived snapshot. It is the online, no-install version of tools like waybackurls and gau.

Key features:

• Plain HTTP against official public endpoints - no browser, no proxy, no login, runs on 128 MB memory.
• Pay only for results: one charge per row saved; your maximum charge per run is always respected.
• maxItems hard cap (default 100) and bounded pagination - runs can never loop forever.
• Incremental mode via a since-date so scheduled runs only fetch new data.
• Polite rate limiting, retries with exponential backoff and Retry-After support.
• Clean, flat, consistently named JSON fields ready for Excel, Google Sheets, BigQuery or your API.

Who is it for?

• SEO site migrations - recover every historical URL so you can build complete 301 redirect maps.
• Broken backlink recovery - find old pages that still have links pointing to them and restore or redirect them.
• Content recovery - list old blog posts, PDFs or images and open the archived copy.
• Security research & bug bounty recon - enumerate forgotten endpoints, parameters and files (in-scope targets only).
• Competitive research - see how a competitor's site structure and product pages changed over time.

How it works

The Actor builds a CDX query with server-side filters (status, MIME type, URL substring, date range) and server-side collapsing, then pages through results using CDX resume keys. Requests are throttled (1.5 s default) with retries and long timeouts, because archive.org is a non-profit and rate-limits heavy users. A local seen-set guarantees no duplicate rows.

Input

Field	Type	Default	Description
targets	array	-	e.g. example.com, blog.example.com, example.com/docs/
matchType	string	"domain"	domain = host + all subdomains; host = exact host; prefix = URL prefix; exact = single URL.
maxItems	integer	100	Hard cap on rows saved and charged.
dedupe	string	"urlkey"	urlkey = one row per unique URL (first capture in range); digest = one row per unique content; none = every capture.
statusCodes	array	["200"]	Keep only these (e.g. 200, 301, 404). Empty = all.
mimeTypes	array	[]	Keep only these (e.g. text/html, application/pdf). Empty = all.
excludeMimeTypes	array	[]	e.g. image/jpeg, text/css
fromDate	string	-	YYYY or YYYY-MM-DD. Use your last run date for incremental exports.
toDate	string	-	YYYY or YYYY-MM-DD.
urlContains	string	""	Keep only URLs containing this text (server-side filter).
urlExcludes	array	[]	Drop URLs containing any of these strings.
pageSize	integer	2000	Rows per CDX request.
delayMs	integer	1500	archive.org throttles aggressive clients; keep >= 1000.

Example input

{
  "targets": [
    "example.com"
  ],
  "matchType": "domain",
  "statusCodes": [
    "200"
  ],
  "mimeTypes": [
    "text/html",
    "application/pdf"
  ],
  "fromDate": "2018",
  "toDate": "2026-10-01",
  "dedupe": "urlkey",
  "maxItems": 5000
}

Output

Each result is one row in the default dataset. You can download it as JSON, CSV, Excel, XML or HTML, or read it through the Apify API. A real row from a test run on 26 October 2026:

{
  "target": "apify.com",
  "url": "https://apify.com/",
  "urlKey": "com,apify)/",
  "timestamp": "20240106234848",
  "capturedAt": "2024-01-06T23:48:48Z",
  "mimeType": "text/html",
  "statusCode": 200,
  "digest": "CI23EQLBQNE7EWSZRZT6HLVWOEIDTFE3",
  "lengthBytes": 52534,
  "archiveUrl": "https://web.archive.org/web/20240106234848/https://apify.com/",
  "rawArchiveUrl": "https://web.archive.org/web/20240106234848id_/https://apify.com/"
}

Tips

• Use matchType: domain to include all subdomains; prefix to export one folder.
• Set statusCodes to empty to also get redirects (301/302) and 404s.
• For incremental runs set fromDate to your previous run date.
• rawArchiveUrl (the id_ form) returns the original bytes without the Wayback toolbar.

Pricing

This Actor uses pay-per-event pricing: about $0.30 per 1,000 results. There is no monthly rental fee. Platform usage is minimal because the Actor runs on 128 MB with plain HTTP requests. Set Maximum cost per run in the run options and the Actor stops cleanly as soon as the limit is reached - you are never charged beyond it.

Scheduling and integrations

Create an Apify Schedule (for example daily at 08:00) and pass a since-date to get only new items each time. Connect the dataset to Google Sheets, Slack, Zapier, Make, n8n or a webhook to receive results automatically, or call the Actor from Python or JavaScript using the Apify API client.

Limitations

Very large domains can have millions of captures; set maxItems sensibly. The archive is occasionally slow or returns 503 - the Actor retries, but extremely large exports are best split by fromDate/toDate.

FAQ

Do I need an account or API key on the source site? No. Only public, unauthenticated endpoints are used.

Do I need proxies? No. The Actor makes a small number of well-spaced requests to official APIs, so datacenter IPs work fine.

Why did my run return fewer items than maxItems? The source simply had no more items matching your filters and date range, or your maximum cost per run was reached.

Is this legal? The Actor only reads publicly available data through documented public endpoints. You are responsible for using the data in line with the source's terms and applicable laws such as GDPR, especially for personal data.

Something broke? Open an issue on the Actor's Issues tab with your run ID and we will fix it quickly.