Wayback Machine Scraper: Archived URLs, Snapshots, History & Page Text

Get Internet Archive Wayback Machine data as clean JSON, CSV or Excel through the official Wayback CDX Server API: every archived URL of a website (a Wayback URL extractor), the full snapshot history of any page (optionally only the captures where the content changed), a one-row archive summary per URL (first and last capture, years archived, snapshot closest to a date), and the archived page text or Markdown of any capture.

No browser, no login, no proxy. $0.50 per 1,000 records, archived page content $2 per 1,000 pages.

What you can use it for

• SEO and site migrations: list every URL a domain ever had to find lost pages, old blog posts and backlinks targets, and build 301 redirect maps.
• Recover deleted content: get the text or Markdown of pages that no longer exist on the live site.
• Competitor research: see how a competitor's pricing, homepage or product pages changed over time (only content changes, one per month, etc.).
• Domain due diligence: before buying an expired domain, check when it was first archived, how long it has been online and what it used to host.
• Compliance and legal evidence: timestamped Wayback links of what a page showed on a given date.
• Security and OSINT on your own assets: forgotten endpoints, old files and PDFs under a domain.
• AI and RAG: historical page text as clean Markdown for LLM pipelines or dataset building.

How to use it

1. Choose a mode: urls (all archived URLs of a site), snapshots (capture history of pages) or summary (one row per URL).
2. Add targets: example.com, https://example.com/pricing, example.com/blog/* or a web.archive.org/web/... link.
3. Optional filters: date range, status codes (e.g. 200, 3xx), MIME types (e.g. text/html, application/pdf), URL regex.
4. Optional: turn on Fetch archived page content to get the title, meta description, H1, word count and text/Markdown.
5. Click Start, or schedule it with Only new to get alerts about newly archived pages or captures.

Why this Wayback Machine scraper

• Official API only: the Internet Archive's public Wayback CDX Server and Availability APIs. Nothing breaks when the Wayback website changes.
• Three tools in one: URL extractor, snapshot history and archive summary, each with date, status, MIME and regex filters.
• Only content changes: collapse: digest returns a capture only when the page content changed. Or limit to one capture per hour, day, month or year.
• Clean page content: archived HTML is fetched in its original form (no Wayback toolbar), scripts and styles removed, as text, Markdown or HTML.
• No 10,000 cap: large domains are paged with the CDX resume key, so you can export hundreds of thousands of URLs.
• Polite and reliable: one request at a time, automatic back-off when the archive asks to slow down.
• Cheap and fair: $0.50 per 1,000 records. Targets that were never archived, already-seen items and failed content fetches are free. 256 MB memory.

Modes

Mode	One row per	Typical use
urls (default)	unique archived URL, with its first capture	URL extractor, lost pages, redirect maps
snapshots	capture of a page (exact URL or prefix)	change history, price/page tracking, evidence
summary	target	first/last capture, years archived, closest snapshot to a date

Input examples

Every archived HTML page of a domain (incl. subdomains) that returned 200:

{ "mode": "urls", "targets": ["example.com"], "statusCodes": ["200"], "mimeTypes": ["text/html"], "maxItemsPerTarget": 10000 }

All archived PDFs under a path:

{ "mode": "urls", "targets": ["example.com/docs/*"], "urlFilter": ".*\\.pdf$", "excludeAssets": false }

A competitor's pricing page, only when it changed, with the page text:

{ "mode": "snapshots", "targets": ["https://www.example.com/pricing"], "collapse": "digest", "statusCodes": ["200"], "fetchContent": true, "contentFormat": "markdown" }

The latest 10 captures of a homepage:

{ "mode": "snapshots", "targets": ["https://example.com/"], "order": "newest", "maxItemsPerTarget": 10 }

Archive summary for a list of domains, plus the snapshot closest to 1 Jan 2020:

{ "mode": "summary", "targets": ["example.com", "apify.com", "python.org"], "closestDate": "2020-01-01" }

Weekly alert of newly archived URLs on a site (schedule it):

{ "mode": "urls", "targets": ["example.com"], "onlyNew": true, "stateKey": "example-weekly" }

Output

urls / snapshots row:

{
  "type": "snapshot",
  "target": "https://www.python.org/",
  "url": "https://www.python.org/",
  "timestamp": "20200101011950",
  "capturedAt": "2020-01-01T01:19:50Z",
  "statusCode": 200,
  "mimeType": "text/html",
  "lengthBytes": 9999,
  "digest": "XMMYQPPTNAC3U4V5J6HCSU6TOMUUYBQK",
  "archiveUrl": "https://web.archive.org/web/20200101011950/https://www.python.org/",
  "rawArchiveUrl": "https://web.archive.org/web/20200101011950id_/https://www.python.org/",
  "urlKey": "org,python)/",
  "title": "Welcome to Python.org",
  "metaDescription": "The official home of the Python Programming Language",
  "h1": null,
  "wordCount": 1021,
  "content": "Python is a programming language that lets you work quickly and integrate systems more effectively...",
  "scrapedAt": "2026-10-23T08:00:00Z"
}

summary row:

{
  "type": "summary",
  "target": "python.org",
  "archived": true,
  "firstCapturedAt": "1997-05-01T01:16:26Z",
  "lastCapturedAt": "2026-10-23T05:46:27Z",
  "lastStatusCode": 200,
  "yearsCaptured": [1997, 1998, 1999, "..."],
  "yearsCount": 30,
  "archiveSpanDays": 10767,
  "closestCapturedAt": "2020-01-01T23:40:54Z",
  "closestArchiveUrl": "https://web.archive.org/web/20200101234054/https://www.python.org/",
  "calendarUrl": "https://web.archive.org/web/*/python.org"
}

Content fields (title, metaDescription, canonicalUrl, h1, language, linksCount, wordCount, content, contentTruncated) appear only with Fetch archived page content. A run summary (records, content pages, not-archived targets, API requests) is saved as OUTPUT in the key-value store.

Pricing

Pay per event, no subscription:

Event	Price
Archive record (URL, snapshot or summary)	$0.0005 ($0.50 per 1,000)
Archived page content (text/Markdown)	$0.002 ($2 per 1,000)
Actor start	$0.0003

Examples: 10,000 archived URLs of a domain is about $5. A year of monthly pricing-page snapshots with text is 12 x $0.0025 = $0.03. Set Max cost per run in the run options to cap spend; the Actor stops cleanly at the limit.

FAQ

Is it legal to use the Wayback Machine API? The CDX Server and Availability APIs are public interfaces the Internet Archive provides for programmatic access. This Actor uses them politely (one request at a time, backs off on HTTP 429). Archived pages are third-party content: respect copyright and the original sites' rights when you reuse text.

Does it collect personal data? No. It returns URL and capture metadata and, on request, the text of archived pages you choose. Use the content option responsibly and comply with privacy law (e.g. GDPR) if pages contain personal information.

Why are some URLs missing? Only what the Internet Archive captured is available, and site owners can ask for exclusion. Some pages are archived only as redirects (3xx); filter with statusCodes: ["200"] to get real pages.

How many URLs can I get for a big domain? The CDX API pages through results with a resume key, so there is no fixed cap; use Max records per target to control cost.

What does "Only content changed" do? It collapses consecutive captures with the same content hash (digest), so you only get captures where the page actually changed.

The archive is slow or returned 429. The Wayback Machine is a free public service with rate limits. The Actor waits and retries automatically; if it still fails, the affected target gets an error row (not charged) and you can rerun later.

Related tools by the same developer

• RSS Feed Scraper & Reader
• Web Page to Markdown for LLMs
• Website SEO Audit: SSL, DNS, Security Headers
• Broken Link Checker