Wayback Machine URLs Scraper - $2.70 per 1000 results

Wayback Machine URL scraper: get the list of all archived URLs of a domain from the Internet Archive CDX API (and optionally the Common Crawl index), unique by URL, with timestamp, HTTP status, MIME type and a direct archive link.

What does it do?

• Unique URLs per domain (collapse by URL), subdomains optional.
• Filter by year range and HTTP 200.
• Direct web.archive.org snapshot links.
• Optional Common Crawl index lookup.
• Polite: one request at a time, backoff on 429/5xx.

Use cases

• SEO: recover lost pages and old URLs for redirects.
• Content audits and site migrations.
• OSINT and security: discover forgotten endpoints.
• Research on a website's history.

Input

Field	Description
domains	Domains to look up, e.g. apify.com
sources	wayback = Internet Archive CDX; commoncrawl = Common Crawl index (can be slow).
includeSubdomains	Also list *.domain
fromYear	Example: 2020
toYear	Example: 2026
onlyStatus200	Skip redirects and errors
maxUrlsPerSource	Per domain and source
commonCrawlIndexes	Latest N crawls to query

Example input:

{}

Output

Real output (first items of a run on 26 Oct 2026):

[
  {
    "url": "http://www.apify.com:80/",
    "domain": "apify.com",
    "source": "wayback",
    "timestamp": "2007-05-31T10:15:38Z",
    "statusCode": 200,
    "mimeType": "text/html",
    "digest": "EE6FCHP3MKBC3EV5D5Q4WQJNZNVUTNU6",
    "lengthBytes": 1013,
    "archiveUrl": "https://web.archive.org/web/20070531101538/http://www.apify.com:80/",
    "crawl": null
  },
  {
    "url": "https://apify.com/.well-known/agent-skills/index.json",
    "domain": "apify.com",
    "source": "wayback",
    "timestamp": "2026-09-13T15:50:47Z",
    "statusCode": 200,
    "mimeType": "application/json",
    "digest": "MO4JIMUGYVNBCZRQD34VZCN5CUJIB6JB",
    "lengthBytes": 4225,
    "archiveUrl": "https://web.archive.org/web/20260913155047/https://apify.com/.well-known/agent-skills/index.json",
    "crawl": null
  }
]

Export the results as JSON, CSV or Excel, read them through the API, or send them to Google Sheets, Make, Zapier, n8n or webhooks. You can also run this Actor from AI agents through the Apify MCP server.

Pricing

Pay per event: $2.70 per 1,000 urls plus $0.00005 per run start. There is no extra platform usage charge.

FAQ

Do you download page content? No, only the URL index.

Why is Common Crawl off by default? Its public index is often slow or overloaded; enable it when you need it.

Limits? Up to 10,000 URLs per domain and source per run.

Do I need a login or API key? No. It uses the source's public API; you only need an Apify account.

Can AI agents use it? Yes. Call it through the Apify API, or from Claude, Cursor or any MCP client via the Apify MCP server.

Data source and license

Sources: Internet Archive Wayback Machine CDX API and Common Crawl index (index.commoncrawl.org), used under their terms of use. Not affiliated with the Internet Archive or Common Crawl.

This Actor is not affiliated with or endorsed by the data source. You are responsible for using the data in line with the source's license and terms.