Web Archive URL Scraper

Web Archive URL Scraper pulls structured data from Internet Archive & Common Crawl and returns it as a clean dataset you can export as JSON, CSV or Excel. It queries the free Wayback Machine and Common Crawl web indexes and returns one row per archived URL for a domain, with capture timestamp, HTTP status, content type, size and digest, useful for SEO URL-history research, redirect and dead-link audits.

No API key, no login. Click Start and it works with the prefilled input.

What does Web Archive URL Scraper do?

It queries Internet Archive & Common Crawl and returns one row per record, with only the fields that matter. Runs on the Apify platform, so you get an API endpoint, scheduling, monitoring and integrations (Zapier, Make, Google Sheets, webhooks) without maintaining any infrastructure.

Why use it?

• Internet Archive & Common Crawl data without the plumbing: no proxy setup, no pagination code, no scraping maintenance.
• Schedule it to monitor changes daily or weekly and trigger alerts on new records.
• Feed it straight into a spreadsheet, database, or an AI agent via the Apify API or MCP server.
• Runs in seconds and returns a typed dataset that is ready for analysis.

How to use it

1. Click Try for free / Start.
2. Leave the default input to see a sample run, or type your own values (see Input below).
3. Press Start and wait a few seconds.
4. Open the Output tab and download as JSON, CSV, Excel or HTML.

Input

Field	Type	Default	Description
domain	string	"example.com"	Domain to look up, e.g. wikipedia.org. A full URL also works.
useWayback	boolean	true	Query the Internet Archive Wayback Machine CDX index.
useCommonCrawl	boolean	true	Query the Common Crawl index as well. Skipped silently if it is unavailable.
maxResults	integer	25	Maximum number of archived URLs to return.

Example input:

{
  "domain": "example.com",
  "maxResults": 5
}

Output

One item per record. Sample:

{
  "source": "wayback",
  "url": "http://www.example.com:80/",
  "domain": "example.com",
  "timestamp": "20020120142510",
  "status": 200,
  "mime": "text/html"
}

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel.

Data fields

Field	Type	Meaning
source	string	Source
url	string	Archived URL
domain	string	Domain
timestamp	string	Capture Timestamp
status	integer	HTTP Status
mime	string	MIME Type
digest	string	Digest
content_length	integer	Content Length
languages	string	Languages
warc_file	string	WARC File

How much does it cost to run?

This Actor uses pay-per-event pricing: you pay per result item pushed to the dataset, and nothing for an empty run. Each run is a few seconds of compute, so typical costs are fractions of a cent per run, and the Apify free tier covers plenty of testing.

Tips and advanced options

• Raise maxResults for a bigger pull; the Actor pages through the source for you.
• Schedule it (e.g. every morning) and combine with a webhook to get only the new records.
• Add the Actor to an AI agent through the Apify MCP server for natural-language queries.

FAQ, disclaimers and support

• Is this legal? It uses publicly available data from Internet Archive & Common Crawl. You are responsible for how you use the data and for complying with the source's terms.
• It returned nothing? Enter a bare domain such as "example.com" or "wikipedia.org".
• Something broken? Open an issue on the Actor's Issues tab. Data sources change and get fixed quickly.
• Need a custom variant (extra fields, a different source, a scheduled report)? Get in touch through the Issues tab.