Internet Archive Snapshot Finder
Pricing
from $2.00 / 1,000 results
Internet Archive Snapshot Finder
Lists the Internet Archive captures of any URL, path or domain: date, status code, type and a link to each archived copy. Newest first for a single page, with date, status and duplicate filters. Pay per snapshot found.
Pricing
from $2.00 / 1,000 results
Rating
0.0
(0)
Developer
Derek Derringer
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
6 days ago
Last modified
Categories
Share
What it does
This Actor lists the captures the Internet Archive holds for a web address. Give it one address or a list. For each capture you get the date and time, the HTTP status the page answered with, the content type, a content fingerprint and a direct link to the archived copy.
It searches one exact address, every page under a path, every page on a host, or a whole domain with its subdomains. You can keep only working pages (2xx), only redirects or only errors, limit the dates, keep one capture per day, month or year, drop captures whose content did not change, or keep one capture per address.
Common uses:
- Find a live copy of a dead page. Exact address, newest first, status 2xx, maximum 1.
- List the pages a site had. Domain search, one per URL, web pages only. Raise Maximum snapshots per URL from its default of 100 (up to 10,000 addresses per line).
- See how often a page changed. Exact address, unique content only.
It reads only the archive's public capture index. It never downloads the archived pages, never logs in and contacts no other site.
How to use it
- Open the Actor in Apify Console.
- Put one address or domain per line in URLs or domains, for example
example.comorhttps://example.com/old-page. - Choose a Match type, and any filters you want.
- Click Start.
- When the run ends, open the Output tab. The Snapshots view shows each capture with a link to the archived copy. Export as CSV, JSON or Excel.
A line ending in /* is searched as a prefix, and a line starting with *. as a whole domain, whatever Match type says.
Input
| Field | What it does | Default | Limits |
|---|---|---|---|
URLs or domains (urls) | The addresses to search, one per line. | None. Prefilled with https://apify.com/ | http and https addresses or bare domain names. Up to 1,000 lines. Lines the archive treats as the same search are searched once |
Match type (matchType) | exact: that one address. prefix: every page under the path. host: every page on the host. domain: the domain and its subdomains. | exact | |
Order (order) | newest or oldest first, for exact addresses. | newest | See the note below |
Maximum snapshots per URL (maxSnapshotsPerUrl) | The most snapshots saved for each line. | 100 | 1 to 10,000 |
From date (fromDate) | Only captures on or after this date: 2019, 2019-06 or 2019-06-30. | None | |
To date (toDate) | Only captures on or before this date. A year or month includes all of it. | None | |
Status code (statusFilter) | any, 2xx, 3xx, 4xx or 5xx. | any | |
Web pages only (htmlOnly) | Keep only text/html captures. | false | |
Collapse (collapse) | none, day, month, year (first capture of each period), content (drop unchanged captures) or url (one capture per address). | none |
About order. The archive's index is sorted by address, then by time. For an exact address, newest first gives the most recent captures. Prefix, host and domain searches always come back in the archive's own order: address by address, oldest first within each address. To get recent captures of a whole site, set From date. Collapse works on captures that sit next to each other in that order.
About duplicate lines. The archive treats http and https, a leading www., upper and lower case, and a trailing slash on an exact address as the same address. Lines that differ only in those ways are searched once; the others are listed in SKIPPED as duplicate. Lines that overlap without being the same, such as a prefix inside a domain, are separate searches, so a capture can appear once for each of them, and each is charged.
A date that cannot be read (including a day that does not exist), or a From date after the To date, stops the run before any search, so nothing is charged for results you did not ask for.
Output
One dataset item per snapshot:
| Field | What it holds |
|---|---|
inputUrl, matchType | The line from your input, as it was searched, and how it was matched |
originalUrl | The address the archive captured. In prefix, host and domain searches, the page found |
archiveUrl | Link to the archived copy |
timestamp, capturedAt | The capture time as the archive writes it (14 digits, UTC) and in ISO 8601 |
statusCode | The HTTP status the page answered with when captured. Empty when the archive did not record one |
mimeType | The content type. warc/revisit means the content was the same as an earlier capture |
contentDigest | The archive's fingerprint of the content. Same digest, same content |
recordLengthBytes | Size of the compressed archive record, not of the page itself |
urlKey | The archive's sort key for the address |
Example row from a test run, the newest working capture of a page:
Pricing
This Actor is priced per event, with two charges:
- Actor start: Apify's automatic start event, $0.0001 per start (one event per GB of run memory, minimum one; this Actor runs at 256 MB, so one per run).
- Snapshot: one charge for each snapshot written to the dataset.
Price per snapshot: $2.00 per 1,000, which is $0.002 each.
These are never charged: lines with no captures, lines the archive could not answer, lines not searched, exact duplicate index rows, rows without a usable time or address, and the REPORT and SKIPPED records.
Each search asks the archive for no more rows than the run's remaining budget can pay for. If you set a maximum cost for the run and it is reached, the Actor stops, lists the rest in SKIPPED as spending_limit_reached, and ends the run without an error.
Records
| Record | What it holds |
|---|---|
| REPORT | Why the run stopped, the settings used, snapshots saved per line, whether more may exist, duplicates dropped, requests made, retries and waits |
| SKIPPED | Every line that gave no snapshots, was not searched, or was cut short, with the reason: no_snapshots, invalid_url, duplicate, archive_error, archive_unavailable, rate_limited, spending_limit_reached and others. Only the snapshots in the dataset are charged; a line cut short keeps the snapshots it already saved |
How it treats the archive
- The archive's robots.txt is read first and obeyed. If it cannot be read, nothing is searched.
- One request at a time, at least one second apart.
- If the archive answers 429 or a server error, the Actor waits (as long as the archive asks, up to five minutes) and tries again, up to three times. If the archive keeps answering 429, or asks for a wait longer than five minutes, the run stops asking and lists the remaining lines in SKIPPED. After three lines in a row the archive could not answer, or answered with something unreadable, the run stops.
- No cookie, login or proxy. The user agent starts with
internet-archive-snapshot-finder/0.1.
Your responsibility
This Actor is not affiliated with or endorsed by the Internet Archive. The Internet Archive's Terms of Use say access to its collections is for scholarship and research purposes. You are responsible for following those terms and the law in how you use the results and the archived pages they link to.
Limits
| Limit | Value |
|---|---|
| Lines per run | 1 to 1,000 |
| Snapshots per line | 1 to 10,000 |
| Search time | Exact addresses usually take 1 to 5 seconds. Prefix, host and domain searches can take 10 seconds to 2 minutes each, longer when the archive is busy |
| Memory | 256 MB |
