# Link & Citation Verifier: Dead Links, Soft 404s, Quote Check (`tallowlanelabs/link-citation-verifier`) Actor

Check cited links really work and say what you claim: soft 404s, parked domains, blocked vs paywalled vs dead, fuzzy quote match with snippet, and a Wayback Machine copy for dead links. Built for AI agents and MCP. Not affiliated with the Internet Archive.

- **URL**: https://apify.com/tallowlanelabs/link-citation-verifier.md
- **Developed by:** [Tallow Lane Labs](https://apify.com/tallowlanelabs) (community)
- **Categories:** Developer tools, AI, SEO tools
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 url checkeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Link & Citation Verifier: Dead Links, Soft 404s, Quote Check

Give it a list of links, optionally with the **quote** or **title** each one is supposed to support. For every link you get back one flat JSON row:

- **Is the page really alive?** Final URL after redirects, HTTP status, and a clear `status`: `alive`, `soft_404`, `parked`, `dead`, `blocked`, `paywalled`, `rate_limited`, `server_error`, `timeout`, `unreachable` or `invalid_url`. A 200 page that says "Page not found", or a deep link that quietly redirects to the home page, is a **soft 404**, not "alive". A bot check is **blocked**, not "dead".
- **Does it say what you cite?** `quoteFound`, a 0-1 `quoteMatchScore`, the matching passage (`quoteMatchedText`) and its context. Small differences in case, accents, punctuation and spacing are ignored; paraphrases come back as `partial`.
- **Is there a copy if it is gone?** For dead links, the nearest **Wayback Machine** capture (`archiveUrl`), and whether your quote is on that copy.
- **One-word answer:** `verdict` = `supported`, `supported_by_archive`, `quote_not_found`, `dead_link`, `unverifiable`, ... so an agent can act without parsing anything else.

Built for AI agents, MCP clients and research pipelines that must check their sources before they answer: hallucinated or rotted citations are caught before anyone reads them.

*By Tallow Lane Labs. Unofficial tool; not affiliated with, endorsed by or sponsored by the Internet Archive (Wayback Machine) or any site it checks.*

### Quick start

Leave the input empty and press **Start**: it checks a 3-link demo (two real citations and one dead link) for about $0.0105.

Typical input:

```json
{
  "links": [
    { "url": "https://en.wikipedia.org/wiki/Link_rot",
      "quote": "is the phenomenon of hyperlinks tending over time to cease to point to their originally targeted file",
      "title": "Link rot" },
    { "url": "https://www.whitehouse.gov/ostp/ai-bill-of-rights/",
      "quote": "The Blueprint for an AI Bill of Rights is a guide for a society that protects all people from these threats",
      "citedDate": "2023-06-01", "id": "ref-7" },
    "https://example.com/some/page"
  ]
}
```

Only `url` is required. `quote` and `title` are optional. `citedDate` (YYYY-MM-DD) picks the Wayback capture nearest that date. `id` is echoed back. Plain URL strings work too, and so does the `urls` list.

### What you get (one row per link, in input order)

```json
{
  "inputIndex": 1,
  "id": "ref-7",
  "url": "https://www.whitehouse.gov/ostp/ai-bill-of-rights/",
  "finalUrl": "https://www.whitehouse.gov/ostp/ai-bill-of-rights/",
  "httpStatus": 404,
  "status": "dead",
  "isAlive": false,
  "statusReason": "http_404",
  "verdict": "supported_by_archive",
  "quote": "The Blueprint for an AI Bill of Rights is a guide for a society that protects all people from these threats",
  "quoteFound": null,
  "archiveFound": true,
  "archiveUrl": "https://web.archive.org/web/20230601.../https://www.whitehouse.gov/ostp/ai-bill-of-rights/",
  "archiveQuoteFound": true,
  "archiveQuoteMatchScore": 1,
  "billableEvents": ["url-checked", "archive-lookup", "quote-verified"]
}
```

Every row has the same 47 keys (unused ones are `null`), so the schema never shifts under an agent. Main fields:

| Field | Meaning |
|---|---|
| `status` | `alive`, `soft_404`, `parked`, `dead`, `blocked`, `paywalled`, `rate_limited`, `server_error`, `timeout`, `unreachable`, `invalid_url` |
| `isAlive` | `true` (alive, paywalled) / `false` (dead, soft 404, parked) / `null` (can't tell: blocked, rate-limited, timeout, 5xx, unreachable) |
| `statusReason` | Why, e.g. `http_404`, `dns_not_found`, `not_found_title`, `redirect_to_homepage`, `http_403_cloudflare_challenge`, `isAccessibleForFree_false` |
| `verdict` | `supported`, `supported_by_archive`, `quote_not_found`, `quote_not_found_in_archive`, `title_mismatch`, `dead_link`, `unverifiable`, `ok`, `invalid_url` |
| `quoteFound`, `quoteMatchScore`, `quoteMatchType` | Match on the live page: `exact`, `fuzzy` (at or above the threshold, default 0.9), `partial` (0.6 to threshold, usually a paraphrase), `none` |
| `quoteMatchedText`, `quoteContext` | The best matching passage, and the same with about 120 characters around it |
| `titleMatch`, `titleMatchScore` | Expected title vs. the page's title, og:title, citation_title or first heading |
| `pageTitle`, `publishedDate`, `modifiedDate` | From the page (JSON-LD, meta and citation tags, PDF info) |
| `contentHash` | sha256 of the visible text: compare runs to see if a page changed. Page text itself is never stored |
| `archiveUrl`, `archiveTimestamp`, `archiveQuoteFound` | The Wayback capture and whether the quote is on it |
| `retryAfterSecs` | From `Retry-After` on 429/503 |
| `billableEvents` | Which paid events this row triggered |

Dataset views: **Overview**, **Quote checks**, **Liveness details**, **Wayback copies**. A run summary (counts by status and verdict, billable events, requests per domain, warnings) is saved as `RUN_SUMMARY` in the key-value store.

PDF links are read too (text and title), up to 20 MB.

### Pricing (pay per event)

You pay only for work that actually happened:

| Event | Price | When |
|---|---|---|
| URL checked | $0.001 | Each unique URL actually requested. Invalid URLs are free; a URL repeated in the input is fetched and charged once |
| Quote verified | $0.003 | Each link whose quote was compared with page text (live, rendered or archived copy; once per link). Not charged when there was no text to check |
| Browser render | $0.002 | Each successful headless-browser render (only if you turn `browserFallback` on; off by default) |
| Wayback lookup | $0.001 | Each archive.org lookup for a dead link (or the statuses you pick in `archiveOn`), whether or not a capture exists |
| Actor start | $0.0005 | Once per run (default 1 GB memory; Apify charges this once per started GB) |

Worked examples:

- Check **100 links** for liveness only: about $0.1005.
- Verify **100 citations with quotes**, 10 of them dead with a Wayback copy: about $0.4105.
- The blank-input demo: about $0.0105.
- `maxUrls` (default 500) caps a default run at about $3.50 even if every event fired on every link.

There is no charge per dataset row. Set **Max cost per run** in the run options to cap any run; the Actor stops cleanly when the cap is reached and says so in `RUN_SUMMARY`.

### How it decides

- **Soft 404:** a 2xx page whose title or first heading says not found / 404 / doesn't exist, a short page whose text says so, a redirect to an error page, or a deep link that redirects to the site's home page.
- **Parked:** "this domain is for sale" pages, parking scripts, or a redirect to a domain marketplace.
- **Blocked:** 401, 403, 406, 451, LinkedIn's 999, and Cloudflare, PerimeterX, DataDome, Imperva, Akamai or CAPTCHA pages (even when served with 200). A blocked page is *not* reported as dead: `isAlive` is `null` and the verdict is `unverifiable`.
- **Paywalled:** 402, schema.org `isAccessibleForFree: false`, a locked or metered `article:content_tier`, or "subscribe to continue reading" text. If the quote is still readable, the verdict is `supported`.
- **Rate limited:** 429, or 503 with `Retry-After`.
- **Quote match:** text is normalised (case, accents, curly quotes, punctuation, whitespace). An exact match scores 1. Otherwise the best window of about the quote's length is scored by character similarity. Use `...` between separate passages; each is checked and the score is length-weighted.

### Fair use: what it will and won't do

- Fetches **only the URLs you give** (and their redirects), plus one Wayback lookup and, if found and needed, one archived copy per link. It never crawls or follows other links.
- **Reports blocks, never bypasses them.** No proxies or unblockers, no CAPTCHA solving, no fingerprint spoofing, no login. It identifies itself honestly as `TallowLaneLabs-LinkCitationVerifier`.
- **Polite per-domain rate limiting:** one request at a time per site with a 1 s gap by default (`perDomainDelayMs`), at most 5 sites in parallel (`maxConcurrency`), no retries on 4xx/429. archive.org always gets at least 1.5 s between requests.
- Refuses private, loopback and cloud-metadata addresses.
- Keeps only titles, dates, a hash and short matched snippets, never full page text.

### Browser fallback (optional)

Some pages only show their text after JavaScript runs. Turn on `browserFallback` and live pages that look JavaScript-rendered, or whose static HTML lacks the quote, are rendered once in headless Chrome and checked again ($0.002 per render). The browser loads the page's own scripts and styles but not images, media or fonts. It is never used on blocked, rate-limited or paywalled pages. One page renders at a time, so 1 GB of memory is enough.

### Use from AI agents (MCP)

Add the Actor to the Apify MCP server (`tallowlanelabs/link-citation-verifier`) and call it with `links`. Read `verdict` first. Then use `quoteMatchedText` as evidence, `archiveUrl` as a replacement link for dead sources, and `status` to decide whether to retry later (`rate_limited`, `timeout`, `server_error`).

### Limits

- Soft-404 and paywall detection are heuristics. Unusual sites can be misread, and `statusReason` always says which rule fired.
- Sites that block automated requests come back as `blocked`, by design. Check those manually.
- No robots.txt check: only the exact URLs you give are requested, one at a time per site, like a link preview.
- Quote matching compares words, not meaning: a faithful paraphrase scores as `partial`, not `supported`.
- The Wayback Machine can be slow or rate-limited. A failed lookup is not charged and is noted in `archiveNote`.
- Pages over 5 MB of HTML or 20 MB of PDF are checked on the first part only (noted in `errorMessage`).

### Support

Open an issue on the **Issues** tab or email Tallow Lane Labs at wdeornellas+tallowlanelabs@gmail.com.

# Actor input Schema

## `links` (type: `array`):

One object per link: {"url": "https://...", "quote": "text the page should contain", "title": "title the page should have", "citedDate": "2024-05-01", "id": "ref-12"}. Only url is required. quote is fuzzy-matched (case, accents, punctuation and whitespace ignored; use ... between separate passages). citedDate (YYYY-MM-DD) picks the Wayback capture nearest that date. id is echoed back. A plain URL string also works.

## `urls` (type: `array`):

Optional shortcut: URLs to check for liveness only. Added after the links above. Each costs one URL check.

## `minQuoteMatchScore` (type: `number`):

quoteFound is true when the best match scores at least this. 1 = verbatim (after ignoring case, accents, punctuation and spacing); 0.9 (default) allows small typos or a changed word; paraphrases usually score 0.6-0.88 and come back as matchType "partial".

## `minTitleMatchScore` (type: `number`):

titleMatch is true when the expected title matches the page's <title>, og:title, citation_title or first heading at least this well. "Link rot" matches "Link rot - Wikipedia".

## `archiveFallback` (type: `boolean`):

For links whose status is in "Look up the archive when", ask the Wayback Machine (archive.org availability API, then CDX) for the nearest capture and return its URL. $0.001 per lookup.

## `archiveOn` (type: `array`):

Statuses that trigger a Wayback lookup. Add "quote_not_found" to also check old captures when a live page no longer contains the quote (the page may have changed since it was cited).

## `verifyArchivedCopy` (type: `boolean`):

When a capture is found and the link has a quote or title, fetch the archived copy once and check it too (archiveQuoteFound, archiveTitleMatch). Counts as the link's single quote check; no extra charge beyond the $0.003 quote check.

## `browserFallback` (type: `boolean`):

Off by default. When on, a live page whose static HTML looks JavaScript-rendered (little text, empty app shell) or does not contain the quote is rendered once in headless Chrome and checked again. $0.002 per successful render. Never used to get past blocks, logins, CAPTCHAs, rate limits or paywalls. Renders load the page's own scripts and styles (not images, media or fonts).

## `browserFallbackMode` (type: `string`):

"auto" (default): only pages that look JavaScript-rendered or lack the quote. "always": every live HTML page (one render each).

## `perDomainDelayMs` (type: `integer`):

Requests to one site (registrable domain, e.g. all of \*.example.co.uk) run one at a time with at least this gap. Default 1000 ms; minimum 250. archive.org always gets at least 1500 ms.

## `maxConcurrency` (type: `integer`):

How many different domains are checked at the same time (each domain is still one request at a time).

## `requestTimeoutSecs` (type: `integer`):

A page that does not answer in time is reported as status "timeout" (not retried).

## `maxUrls` (type: `integer`):

Safety cap: links beyond this are ignored (noted in RUN_SUMMARY.warnings). Default 500 keeps a default run under about $3.50 even if every event fires on every link.

## `includeRedirectChain` (type: `boolean`):

List each redirect hop as "<status> <url>" in redirectChain.

## Actor input object example

```json
{
  "links": [
    {
      "url": "https://en.wikipedia.org/wiki/Link_rot",
      "quote": "is the phenomenon of hyperlinks tending over time to cease to point to their originally targeted file, web page, or server",
      "title": "Link rot"
    },
    {
      "url": "https://www.rfc-editor.org/rfc/rfc9110.html",
      "quote": "The 404 (Not Found) status code indicates that the origin server did not find a current representation for the target resource",
      "title": "RFC 9110: HTTP Semantics"
    },
    {
      "url": "https://example.com/this-page-was-never-here",
      "quote": "This page was cited but never existed."
    }
  ],
  "urls": [],
  "minQuoteMatchScore": 0.9,
  "minTitleMatchScore": 0.8,
  "archiveFallback": true,
  "archiveOn": [
    "dead",
    "soft_404",
    "parked",
    "unreachable"
  ],
  "verifyArchivedCopy": true,
  "browserFallback": false,
  "browserFallbackMode": "auto",
  "perDomainDelayMs": 1000,
  "maxConcurrency": 5,
  "requestTimeoutSecs": 20,
  "maxUrls": 500,
  "includeRedirectChain": true
}
```

# Actor output Schema

## `results` (type: `string`):

One row per input link: status, isAlive, verdict, quote match, Wayback copy.

## `runSummary` (type: `string`):

JSON summary: counts by status and verdict, billable events, requests per domain, warnings.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "links": [
        {
            "url": "https://en.wikipedia.org/wiki/Link_rot",
            "quote": "is the phenomenon of hyperlinks tending over time to cease to point to their originally targeted file, web page, or server",
            "title": "Link rot"
        },
        {
            "url": "https://www.rfc-editor.org/rfc/rfc9110.html",
            "quote": "The 404 (Not Found) status code indicates that the origin server did not find a current representation for the target resource",
            "title": "RFC 9110: HTTP Semantics"
        },
        {
            "url": "https://example.com/this-page-was-never-here",
            "quote": "This page was cited but never existed."
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("tallowlanelabs/link-citation-verifier").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "links": [
        {
            "url": "https://en.wikipedia.org/wiki/Link_rot",
            "quote": "is the phenomenon of hyperlinks tending over time to cease to point to their originally targeted file, web page, or server",
            "title": "Link rot",
        },
        {
            "url": "https://www.rfc-editor.org/rfc/rfc9110.html",
            "quote": "The 404 (Not Found) status code indicates that the origin server did not find a current representation for the target resource",
            "title": "RFC 9110: HTTP Semantics",
        },
        {
            "url": "https://example.com/this-page-was-never-here",
            "quote": "This page was cited but never existed.",
        },
    ] }

# Run the Actor and wait for it to finish
run = client.actor("tallowlanelabs/link-citation-verifier").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "links": [
    {
      "url": "https://en.wikipedia.org/wiki/Link_rot",
      "quote": "is the phenomenon of hyperlinks tending over time to cease to point to their originally targeted file, web page, or server",
      "title": "Link rot"
    },
    {
      "url": "https://www.rfc-editor.org/rfc/rfc9110.html",
      "quote": "The 404 (Not Found) status code indicates that the origin server did not find a current representation for the target resource",
      "title": "RFC 9110: HTTP Semantics"
    },
    {
      "url": "https://example.com/this-page-was-never-here",
      "quote": "This page was cited but never existed."
    }
  ]
}' |
apify call tallowlanelabs/link-citation-verifier --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,tallowlanelabs/link-citation-verifier"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/gAtp5NSyTjZEFTzv7/builds/pJSfTfhE1z4RLyVyf/openapi.json