# Bulk URL & Broken Link Checker (`humble-echidna/url-checker`) Actor Check URLs in bulk: status code, every redirect hop, final and canonical URL, response time, content type and size. Paste URLs, check a whole sitemap, or find broken links on your pages. HEAD with GET fallback, robots.txt respected, only-changed monitoring. - **URL**: https://apify.com/humble-echidna/url-checker.md - **Developed by:** [Michael Costa](https://apify.com/humble-echidna) (community) - **Categories:** SEO tools, Developer tools, Automation - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $1.00 / 1,000 url checkeds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ### What does Bulk URL & Broken Link Checker do? **Bulk URL & Broken Link Checker** checks **URLs in bulk** and finds **broken links**. Give it a list of URLs, a sitemap or pages to scan; for each URL you get the **HTTP status code**, **every redirect hop**, the final and canonical URL, and the response time. | Three ways to choose what to check | | |---|---| | **Check these URLs** | Paste a list: a migration's redirect map, a backlink list, a CSV column. | | **All URLs in a site's sitemap** | Give a website or a sitemap URL; every URL its sitemaps list is checked. | | **Links on these pages** | Give pages; every link on them is checked, with the pages it was found on. Same-site pages can be followed a few clicks deep. That's a broken link check. | **Try it in one click:** the input comes pre-filled with four apify.com addresses: a page, an http:// address that redirects, a docs page and a page that doesn't exist. That's 4 results, about $0.004 (4 × $0.001, plus $0.00005 for the run start). **Then replace them with your own URLs.** ### Monitor URLs: get only the ones that changed, in Slack, email or a webhook With **Only report URLs that changed since the last run** on, each run reports just the URLs whose outcome, status code, final URL or canonical URL changed: the ones that broke, recovered or started redirecting somewhere else. A run where nothing changed returns 0 results and costs only the $0.00005 start fee. 1. Fill in **What to check** and **URLs** (for example `sitemap` and your site), turn on **Only report URLs that changed since the last run** (`"onlyChanged": true`), and click **Start**. This first run reports every URL, and is the baseline the next runs compare with. 2. Click **Save as a new task** (top right of the actor page). The memory belongs to that exact input (the same mode, URLs and options), so the task keeps comparing with its own last run. Changing the input starts a new baseline. 3. In Apify Console, open **Schedules**, click **Create new**, set how often in **Schedule setup** (for example daily at 07:00), then **Add** your task. 4. On the task, open the **Integrations** tab and pick where the changes go: - **Slack**: click **Configure**, sign in, pick the workspace and channel, and the "run succeeded" event. A useful message: `{{resource.statusMessage}}` (the example below's repeat run said "4 URLs checked (3 ok, 1 broken); 4 unchanged since the last run, not repeated") and a link to the results, ``. - **Gmail**: click **Connect with Google**, set the subject and body, and attach the dataset (for example as CSV). It sends after each successful run. - **HTTP webhook**: event `ACTOR.RUN.SUCCEEDED`, your URL. Apify POSTs `{"eventType": ..., "resource": {...}}`; `resource.defaultDatasetId` is the run's dataset, and `GET https://api.apify.com/v2/datasets//items?format=json` (with your API token) returns the changed URLs. Apify's integrations fire after every successful run, including quiet ones: a quiet run's dataset is empty, and its status message says how many URLs were unchanged. Each result's `previous` field says what the URL's status code and outcome were last time. ### What data does Bulk URL & Broken Link Checker return? | Field | Example | Notes | |---|---|---| | `url` | `http://apify.com/` | The URL checked. | | `outcome` | `ok` | `ok`, `broken`, `refused`, `dns-error` or `error` (see [Output](#output)). | | `statusCode` | `200` | The final answer's status code. `null` when no answer came back. | | `redirectCount` | `1` | Hops before the final answer. | | `redirectChain` | `[{"url": ..., "statusCode": 301, "location": ..., "timeMs": 41.2}, ...]` | Every response on the way, including the last. | | `finalUrl` | `https://apify.com/` | Where the chain ended. | | `canonicalUrl` | `https://apify.com/` | From the page's HTML or the `Link` header; `null` if none. | | `responseTimeMs` | `160.1` | Summed over the chain. | | `contentType`, `sizeBytes` | `text/html`, `412345` | From the final answer. | | `foundOn`, `linkText` | `["https://example.com/blog/"]`, `"Our old post"` | Links mode only: where the link was and its text. | | `previous` | `{"statusCode": 200, "outcome": "ok"}` | Only-changed mode: last run's answer; `null` when new. | | `error` | `null` | The reason, for anything but `ok`. | One row per URL, checked once per run even when it's listed or linked many times. The full list is under [Output](#output). ### How much does it cost to check URLs and broken links? You pay per URL in the results: **$1.00 per 1,000 URLs checked**, plus $0.00005 each time a run starts. - **The example below:** 4 URLs × $0.001 = $0.004, plus the start fee; the repeat run with nothing changed, $0.00005. - **A month, for example:** a daily check of a 1,000-URL sitemap with only-changed on: the first run is 1,000 × $0.001 = $1.00; if 20 URLs change on each of the other 29 days, that's 580 × $0.001 = $0.58; plus 30 starts ($0.0015): **about $1.58**. - **Caps:** **Max results per run** in the input, and **Maximum cost per run** in the run options. The run stops cleanly at whichever comes first; nothing is requested past the limit, so a capped run is also a fast one. **Never charged:** URLs that were never requested: addresses robots.txt disallows, private or local addresses, ports other than 80/443, addresses that aren't valid URLs, sites that ask for more than 30 seconds between requests, and (links mode) the other links to a site that answered a check with HTTP 429. They aren't in the results; they're listed in the **`NOT_CHECKED`** record of the run's key-value store (the "URLs not requested" link on the run's Output tab), counted in `RUN_STATS`, and named in the run log. They don't use up your maximum cost per run either. Sitemap files and robots.txt files read along the way are free, and so are URLs whose result didn't change (only-changed mode). **Charged, because they're answers:** a URL that *was* requested and then redirected into one of the above (its redirect chain is the answer), and a `dns-error` (the domain was looked up and doesn't exist). ### How to check URLs and broken links 1. Open Bulk URL & Broken Link Checker and click **Try for free** (or **Start** if you're signed in). 2. Pick **What to check**: `urls`, `sitemap` or `links`. 3. Paste your addresses into **URLs**, one per line (`example.com` means `https://example.com/`). 4. Optional: for links mode, set **Crawl depth** and **Max pages to read**; turn on **Only report URLs that changed since the last run** for monitoring. 5. Click **Start**, then open the **Output** tab and export as JSON, CSV or Excel. ### Example: four addresses, then a repeat run The pre-filled input, with **Only report URLs that changed since the last run** turned on: ```json {"mode": "urls", "urls": ["https://apify.com", "http://apify.com", "https://docs.apify.com/platform", "https://docs.apify.com/this-page-does-not-exist"], "method": "auto", "readCanonical": true, "onlyChanged": true} ``` Two of the four results (real output from a local run on 2026-09-25, some fields left out): ```json [ {"url": "https://docs.apify.com/platform", "outcome": "ok", "statusCode": 200, "redirectCount": 1, "redirectChain": [ {"url": "https://docs.apify.com/platform", "statusCode": 301, "location": "https://docs.apify.com/", "timeMs": 25.9}, {"url": "https://docs.apify.com/", "statusCode": 200, "location": null, "timeMs": 73.7}], "finalUrl": "https://docs.apify.com/", "canonicalUrl": "https://docs.apify.com/", "responseTimeMs": 99.6, "method": "GET", "previous": null}, {"url": "https://docs.apify.com/this-page-does-not-exist", "outcome": "broken", "statusCode": 404, "redirectCount": 0, "finalUrl": "https://docs.apify.com/this-page-does-not-exist", "method": "GET", "previous": null} ] ``` Run again straight after, the same input returned **0 results**: nothing had changed, so nothing was charged but the start fee. ### Input | Field | What it does | |---|---| | **What to check** (`mode`) | `urls`: each line is a URL. `sitemap`: each line is a website or sitemap URL. `links`: each line is a page whose links are checked. | | **URLs** (`urls`) | One per line. `example.com` means `https://example.com/`. | | Request method (`method`) | `auto` (HEAD, GET when needed; recommended), `head` or `get`. | | Read the canonical URL (`readCanonical`) | On by default. Off: HTML pages are checked with HEAD too (faster, lighter); a canonical URL in the HTTP `Link` header is still reported. | | Crawl depth (`maxDepth`, links mode) | 0 (default): the links on your pages. 1: also read the same-site pages they link to, and check their links. Up to 5. | | Max pages to read (`maxPages`, links mode) | Pages read for links, your start pages included. Default 20. | | Check links to other sites (`checkExternalLinks`, links mode) | On by default. Other sites' pages are checked, never crawled. | | Also check images, scripts and stylesheets (`includeResources`, links mode) | Off by default. | | Max URLs per site (`maxUrlsPerSite`, sitemap mode) | Default 500, up to 50,000. | | Only report URLs that changed since the last run (`onlyChanged`) | Off by default. | | Max results per run (`maxResults`) | Cap the total number of URLs in the results. | ```json { "mode": "links", "urls": ["https://example.com/blog/"], "maxDepth": 1, "maxPages": 50, "maxResults": 2000 } ``` ### Output One row per URL, checked once per run even when it's listed or linked many times. Fields that don't apply are `null`. ```json { "url": "http://apify.com/", "outcome": "ok", "statusCode": 200, "ok": true, "redirectCount": 1, "redirectChain": [ {"url": "http://apify.com/", "statusCode": 301, "location": "https://apify.com/", "timeMs": 41.2}, {"url": "https://apify.com/", "statusCode": 200, "location": null, "timeMs": 118.9} ], "finalUrl": "https://apify.com/", "canonicalUrl": "https://apify.com/", "responseTimeMs": 160.1, "contentType": "text/html", "sizeBytes": 412345, "method": "GET", "error": null, "source": "http://apify.com", "foundOn": null, "foundOnCount": null, "linkText": null, "previous": null, "checkedAt": "2026-01-01T00:00:00Z" } ``` - `outcome`: `ok` (final status 2xx), `broken` (other 4xx and 5xx, and 3xx without a target), `refused` (401, 403, 407, 429, 451), `dns-error` (the domain doesn't resolve), `error` (timeout, TLS failure, more than 10 redirects, and the like; `error` says which). After a redirect, also `robots-blocked`, `not-public` (a private network address, a port other than 80/443, or not http/https) or `invalid-url` for where it pointed. - `redirectChain`: every response on the way, including the last. When a redirect leads somewhere that can't be requested (robots.txt, a private address), the chain shows the hops up to there and `finalUrl` is where it stopped. - `responseTimeMs`: from sending each request to receiving its response headers, summed over the chain. - `sizeBytes`: the `Content-Length` header, or the downloaded size when the whole body was read. - `source`: the input line the URL came from. `foundOn`, `foundOnCount`, `linkText` (links mode): the pages read this run that link to it (first 20), how many there are, and the link's text or image alt text. - `previous` (only-changed mode): the status code and outcome from the last run that reported this URL; `null` when it's new. #### URLs not requested (`NOT_CHECKED`) Every run writes a `NOT_CHECKED` record to its default key-value store: `{"count": 3, "items": [...], "truncated": false}`. Each item has `url`, `outcome` (`robots-blocked`, `not-public`, `invalid-url`, `error` for a site asking for more than 30 s between requests, or `rate-limited` in links mode), `error` (the reason), `source`, `foundOn`, `foundOnCount` and `checkedAt`. The first 10,000 are listed; `count` is the full number. `RUN_STATS` has `notChecked` and `notCheckedOutcomes`. ### Run it on a schedule, or from your own code 1. Save your input as a **task** and add it to a **schedule** (Console → Schedules), as in *Monitor URLs* above. 2. Collect results: download the dataset as JSON, CSV or Excel; fetch the latest run's results from the API (`GET https://api.apify.com/v2/actor-tasks//runs/last/dataset/items?status=SUCCEEDED&format=csv`, with your API token); let a webhook tell your system when a run succeeds; or connect it to Make, Zapier or n8n through Apify's integrations. The only-changed memory lives in a key-value store in your own account (`url-checker-memory`), per input, so a task keeps its memory from run to run. #### Can I use Bulk URL & Broken Link Checker from an AI agent (MCP)? Yes, through Apify's MCP server: add `https://mcp.apify.com?tools=humble-echidna/url-checker` to your MCP client (or let the agent find it with the server's actor search). The agent passes the same input as above, e.g. `{"mode": "urls", "urls": ["https://example.com/old-page"]}`; keep `maxResults` small for quick answers. ### Who it's for People who own a list of URLs and need to know when one breaks or starts redirecting somewhere else: SEO and web teams after a migration (checking the redirect map), site owners watching their sitemap, and anyone checking a backlink or partner-link list. ### Why this one? - **The whole redirect chain, hop by hop**: each hop's address, status code, where it pointed and how long it took. Not just the first and last status. - **HEAD first, GET only when needed.** Files are checked with a HEAD request, so nothing is downloaded. Many servers answer HEAD wrongly (or not at all), so an error answer, or a HEAD request that fails, is confirmed with GET before a URL is reported broken. Pages are read with GET when you want their canonical URL (one request, not two). - **Clear outcomes, not just numbers**: `ok`, `broken`, `refused` (the site turned away an automated check; it may work in a browser), `dns-error` or `error`, with the reason. URLs it won't request (robots.txt, private or invalid addresses) are listed separately, free. - **Only changed since last run**: schedule it, and each run reports just the URLs whose result changed. Good for monitoring redirects and catching new broken links. - **Polite and safe.** It identifies itself honestly (User-Agent `HumbleEchidnaApify`), follows each site's robots.txt (read once per site per run) and Crawl-delay, sends at most 2 requests at a time to one site, and only requests public web addresses on the standard ports (80 and 443). - **Reliable.** One failing URL, page or sitemap never affects the others in your run. The run log and the `RUN_STATS` record say exactly what happened. ### Limits - Only public web addresses on the standard ports (80 for http, 443 for https) are requested, for every redirect hop too. Any other address is never requested (see `NOT_CHECKED`; free); a redirect to one stops there. - robots.txt is respected on every hop: a disallowed URL is never requested (see `NOT_CHECKED`; free). A site whose robots.txt can't be read (server error) is treated as disallowing everything for the run, as the robots.txt standard says. A site that asks for more than 30 seconds between requests isn't requested either (`NOT_CHECKED`, outcome `error`, free). - No proxies and no login: pages behind a login, a paywall or bot protection come back as `refused` (usually 403). - A failed request is retried once. Each URL gets up to 60 seconds in total. Pages are read up to 2 MB. - Links mode reads HTML as the server sends it: links added by JavaScript after the page loads aren't seen. - Links mode checks the links to different sites in turns, at most 2 at a time per site, so a slow site doesn't hold up the rest. A site that answers a link check with HTTP 429 (too many requests, even after one retry) is left alone for the rest of the run: that link is a `refused` result, and the site's other links aren't requested (`NOT_CHECKED`, outcome `rate-limited`, free; the site is named in `linkHostsThrottled` in `RUN_STATS`). Large sites such as GitHub often do this when a page links to hundreds of their pages. ### FAQ #### Why is a URL `refused` when it opens fine in my browser? Some sites block automated requests or require a login. This checker doesn't get around that, so it reports what the site answered. #### Can I get only the broken links or changed URLs since my last run? Yes: turn on **Only report URLs that changed since the last run**. A URL counts as changed when its outcome, status code, final URL or canonical URL differs from the last run of the same check (same mode, URLs and options). Response time and size don't count. The memory lives in a key-value store named `url-checker-memory` in your own account. See *Monitor URLs* above. #### Why did a URL or input return nothing? Look in `NOT_CHECKED` (URLs that were never requested, with the reason) and in the run log and `RUN_STATS`, which name the input line and what went wrong; every other URL in the run is unaffected. #### Is it legal to check URLs and broken links? It requests only the addresses you give it, or that your pages and sitemaps list, the way a browser or a search engine would: no login, honouring robots.txt, identified by its User-Agent. It collects HTTP metadata about URLs, not personal data. You're responsible for checking only sites you're allowed to. ### Related actors | Actor | Use it when | |---|---| | [Sitemap URL Extractor](https://apify.com/humble-echidna/sitemap-urls) | You want the sitemap's URLs themselves (with `lastmod`, images, hreflang) rather than their status. | | [SEO Audit Crawler](https://apify.com/humble-echidna/seo-audit) | You want a full on-page audit (titles, H1s, canonicals, broken links per page) of the same site, for weekly or monthly reports. | ### Feedback and support Found a bug, or need a field or option that isn't here? Open an issue on the **Issues** tab with the input you used. ### Versions Current version: **1.0**. See the Changelog tab for what changed in each version. # Changelog This Actor's version history is a separate document: https://apify.com/humble-echidna/url-checker/changelog.md # Actor input Schema ## `mode` (type: `string`): What the lines in urls are. `urls` (Check these URLs, the default): each line is one URL to check. `sitemap`: each line is a website (example.com) or a sitemap URL; every URL its sitemaps list is checked, up to maxUrlsPerSite per site. `links` (broken link check): each line is a page; every link on it is checked, and same-site pages it links to are read too, up to maxDepth and maxPages. ## `urls` (type: `array`): One per line, read according to mode: URLs to check (e.g. https://example.com/old-page), websites or sitemap URLs, or start pages for the link check. Only public web addresses on the standard ports (80 and 443) are requested; anything else is listed in the NOT\_CHECKED record, not fetched and not charged. The default is 4 URLs on apify.com, one of them a redirect and one a 404. ## `method` (type: `string`): Which HTTP request checks each URL. `auto` (the default, recommended): HEAD first, and GET when the HEAD answer is an error (many servers answer HEAD wrongly) or when the page's HTML is needed for its canonical URL (readCanonical). `head`: HEAD only. `get`: GET only. ## `readCanonical` (type: `boolean`): Default true: read from HTML pages, which needs a GET request. Set false for faster, lighter HEAD-only checks; a canonical URL sent in the HTTP Link header is still reported. ## `maxDepth` (type: `integer`): Links mode only (mode `links`): how many levels of same-site pages to read, 0 to 5, default 0. 0 checks the links on your pages only. 1 also reads the same-site pages they link to and checks their links, and so on, up to maxPages pages. Other sites' pages are checked but never crawled. ## `maxPages` (type: `integer`): Links mode only (mode `links`): at most this many pages are read for links, your start pages included. 1 to 5,000, default 20. ## `checkExternalLinks` (type: `boolean`): Links mode only (mode `links`), default true: also check links that point to other websites (checked, never crawled). false: only links within the start pages' sites. ## `includeResources` (type: `boolean`): Links mode only (mode `links`), default false: besides links, also check images, scripts, stylesheets, iframes and media sources the pages load. Each one is a result like any other URL. ## `maxUrlsPerSite` (type: `integer`): Sitemap mode only (mode `sitemap`): check at most this many URLs from each site's sitemaps. 1 to 50,000, default 500. Set it low (e.g. 50) for a quick sample of a big site. ## `onlyChanged` (type: `boolean`): Default false. When true, remembers each URL's result between runs of the same check (same mode, URLs and options), in a key-value store in your account. A URL is reported, and charged, only when it's new or its outcome, status code, final URL or canonical URL changed. Good for scheduled monitoring. ## `maxResults` (type: `integer`): Stop after this many URLs in the results in total, e.g. 20. Minimum 1; leave empty (the default) for no limit. Nothing is fetched past the limit. The run also stops cleanly at the maximum cost per run you set in the run options, whichever comes first. Set it to keep a run small and quick. ## Actor input object example ```json { "mode": "urls", "urls": [ "https://apify.com", "http://apify.com", "https://docs.apify.com/platform", "https://docs.apify.com/this-page-does-not-exist" ], "method": "auto", "readCanonical": true, "maxDepth": 0, "maxPages": 20, "checkExternalLinks": true, "includeResources": false, "maxUrlsPerSite": 500, "onlyChanged": false } ``` # Actor output Schema ## `results` (type: `string`): No description ## `runStats` (type: `string`): No description ## `notChecked` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "urls": [ "https://apify.com", "http://apify.com", "https://docs.apify.com/platform", "https://docs.apify.com/this-page-does-not-exist" ] }; // Run the Actor and wait for it to finish const run = await client.actor("humble-echidna/url-checker").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "urls": [ "https://apify.com", "http://apify.com", "https://docs.apify.com/platform", "https://docs.apify.com/this-page-does-not-exist", ] } # Run the Actor and wait for it to finish run = client.actor("humble-echidna/url-checker").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "urls": [ "https://apify.com", "http://apify.com", "https://docs.apify.com/platform", "https://docs.apify.com/this-page-does-not-exist" ] }' | apify call humble-echidna/url-checker --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,humble-echidna/url-checker" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/qxXHqIZMbyWF2XfhU/builds/hO77vpsCfEGJRmaRo/openapi.json