# Bulk URL Status Checker (`fetch_cat/bulk-url-status-checker`) Actor Check large URL lists for status codes, redirects, broken links, response timing, headers, titles, canonical URLs, and robots meta. - **URL**: https://apify.com/fetch\_cat/bulk-url-status-checker.md - **Developed by:** [Hanna Nosova](https://apify.com/fetch_cat) (community) - **Categories:** SEO tools, Developer tools, Automation - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.15 / 1,000 url checkeds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Bulk URL Status Checker Check thousands of URLs for HTTP status codes, broken links, redirect chains, response timing, content headers, page titles, meta descriptions, canonical URLs, and robots meta tags. Use this Actor for SEO audits, website migrations, QA checks, content operations, campaign link validation, and scheduled link-health monitoring. ### At a glance - **Primary job**: Turn a URL list, pasted text block, hosted list, or sitemap into a structured URL health report. - **Input**: URLs, `urlsText`, `listUrl`, or `sitemapUrl` with configurable method, redirects, timeout, and concurrency. - **Output**: One dataset row per attempted URL, including both successful responses and failures. - **Best for**: SEO teams, agencies, site migration teams, QA teams, marketers, and developers building monitoring workflows. ### Who is it for? - **SEO teams and agencies** validating migrations, redirects, canonical tags, and broken links. - **QA teams** checking landing pages before launches. - **Content operations teams** finding removed, redirected, or slow pages. - **Developers** feeding URL health data into dashboards, alerts, and CI workflows. ### Input recipes Start from a verified public Example task when you want a ready-made configuration: - [Check website status codes and SEO metadata](https://apify.com/fetch_cat/bulk-url-status-checker/examples/website-status-seo-check) - [Find broken links and redirect chains](https://apify.com/fetch_cat/bulk-url-status-checker/examples/broken-links-redirect-check) - [Check a pasted list of URLs in bulk](https://apify.com/fetch_cat/bulk-url-status-checker/examples/pasted-url-list-status-check) - [Check URLs from an XML sitemap](https://apify.com/fetch_cat/bulk-url-status-checker/examples/xml-sitemap-status-audit) ### Example input ```json { "urls": ["https://example.com/", "https://www.iana.org/domains/example"], "maxUrls": 20, "concurrency": 10, "method": "head-get-fallback", "followRedirects": true, "includeSeoMeta": true } ```` ### What data can you export? | Field | Description | | --- | --- | | `inputUrl` | Original URL supplied by the user. | | `normalizedUrl` | URL after scheme normalization and fragment removal. | | `finalUrl` | Final URL after redirects. | | `statusCode` / `statusText` | Final HTTP response status. | | `ok` | True for final 2xx or 3xx responses. | | `isBroken` | True for request errors or final 4xx/5xx responses. | | `isRedirect` / `redirectCount` | Redirect classification and number of followed hops. | | `redirectChain` | Redirect hops with source URL, status code, and Location. | | `responseTimeMs` | Elapsed check time in milliseconds. | | `contentType` / `contentLength` | Response headers commonly used for QA and SEO exports. | | `title` | HTML page title when SEO extraction is enabled and available. | | `metaDescription` | HTML meta description when available. | | `canonicalUrl` | HTML canonical URL when available. | | `robotsMeta` | HTML robots meta tag content when available. | | `errorType` / `errorMessage` | Request failure details for DNS, TLS, timeout, and invalid URL cases. | | `responseHeaders` | Optional raw response headers. | | `checkedAt` | ISO timestamp for the check. | ### Input configuration | Setting | JSON key | Use it for | Default | | --- | --- | --- | --- | | URLs | `urls` | Direct list of URLs to check. | sample URLs | | URLs as text | `urlsText` | Paste newline, comma, tab, semicolon, or space separated URLs. | empty | | Hosted URL list | `listUrl` | Fetch a remote text/CSV-like URL list. | empty | | Sitemap URL | `sitemapUrl` | Extract and check `` entries from an XML sitemap. | empty | | Max URLs | `maxUrls` | Cap unique URLs after de-duplication. | `20` | | Concurrency | `concurrency` | Parallel checks. Lower for fragile sites. | `20` | | Timeout | `timeoutSecs` | Per-URL request timeout. | `15` | | Request method | `method` | `head-get-fallback`, `get`, or `head`. | `head-get-fallback` | | Follow redirects | `followRedirects` | Follow redirects and report final destination. | `true` | | Max redirects | `maxRedirects` | Redirect-hop limit per URL. | `10` | | Include SEO meta | `includeSeoMeta` | Extract title, description, canonical, and robots meta for HTML pages. | `true` | | Include headers | `includeResponseHeaders` | Include raw response headers. | `false` | | User-Agent | `userAgent` | Send a custom User-Agent. | built-in crawler UA | | Proxy | `proxyConfiguration` | Optional Apify Proxy routing. | off | ### Example output ```json { "inputUrl": "https://example.com/", "normalizedUrl": "https://example.com/", "finalUrl": "https://example.com/", "statusCode": 200, "statusText": "OK", "ok": true, "isBroken": false, "isRedirect": false, "redirectChain": [], "redirectCount": 0, "responseTimeMs": 184, "contentType": "text/html", "contentLength": 1256, "title": "Example Domain", "metaDescription": null, "canonicalUrl": null, "robotsMeta": null, "errorType": null, "errorMessage": null, "checkedAt": "2026-07-10T00:00:00.000Z" } ``` ### Pricing This Actor uses pay-per-event pricing. | Event | Free | Bronze | Silver | Gold | Platinum | Diamond | Charged when | | --- | ---: | ---: | ---: | ---: | ---: | ---: | --- | | Run start | `$0.005` | `$0.005` | `$0.005` | `$0.005` | `$0.005` | `$0.005` | Once when the Actor starts. | | URL checked | `$0.000030025` | `$0.000026108` | `$0.000020365` | `$0.000015665` | `$0.000010443` | `$0.0000073103` | Each URL is checked and saved as a dataset row, including failures. | ### Tips for best results - **Start small**: Run 10-20 URLs first to verify target behavior and output columns. - **Use `head-get-fallback` by default**: It is fast and falls back when servers reject HEAD or HTML meta is requested. - **Use GET-only when needed**: Some servers return inaccurate HEAD responses. - **Lower concurrency for one domain**: If many URLs hit the same small site, lower `concurrency` to avoid rate limits. - **Keep raw headers off unless needed**: `includeResponseHeaders` increases dataset size. ### Limits and caveats - Some websites block automated checks or return different results by User-Agent, method, region, or proxy. - 403, 429, timeout, DNS, TLS, and invalid URL cases are recorded as rows, not treated as actor failures. - The Actor reports the observed HTTP result; it does not bypass access controls. ### API usage Node.js: ```js import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('fetch_cat/bulk-url-status-checker').call({ urls: ['https://example.com/', 'https://httpstat.us/404'], followRedirects: true, }); console.log(run.defaultDatasetId); ``` Python: ```python from apify_client import ApifyClient client = ApifyClient('YOUR_APIFY_TOKEN') run = client.actor('fetch_cat/bulk-url-status-checker').call(run_input={ 'urls': ['https://example.com/', 'https://httpstat.us/404'], 'followRedirects': True, }) print(run['defaultDatasetId']) ``` cURL: ```bash curl -X POST 'https://api.apify.com/v2/acts/fetch_cat~bulk-url-status-checker/runs?token=YOUR_APIFY_TOKEN' \ -H 'Content-Type: application/json' \ -d '{"urls":["https://example.com/","https://httpstat.us/404"],"followRedirects":true}' ``` ### MCP and AI agents Use the Actor from MCP-compatible clients through the Apify MCP Server: ```bash claude mcp add --transport http apify "https://mcp.apify.com?tools=fetch_cat/bulk-url-status-checker" ``` Claude Desktop JSON configuration: ```json { "mcpServers": { "apify": { "url": "https://mcp.apify.com?tools=fetch_cat/bulk-url-status-checker" } } } ``` Example prompts: - "Check these landing page URLs and summarize broken links." - "Run a sitemap status audit and group results by status code." - "Find redirect chains in this migration URL map." ### Related actors - [Sitemap URL Extractor](https://apify.com/fetch_cat/sitemap-url-extractor) - [Website Contact Finder](https://apify.com/fetch_cat/website-contact-finder) - [Bing Search Results Scraper](https://apify.com/fetch_cat/bing-search-results-scraper) ### FAQ #### Can I check thousands of URLs? Yes. Increase `maxUrls` and choose a safe `concurrency` for the target sites. #### Does this use a browser? No. It is a lightweight HTTP checker for status, redirects, headers, and basic HTML meta signals. #### Why do I see 403 or 429? The target server is refusing or rate limiting the request. Try lower concurrency, GET-only mode, a custom User-Agent, or an approved proxy setup. #### Can I export the results? Yes. Apify datasets can be exported as JSON, CSV, Excel, XML, RSS, HTML, or through the API. ### Legality and responsible use Only check URLs you are allowed to audit. Respect target websites, rate limits, robots policies, contracts, and applicable laws. Do not use the Actor to overload third-party servers. ### Support Open an issue from the Actor page if a run fails or output looks wrong. Include the run ID or run URL, input JSON, expected output, actual output, and one reproducible public URL. ### Changelog - 2026-07-15 - Feature: Added ready-to-run example tasks on the Apify Store - 2026-07-15 - Feature: Launched Bulk URL Status Checker on Apify Store # Actor input Schema ## `urls` (type: `array`): URLs to check directly. Add one URL per row. Domains without a scheme are checked as HTTPS. ## `urlsText` (type: `string`): Optional newline, comma, semicolon, tab, or space separated URLs pasted from a spreadsheet or export. ## `listUrl` (type: `string`): Optional URL of a text or CSV-like file containing URLs separated by newlines, commas, spaces, tabs, or semicolons. ## `sitemapUrl` (type: `string`): Optional XML sitemap URL. The actor extracts entries and checks those URLs. ## `maxUrls` (type: `integer`): Maximum number of unique URLs to check in this run after de-duplication. ## `concurrency` (type: `integer`): How many URLs to check in parallel. Lower this for small or fragile websites. ## `timeoutSecs` (type: `integer`): Hard request timeout for each URL. ## `method` (type: `string`): HEAD with GET fallback is fastest. GET only is useful when target servers reject or misreport HEAD requests. ## `followRedirects` (type: `boolean`): Follow redirects and report the final URL plus redirect chain. ## `maxRedirects` (type: `integer`): Maximum redirect hops to follow for each URL. ## `includeSeoMeta` (type: `boolean`): For HTML responses, parse title, meta description, canonical URL, and robots meta. Disable for fastest status-only runs. ## `includeResponseHeaders` (type: `boolean`): Include raw response headers in each dataset item. Disable for smaller exports. ## `userAgent` (type: `string`): Optional custom User-Agent string sent to target URLs. ## `proxyConfiguration` (type: `object`): Optional Apify Proxy settings. Off by default; enable only when you need proxy routing for target domains. ## Actor input object example ```json { "urls": [ "https://example.com/", "https://www.iana.org/domains/example", "https://httpstat.us/404" ], "maxUrls": 20, "concurrency": 20, "timeoutSecs": 15, "method": "head-get-fallback", "followRedirects": true, "maxRedirects": 10, "includeSeoMeta": true, "includeResponseHeaders": false, "proxyConfiguration": { "useApifyProxy": false } } ``` # Actor output Schema ## `overview` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "urls": [ "https://example.com/", "https://www.iana.org/domains/example", "https://httpstat.us/404" ], "maxUrls": 20, "concurrency": 20, "timeoutSecs": 15, "method": "head-get-fallback", "followRedirects": true, "maxRedirects": 10, "includeSeoMeta": true, "includeResponseHeaders": false }; // Run the Actor and wait for it to finish const run = await client.actor("fetch_cat/bulk-url-status-checker").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "urls": [ "https://example.com/", "https://www.iana.org/domains/example", "https://httpstat.us/404", ], "maxUrls": 20, "concurrency": 20, "timeoutSecs": 15, "method": "head-get-fallback", "followRedirects": True, "maxRedirects": 10, "includeSeoMeta": True, "includeResponseHeaders": False, } # Run the Actor and wait for it to finish run = client.actor("fetch_cat/bulk-url-status-checker").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "urls": [ "https://example.com/", "https://www.iana.org/domains/example", "https://httpstat.us/404" ], "maxUrls": 20, "concurrency": 20, "timeoutSecs": 15, "method": "head-get-fallback", "followRedirects": true, "maxRedirects": 10, "includeSeoMeta": true, "includeResponseHeaders": false }' | apify call fetch_cat/bulk-url-status-checker --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=fetch_cat/bulk-url-status-checker", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Bulk URL Status Checker", "description": "Check large URL lists for status codes, redirects, broken links, response timing, headers, titles, canonical URLs, and robots meta.", "version": "0.1", "x-build-id": "QDh4YkNaiFV7kANKF" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/fetch_cat~bulk-url-status-checker/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-fetch_cat-bulk-url-status-checker", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/fetch_cat~bulk-url-status-checker/runs": { "post": { "operationId": "runs-sync-fetch_cat-bulk-url-status-checker", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/fetch_cat~bulk-url-status-checker/run-sync": { "post": { "operationId": "run-sync-fetch_cat-bulk-url-status-checker", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "urls": { "title": "URLs", "type": "array", "description": "URLs to check directly. Add one URL per row. Domains without a scheme are checked as HTTPS.", "items": { "type": "string" } }, "urlsText": { "title": "URLs as text", "type": "string", "description": "Optional newline, comma, semicolon, tab, or space separated URLs pasted from a spreadsheet or export." }, "listUrl": { "title": "Hosted URL list", "type": "string", "description": "Optional URL of a text or CSV-like file containing URLs separated by newlines, commas, spaces, tabs, or semicolons." }, "sitemapUrl": { "title": "Sitemap URL", "type": "string", "description": "Optional XML sitemap URL. The actor extracts entries and checks those URLs." }, "maxUrls": { "title": "Max URLs", "minimum": 1, "maximum": 100000, "type": "integer", "description": "Maximum number of unique URLs to check in this run after de-duplication.", "default": 20 }, "concurrency": { "title": "Concurrency", "minimum": 1, "maximum": 200, "type": "integer", "description": "How many URLs to check in parallel. Lower this for small or fragile websites.", "default": 20 }, "timeoutSecs": { "title": "Timeout per URL (seconds)", "minimum": 3, "maximum": 120, "type": "integer", "description": "Hard request timeout for each URL.", "default": 15 }, "method": { "title": "Request method", "enum": [ "head-get-fallback", "get", "head" ], "type": "string", "description": "HEAD with GET fallback is fastest. GET only is useful when target servers reject or misreport HEAD requests.", "default": "head-get-fallback" }, "followRedirects": { "title": "Follow redirects", "type": "boolean", "description": "Follow redirects and report the final URL plus redirect chain.", "default": true }, "maxRedirects": { "title": "Max redirects", "minimum": 0, "maximum": 30, "type": "integer", "description": "Maximum redirect hops to follow for each URL.", "default": 10 }, "includeSeoMeta": { "title": "Include SEO meta", "type": "boolean", "description": "For HTML responses, parse title, meta description, canonical URL, and robots meta. Disable for fastest status-only runs.", "default": true }, "includeResponseHeaders": { "title": "Include response headers", "type": "boolean", "description": "Include raw response headers in each dataset item. Disable for smaller exports.", "default": false }, "userAgent": { "title": "User-Agent", "type": "string", "description": "Optional custom User-Agent string sent to target URLs." }, "proxyConfiguration": { "title": "Proxy configuration", "type": "object", "description": "Optional Apify Proxy settings. Off by default; enable only when you need proxy routing for target domains.", "default": { "useApifyProxy": false } } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```