Bulk URL Status Checker

Check public URL lists for status codes, broken-link flags, followed redirects, response timing and optional page metadata. Bulk URL Status Checker accepts direct URLs, pasted text, a hosted list or a sitemap and saves a dataset you can export to CSV, Excel or JSON.

It is a direct HTTP checker, not a browser: it does not render JavaScript, sign in or bypass access restrictions. Start with a small list of public pages.

Quick start

Paste URLs into the input form, set a small maxUrls, then start the Actor and open its dataset.

{
  "urls": ["https://example.com/", "https://www.iana.org/domains/example"],
  "maxUrls": 20,
  "concurrency": 5,
  "includeSeoMeta": true
}

The default method reads HTML when SEO metadata is enabled. Status-only checks can use includeSeoMeta: false. Read the input-field help before changing methods or redirect limits.

Prefer a video guide? This short Apify walkthrough shows you how to set up and run an Actor. It uses a different Actor, but the setup steps are the same.

What you get

• URL status snapshots: status codes, response timing and honest per-URL request errors.
• Redirect details: observed destinations and followed hop chains within your chosen limit.
• Optional page metadata: title, description, canonical URL and robots meta from returned HTML.
• Flexible URL sources: direct URLs, pasted text, a public hosted list or one sitemap file.
• Progress records: completed dataset rows and run-summary counts, including pending work.

Who is it for?

• SEO teams: inspect status codes, canonical URLs and robots meta on known page lists.
• Migration teams: compare old URLs with their observed redirect destinations.
• QA and content teams: find rejected, missing or slow landing pages before a launch.
• Developers: feed structured status snapshots into dashboards and deployment checks.

Input settings

Setting	JSON key	Purpose
URLs	urls	Direct URL list. The form supplies sample URLs.
URLs as text	urlsText	URLs separated by whitespace, commas or semicolons.
Hosted URL list	listUrl	Public text or CSV-like file containing URL tokens.
Sitemap URL	sitemapUrl	Public XML file containing page <loc> entries.
Max URLs	maxUrls	Limit after identical input strings are removed. Set explicitly in API inputs.
Include SEO meta	includeSeoMeta	Read title, description, canonical and robots meta when available.
Concurrency	concurrency	Parallel checks. Default 20.
Request timeout	timeoutSecs	Per-request timeout. Default 15 seconds.
Request method	method	head-get-fallback, get or head.
Follow redirects	followRedirects	Follow supported redirects. Default true.
Max redirects	maxRedirects	Maximum followed hops. Default 10.
Include headers	includeResponseHeaders	Optional response headers, with sensitive values redacted.
User-Agent	userAgent	Optional custom User-Agent string.

The form/schema default for maxUrls is 20. The raw runtime fallback when it receives no value is 1000. Explicit limits make API and saved-task inputs predictable. All connections are direct. Proxy routing is not supported.

Input recipes

These are JSON inputs, not public saved-task links. Supply at least one URL source. Multiple sources are combined.

Check a pasted URL list:

{
  "urlsText": "https://example.com/\nhttps://www.iana.org/domains/example",
  "maxUrls": 10,
  "includeSeoMeta": false
}

Inspect redirect responses without following them:

{
  "urls": ["http://example.com/"],
  "maxUrls": 1,
  "followRedirects": false,
  "includeSeoMeta": false
}

For a hosted list, set listUrl to a public text file of URLs. For a sitemap, set sitemapUrl to a public page sitemap. Only the <loc> entries in that file are checked, not recursively expanded child sitemaps.

Output data

One row is saved per completed URL check, including HTTP error responses and request failures reached during checking. Rows arrive as checks finish, not in guaranteed input order. The following shortened example shows the output shape, not a fresh measurement:

{
  "inputUrl": "https://example.com/",
  "finalUrl": "https://example.com/",
  "statusCode": 200,
  "ok": true,
  "isBroken": false,
  "isRedirect": false,
  "redirectCount": 0,
  "title": "Example Domain",
  "canonicalUrl": null,
  "errorMessage": null
}

Fields	Meaning
inputUrl, normalizedUrl	Supplied URL and normalized URL with scheme added if missing and fragment removed.
finalUrl	Last checked destination. It may still return a redirect when following is off or the hop limit is reached.
statusCode, statusText	Observed status and common status label, or null when unavailable.
ok, isBroken	ok means 2xx/3xx. isBroken means a request error or status 400+, including blocked requests. It is not proof a page was deleted.
isRedirect, redirectCount, redirectChain	Whether hops were followed, how many, and their source URL/status/location.
responseTimeMs	Elapsed check time, including followed hops and method fallback.
contentType, contentLength	Response header values when supplied by the site.
title, metaDescription, canonicalUrl, robotsMeta	Optional HTML metadata. Missing or unavailable values are null.
errorType, errorMessage	Request-error classification and sanitized message, otherwise null.
responseHeaders	Optional redacted response headers on completed responses.
checkedAt	ISO timestamp when the check began.

The key-value store can also contain RUN_SUMMARY with completed/pending counts and PENDING_WORK while work remains.

Download your results

Open your run's dataset, choose Export and select JSON, CSV or Excel. JSON keeps nested fields together. CSV and Excel are useful when you want to review the records in a spreadsheet.

See Apify's download and export guide for the available formats and field options.

Pricing

This Actor uses pay-per-event pricing. start is a one-time run-start event.

The url event counts attempted URL checks saved as dataset rows. HTTP error/status rows and saved request-failure rows count too.

See the live Pricing tab for current rates and discounts.

Tips for best results

• Start with a few URLs. Set maxUrls explicitly so API inputs stay bounded.
• Match the method to the goal. Keep SEO metadata enabled for returned HTML fields, or disable it for status-only checks.
• Use modest concurrency. Lower it when sites refuse requests or rate-limit your list.
• Read the run summary. Compare completed and pending counts before treating a large audit as complete.

Limits and caveats

• Bound each input: set maxUrls, use modest concurrency and begin with a few URLs. The maximum accepted maxUrls is 100000, not a promise that every URL finishes in one run.
• Use public targets: only HTTP(S) on standard ports is supported. Private hosts, embedded credentials and credential-like query parameters are rejected. Invalid or security-refused supplied targets fail initial validation. A valid public-host URL with no DNS records instead produces an error row (statusCode: null, ok: false, isBroken: true) without discarding other URL results. This is not proof that the page was deleted. Resolver outages and essential storage or billing failures still fail the run.
• Understand partial runs: completed rows are saved progressively. Remaining work can resume when the same run is restarted with the same input. Check RUN_SUMMARY rather than assuming every supplied URL was checked.
• Expect site differences: 403/429 can indicate refusal or rate limiting. They do not establish privacy, deletion or an extraction bypass. HTML metadata is not JavaScript-rendered content.
• Check redirect bounds: isRedirect describes followed hops, not every 3xx response. A last observed redirect is not necessarily the ultimate destination.

Automation and integrations

Save your settings for next time

Once you have an input that works for you, save it as an Apify task. This keeps your settings together so you can use them again without filling in the form from scratch.

Run on a schedule

Need fresh results regularly? You can schedule the Actor or a saved task to run at a time that suits you.

Each scheduled run collects the data again. If you want to compare results or highlight changes, add that step in your own workflow.

Send results to your other tools

Use n8n, Make or Zapier to connect this scraper to the rest of your workflow. For example, you could pass its results to an AI assistant or add records to a spreadsheet. Follow the Apify integration guides for your chosen tool.

For a custom connection, an Apify webhook can notify your app when a run finishes or fails. Your app can then fetch the results or handle the failure.

API usage

Want to collect this data from your own app or automation? The examples below send input to the scraper and show how to collect its results.

The Node.js and Python examples wait for the run to finish before reading its results. The cURL example starts the run and returns its details. Wait for that run to finish, then use its defaultDatasetId to fetch the results.

For setup instructions, see Apify's JavaScript client guide, Python client guide and Actor runs API reference.

cURL

curl -X POST 'https://api.apify.com/v2/acts/fetch_cat~bulk-url-status-checker/runs' \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"urls":["https://example.com/"],"maxUrls":1,"concurrency":1}'

Node.js

import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('fetch_cat/bulk-url-status-checker').call({
  urls: ['https://example.com/'], maxUrls: 1, concurrency: 1,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Python

import os
from apify_client import ApifyClient
client = ApifyClient(os.environ['APIFY_TOKEN'])
run = client.actor('fetch_cat/bulk-url-status-checker').call(run_input={
    'urls': ['https://example.com/'], 'maxUrls': 1, 'concurrency': 1,
})
print(client.dataset(run['defaultDatasetId']).list_items().items)

MCP and AI agents

You can ask your AI assistant to collect data with Bulk URL Status Checker and help you work with the results. The scraper returns the data, and your assistant can then summarize it, compare records or prepare notes.

Connect your assistant through the Apify MCP server. MCP gives your assistant a way to find this scraper, run it and read the results. Follow the official setup and sign-in guide for your app.

Easier setup with an Apify plugin

If you use a supported coding assistant, the official Apify plugin can set up the connection for you. It connects your assistant to Apify and includes instructions for common tasks. Follow the guide for your particular app.

Use Apify AI in Console

Prefer to keep everything in Apify? Open Apify AI in Console and ask it to find Fetch Cat's Bulk URL Status Checker. You can describe the data you want and ask questions about the results.

Apify AI is currently in beta. It asks for confirmation before starting an Actor, and normal run charges apply.

Choose your assistant

• Claude and Claude Code (Anthropic): Apify setup guide.
• ChatGPT and Codex (OpenAI): Apify setup guides.
• Grok Bot (xAI): Apify plugin guide.
• Gemini CLI (Google): MCP server setup guide. Use the server URL below with the HTTP configuration supported by Gemini CLI.

Connect just this scraper

To keep your setup focused on this scraper, add this server configuration to a compatible MCP client:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=fetch_cat/bulk-url-status-checker"
    }
  }
}

Sign in to Apify when your client asks. If your client uses a Bearer token instead, follow its setup guide and keep the token in its secure configuration. Configuration formats vary by app, so use the linked guide if your client needs a different format.

Use Apify's general server

Already connected to the general Apify MCP server at https://mcp.apify.com? You can use it to find this scraper without adding another server. Ask your assistant to use Bulk URL Status Checker by Fetch Cat, or give it the identifier fetch_cat/bulk-url-status-checker.

Your assistant can check the available settings and retrieve the results from the returned run. If the run is still working, it should follow that run rather than start it again. Both connection options use Apify's hosted service.

Example requests

• “Collect status codes for these public landing pages and summarize which returned successful responses and which reported errors.”
• “Read this page sitemap, collect status results for up to twenty URLs and list their observed redirect destinations.”
• “Extract page titles, canonical URLs and status codes from this short list of public pages, then organize the results in a table.”

Normal Apify charges apply. Check the live Pricing tab before asking your assistant to run a larger batch.

Pay with x402

If your AI agent has a funded wallet, it can pay for Apify usage through x402. This experimental option lets an agent use Apify without a conventional account.

Apify AGI provides a prepaid API token with a spending cap after payment. Your agent can use that token to run this scraper through the Apify API or an authenticated MCP connection.

Some MCP setups also support paying for each request directly. That is a different payment option, so check the official guide for its availability.

Follow the official guide for wallet setup, supported payment options and current terms. Apify handles the payments. Keep wallet keys and payment tokens in your agent's secure configuration.

Other ways for agents to pay

Apify also supports MPP for prepaid access through AGI and Skyfire for eligible Actors. If your agent already uses one of these, see the agentic payment guide to choose the right setup.

FAQ

Can I check thousands of URLs or an entire sitemap?

Increase maxUrls for a larger list, while keeping concurrency appropriate for the sites. A sitemap input reads only that file's <loc> entries. It does not recursively traverse a sitemap index. Large lists may leave pending work at the run deadline.

Why are page titles empty in a status-only audit?

Metadata must be enabled and present in the returned HTML. HEAD-only responses usually contain no HTML body, and this Actor does not render JavaScript. Missing metadata is represented by null.

What can I do about 403 or 429 responses?

Try lower concurrency, GET-only mode or an appropriate User-Agent. Access may still be refused. Proxy routing and login are not supported. These responses do not prove the URL is missing.

Can I export results or call this from an application?

Yes. Use Apify dataset exports for CSV, Excel or JSON, or the API/MCP instructions above. This is a point-in-time check, not a built-in recurring monitoring service.

Related Actors

• Sitemap URL Extractor — prepare a page URL list for a status audit.
• Website Contact Finder — inspect contact information on selected websites.
• Bing Search Results Scraper — discover URLs before checking them.
• Website Content Crawler Lite — collect public page text and available links from selected websites after checking their URLs.
• Website Screenshot Generator — inspect a page's visual appearance after checking its status.

Support and feedback

For custom additions or integrations, email hello@fetchcat.dev.

Found an error or have a feature request? Open an issue with the run URL, a non-sensitive input example and what you expected.

Need help sharing it? See Apify’s guide to sharing a run when reporting an issue. Sharing a run can give the developer access to its input, logs and results, so use a run that contains no sensitive data. Please leave out API tokens, passwords, cookies, private documents and personal data. Use a public sample URL or a redacted example instead.

Tried this scraper? We'd love to hear how it worked for you. Leave a review to help others choose the right tool and help us improve it. Your feedback means a lot to us.

Made with care for curious people, AI agents and the possibilities in data. ♥