Bulk HTTP Header Checker — Security & Response Analysis

Bulk HTTP Header Checker — Security & Response Analysis

Check HTTP response headers for multiple URLs at once with deep security header analysis, redirect chain tracking, and response timing. Whether you're auditing website security, debugging redirect issues, or monitoring server configurations, this actor processes dozens of URLs in a single run.

What does it do?

This actor takes one or more URLs and performs comprehensive HTTP header analysis on each. It captures the full response header set, inspects security headers like Content-Security-Policy and Strict-Transport-Security, tracks the complete redirect chain, and measures response time.

Each URL is checked concurrently with configurable concurrency, and individual URL failures never block the rest of the batch. Results are streamed to the dataset as each URL completes, making it suitable for real-time monitoring dashboards and automated security audits.

Features

1. Batch processing — Check dozens of URLs in a single run with concurrent connections.
2. Security header detection — Automatically detects and reports on 10 key security headers: CSP, HSTS, X-Frame-Options, X-Content-Type-Options, X-XSS-Protection, Referrer-Policy, Permissions-Policy, Cross-Origin-Opener-Policy, Cross-Origin-Embedder-Policy, and Cross-Origin-Resource-Policy.
3. Redirect chain tracking — Follows up to 10 redirects and records every intermediate URL with its status code.
4. Response timing — Measures total response time in milliseconds for performance benchmarking.
5. Comprehensive header capture — Returns all HTTP response headers as a structured object.
6. Error resilience — Timeouts, connection failures, and HTTP errors are captured per-URL without affecting other checks.

Why use this?

Problem	Solution
You need to audit security headers across your entire domain portfolio	Run all URLs in one batch and get a security header report for each
You're debugging a redirect chain that's causing SEO issues	See every redirect hop with status codes for each intermediate URL
You want to monitor server response times across multiple endpoints	Get millisecond-precision response times for every URL
You're migrating infrastructure and need to verify headers are correct	Check old and new URLs side by side with full header comparison
You maintain a CDN configuration and need to verify cache headers	Inspect Cache-Control, Age, and CDN-specific headers from multiple origins

Who is it for?

Persona	What they use it for
Security Engineer	Auditing security headers across a fleet of web properties
DevOps Engineer	Verifying redirect configurations after infrastructure changes
SEO Specialist	Checking redirect chains and HTTP status codes for indexed URLs
Web Developer	Debugging CORS headers, cache configuration, and server responses
System Administrator	Monitoring server header consistency across load-balanced environments
QA Engineer	Verifying HTTP response behavior across staging, testing, and production

Input Parameters

Field	Type	Default	Description
urls	array	—	Array of URLs to check. Scheme defaults to https:// if omitted.
url	string	—	Single URL to check (alternative to the urls array).
followRedirects	boolean	true	Whether to follow HTTP redirects and report the final destination.
maxConcurrency	integer	10	Maximum simultaneous connections. Range: 1-30. Lower for rate-limited targets.

Example Input JSON

{
  "urls": [
    "example.com",
    "httpbin.org/status/301",
    "twitter.com",
    "github.com/404-page"
  ],
  "followRedirects": true,
  "maxConcurrency": 15
}

Example Single URL Input

{
  "url": "google.com",
  "followRedirects": true
}

Output Format

Field	Type	Description
url	string	The final URL after redirects (or original URL if no redirects)
statusCode	integer	HTTP response status code (e.g., 200, 301, 404, 500)
statusText	string	HTTP status text (e.g., "OK", "Moved Permanently")
headers	object	All HTTP response headers as key-value pairs
securityHeaders	object	Security header analysis (see below)
redirectChain	array	Redirect chain entries: {url, statusCode}
responseTimeMs	number	Total response time in milliseconds
contentType	string	Value of the Content-Type response header
contentLength	string	Value of the Content-Length response header
server	string	Value of the Server response header
error	string or null	Error message if the check failed

Each entry in securityHeaders:

Field	Type	Description
present	boolean	Whether the security header was found in the response
value	string or null	The header value if present

Example Output JSON

{
  "url": "https://example.com",
  "statusCode": 200,
  "statusText": "OK",
  "headers": {
    "content-type": "text/html; charset=UTF-8",
    "server": "ECS (dcb/7F2E)",
    "cache-control": "max-age=604800"
  },
  "securityHeaders": {
    "CSP": {"present": false, "value": null},
    "HSTS": {"present": false, "value": null},
    "XFO": {"present": false, "value": null}
  },
  "redirectChain": [],
  "responseTimeMs": 142.3,
  "contentType": "text/html; charset=UTF-8",
  "contentLength": "1256",
  "server": "ECS (dcb/7F2E)",
  "error": null
}

API Usage

cURL

curl -X POST "https://api.apify.com/v2/acts/perryay~bulk-http-header-checker/runs?token=YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["example.com", "google.com"]}'

Python (ApifyClient)

from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")
run = client.actor("perryay~bulk-http-header-checker").call(
    run_input={"urls": ["example.com", "google.com"]}
)
dataset = client.dataset(run["defaultDatasetId"]).list_items()
for item in dataset.items:
    print(f'{item["url"]}: {item["statusCode"]} ({item["responseTimeMs"]}ms)')

Use Cases

1. Security header audit — Run a batch of your company's domains through the checker and identify which ones are missing critical security headers like Content-Security-Policy and Strict-Transport-Security.

2. Redirect chain debugging — An SEO audit reveals 302 redirects on pages that should return 301. Point the checker at affected URLs and inspect the exact redirect chain with status codes.

3. CDN configuration verification — After updating CDN cache rules, verify that Cache-Control and Age headers are correct across all edge servers by checking multiple geographic endpoints.

4. Server migration validation — During infrastructure migration, run the same URLs against old and new servers to compare header sets and confirm all expected headers are present.

5. Load balancer consistency check — When running behind multiple load balancers, check several URLs that route to different backends to verify server headers are consistent.

6. API endpoint monitoring — Periodically check API endpoints for unexpected header changes (e.g., missing CORS headers, changed content-type) that could break integrations.

7. Certificate migration verification — After updating TLS certificates, check URLs for HSTS headers and ensure the upgrade path is working correctly.

8. Performance benchmarking — Measure response times across multiple URLs or CDN endpoints to identify slow paths or inconsistent performance.

FAQ

Q: Can I check URLs without typing https://? A: Yes. If a URL doesn't have a scheme prefix, https:// is added automatically. For example, example.com becomes https://example.com.

Q: How many URLs can I check in one run? A: There is no hard limit, but runs are subject to the actor's timeout (default 300 seconds). With default concurrency of 10, you can typically check 100-200 URLs within the timeout window.

Q: What security headers are checked? A: The actor checks for 10 security headers: Content-Security-Policy (CSP), Strict-Transport-Security (HSTS), X-Frame-Options (XFO), X-Content-Type-Options (XCTO), X-XSS-Protection, Referrer-Policy, Permissions-Policy, Cross-Origin-Opener-Policy (COOP), Cross-Origin-Embedder-Policy (COEP), and Cross-Origin-Resource-Policy (CORP).

Q: Does the actor follow JavaScript redirects? A: No. It follows standard HTTP redirects (301, 302, 303, 307, 308) up to a maximum of 10 hops. Meta refresh and JavaScript-based redirects are not followed.

Q: What happens when a URL times out? A: The check returns with an error field set to "Request timed out" and the response time recorded as the time elapsed before timeout. Other URLs in the batch continue processing.

Q: Can I disable redirect following? A: Yes. Set followRedirects to false. The actor will return the intermediate redirect response headers and status code without following through.

Q: Are the results available as they come in, or only after all URLs finish? A: Results are streamed to the dataset as each URL completes. You can start reading the dataset while the run is still in progress.

Q: What does the security header analysis tell me? A: For each of the 10 tracked security headers, the actor reports whether the header is present and its value. This lets you quickly identify which security headers are missing from a URL's response.

Q: Does this actor make any changes to the URLs being checked? A: No. This is a read-only analysis tool. It only fetches HTTP headers and does not modify, submit data to, or interact with the target URLs beyond a GET request.

Q: What's the difference between this and the single-URL URL Health actor? A: URL Health checks basic URL status (200/404/error) with SSL expiry, while this actor provides comprehensive header-level analysis including security headers, redirect chains, and detailed response metadata — optimized for batch processing.

Usage & Billing

This actor uses Apify's PAY_PER_EVENT pricing model. You are charged per event:

Event Name	Price (USD)	Trigger
apify-actor-start	$0.025	Charged on every actor start
batch-header-check	$0.010	Charged per URL checked
security-audit	$0.005	Charged when security headers are analyzed
redirect-trace	$0.005	Charged when redirects are detected

Platform infrastructure costs (Apify's compute and storage) are passed through at cost.

MCP Integration

This actor can be used through the Apify MCP server. Once connected, your MCP client (Claude Desktop, Cursor, etc.) can discover and run this actor from the Apify Store.

Quick Start

1. Install the Apify connector in your MCP client:

   • Claude Desktop: Search for "Apify" in the connector directory, or use the remote server at https://mcp.apify.com
   • Other clients: See the Apify MCP server docs for setup instructions

2. Ask your AI assistant to use the actor.

Claude Desktop Configuration

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com"
    }
  }
}

Bearer Token Alternative

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com",
      "headers": {
        "Authorization": "Bearer <APIFY_TOKEN>"
      }
    }
  }
}

Related Tools

• URL Health — Check URL status, SSL certificate validity, and response times for individual URLs.
• HTTP Security Headers Analyzer — Deep analysis of HTTP security headers for individual URLs with detailed recommendations.
• Link Quality Analyzer — Comprehensive link analysis including status, redirect chains, and security posture.