# Bulk HTTP Header Checker — Security & Response Analysis (`perryay/bulk-http-header-checker`) Actor Check HTTP response headers for multiple URLs in batch. Analyzes security headers (CSP, HSTS, XFO), redirect chains, response times, and server info. - **URL**: https://apify.com/perryay/bulk-http-header-checker.md - **Developed by:** [Perry AY](https://apify.com/perryay) (community) - **Categories:** Developer tools - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.025 / actor start This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Bulk HTTP Header Checker — Security & Response Analysis ### Bulk HTTP Header Checker — Security & Response Analysis **Check HTTP response headers for multiple URLs at once with deep security header analysis, redirect chain tracking, and response timing.** Whether you're auditing website security, debugging redirect issues, or monitoring server configurations, this actor processes dozens of URLs in a single run. --- ### What does it do? This actor takes one or more URLs and performs comprehensive HTTP header analysis on each. It captures the full response header set, inspects security headers like Content-Security-Policy and Strict-Transport-Security, tracks the complete redirect chain, and measures response time. Each URL is checked concurrently with configurable concurrency, and individual URL failures never block the rest of the batch. Results are streamed to the dataset as each URL completes, making it suitable for real-time monitoring dashboards and automated security audits. ### Features 1. **Batch processing** — Check dozens of URLs in a single run with concurrent connections. 2. **Security header detection** — Automatically detects and reports on 10 key security headers: CSP, HSTS, X-Frame-Options, X-Content-Type-Options, X-XSS-Protection, Referrer-Policy, Permissions-Policy, Cross-Origin-Opener-Policy, Cross-Origin-Embedder-Policy, and Cross-Origin-Resource-Policy. 3. **Redirect chain tracking** — Follows up to 10 redirects and records every intermediate URL with its status code. 4. **Response timing** — Measures total response time in milliseconds for performance benchmarking. 5. **Comprehensive header capture** — Returns all HTTP response headers as a structured object. 6. **Error resilience** — Timeouts, connection failures, and HTTP errors are captured per-URL without affecting other checks. ### Why use this? | Problem | Solution | |---------|----------| | You need to audit security headers across your entire domain portfolio | Run all URLs in one batch and get a security header report for each | | You're debugging a redirect chain that's causing SEO issues | See every redirect hop with status codes for each intermediate URL | | You want to monitor server response times across multiple endpoints | Get millisecond-precision response times for every URL | | You're migrating infrastructure and need to verify headers are correct | Check old and new URLs side by side with full header comparison | | You maintain a CDN configuration and need to verify cache headers | Inspect Cache-Control, Age, and CDN-specific headers from multiple origins | ### Who is it for? | Persona | What they use it for | |---------|---------------------| | Security Engineer | Auditing security headers across a fleet of web properties | | DevOps Engineer | Verifying redirect configurations after infrastructure changes | | SEO Specialist | Checking redirect chains and HTTP status codes for indexed URLs | | Web Developer | Debugging CORS headers, cache configuration, and server responses | | System Administrator | Monitoring server header consistency across load-balanced environments | | QA Engineer | Verifying HTTP response behavior across staging, testing, and production | ### Input Parameters | Field | Type | Default | Description | |-------|------|---------|-------------| | `urls` | array | — | Array of URLs to check. Scheme defaults to `https://` if omitted. | | `url` | string | — | Single URL to check (alternative to the `urls` array). | | `followRedirects` | boolean | `true` | Whether to follow HTTP redirects and report the final destination. | | `maxConcurrency` | integer | `10` | Maximum simultaneous connections. Range: 1-30. Lower for rate-limited targets. | #### Example Input JSON ```json { "urls": [ "example.com", "httpbin.org/status/301", "twitter.com", "github.com/404-page" ], "followRedirects": true, "maxConcurrency": 15 } ```` #### Example Single URL Input ```json { "url": "google.com", "followRedirects": true } ``` ### Output Format | Field | Type | Description | |-------|------|-------------| | `url` | string | The final URL after redirects (or original URL if no redirects) | | `statusCode` | integer | HTTP response status code (e.g., 200, 301, 404, 500) | | `statusText` | string | HTTP status text (e.g., "OK", "Moved Permanently") | | `headers` | object | All HTTP response headers as key-value pairs | | `securityHeaders` | object | Security header analysis (see below) | | `redirectChain` | array | Redirect chain entries: `{url, statusCode}` | | `responseTimeMs` | number | Total response time in milliseconds | | `contentType` | string | Value of the Content-Type response header | | `contentLength` | string | Value of the Content-Length response header | | `server` | string | Value of the Server response header | | `error` | string or null | Error message if the check failed | Each entry in `securityHeaders`: | Field | Type | Description | |-------|------|-------------| | `present` | boolean | Whether the security header was found in the response | | `value` | string or null | The header value if present | #### Example Output JSON ```json { "url": "https://example.com", "statusCode": 200, "statusText": "OK", "headers": { "content-type": "text/html; charset=UTF-8", "server": "ECS (dcb/7F2E)", "cache-control": "max-age=604800" }, "securityHeaders": { "CSP": {"present": false, "value": null}, "HSTS": {"present": false, "value": null}, "XFO": {"present": false, "value": null} }, "redirectChain": [], "responseTimeMs": 142.3, "contentType": "text/html; charset=UTF-8", "contentLength": "1256", "server": "ECS (dcb/7F2E)", "error": null } ``` ### API Usage #### cURL ```bash curl -X POST "https://api.apify.com/v2/acts/perryay~bulk-http-header-checker/runs?token=YOUR_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{"urls": ["example.com", "google.com"]}' ``` #### Python (ApifyClient) ```python from apify_client import ApifyClient client = ApifyClient("YOUR_API_TOKEN") run = client.actor("perryay~bulk-http-header-checker").call( run_input={"urls": ["example.com", "google.com"]} ) dataset = client.dataset(run["defaultDatasetId"]).list_items() for item in dataset.items: print(f'{item["url"]}: {item["statusCode"]} ({item["responseTimeMs"]}ms)') ``` ### Use Cases 1. **Security header audit** — Run a batch of your company's domains through the checker and identify which ones are missing critical security headers like Content-Security-Policy and Strict-Transport-Security. 2. **Redirect chain debugging** — An SEO audit reveals 302 redirects on pages that should return 301. Point the checker at affected URLs and inspect the exact redirect chain with status codes. 3. **CDN configuration verification** — After updating CDN cache rules, verify that Cache-Control and Age headers are correct across all edge servers by checking multiple geographic endpoints. 4. **Server migration validation** — During infrastructure migration, run the same URLs against old and new servers to compare header sets and confirm all expected headers are present. 5. **Load balancer consistency check** — When running behind multiple load balancers, check several URLs that route to different backends to verify server headers are consistent. 6. **API endpoint monitoring** — Periodically check API endpoints for unexpected header changes (e.g., missing CORS headers, changed content-type) that could break integrations. 7. **Certificate migration verification** — After updating TLS certificates, check URLs for HSTS headers and ensure the upgrade path is working correctly. 8. **Performance benchmarking** — Measure response times across multiple URLs or CDN endpoints to identify slow paths or inconsistent performance. ### FAQ **Q: Can I check URLs without typing `https://`?** A: Yes. If a URL doesn't have a scheme prefix, `https://` is added automatically. For example, `example.com` becomes `https://example.com`. **Q: How many URLs can I check in one run?** A: There is no hard limit, but runs are subject to the actor's timeout (default 300 seconds). With default concurrency of 10, you can typically check 100-200 URLs within the timeout window. **Q: What security headers are checked?** A: The actor checks for 10 security headers: Content-Security-Policy (CSP), Strict-Transport-Security (HSTS), X-Frame-Options (XFO), X-Content-Type-Options (XCTO), X-XSS-Protection, Referrer-Policy, Permissions-Policy, Cross-Origin-Opener-Policy (COOP), Cross-Origin-Embedder-Policy (COEP), and Cross-Origin-Resource-Policy (CORP). **Q: Does the actor follow JavaScript redirects?** A: No. It follows standard HTTP redirects (301, 302, 303, 307, 308) up to a maximum of 10 hops. Meta refresh and JavaScript-based redirects are not followed. **Q: What happens when a URL times out?** A: The check returns with an error field set to "Request timed out" and the response time recorded as the time elapsed before timeout. Other URLs in the batch continue processing. **Q: Can I disable redirect following?** A: Yes. Set `followRedirects` to `false`. The actor will return the intermediate redirect response headers and status code without following through. **Q: Are the results available as they come in, or only after all URLs finish?** A: Results are streamed to the dataset as each URL completes. You can start reading the dataset while the run is still in progress. **Q: What does the security header analysis tell me?** A: For each of the 10 tracked security headers, the actor reports whether the header is present and its value. This lets you quickly identify which security headers are missing from a URL's response. **Q: Does this actor make any changes to the URLs being checked?** A: No. This is a read-only analysis tool. It only fetches HTTP headers and does not modify, submit data to, or interact with the target URLs beyond a GET request. **Q: What's the difference between this and the single-URL URL Health actor?** A: URL Health checks basic URL status (200/404/error) with SSL expiry, while this actor provides comprehensive header-level analysis including security headers, redirect chains, and detailed response metadata — optimized for batch processing. ### Usage & Billing This actor uses Apify's PAY\_PER\_EVENT pricing model. You are charged per event: | Event Name | Price (USD) | Trigger | |------------|-------------|---------| | `apify-actor-start` | $0.025 | Charged on every actor start | | `batch-header-check` | $0.010 | Charged per URL checked | | `security-audit` | $0.005 | Charged when security headers are analyzed | | `redirect-trace` | $0.005 | Charged when redirects are detected | Platform infrastructure costs (Apify's compute and storage) are passed through at cost. ### MCP Integration This actor can be used through the [Apify MCP server](https://docs.apify.com/integrations/mcp). Once connected, your MCP client (Claude Desktop, Cursor, etc.) can discover and run this actor from the Apify Store. #### Quick Start 1. **Install the Apify connector** in your MCP client: - **Claude Desktop**: Search for "Apify" in the connector directory, or use the remote server at `https://mcp.apify.com` - **Other clients**: See the [Apify MCP server docs](https://docs.apify.com/integrations/mcp) for setup instructions 2. **Ask your AI assistant** to use the actor. #### Claude Desktop Configuration ```json { "mcpServers": { "apify": { "url": "https://mcp.apify.com" } } } ``` #### Bearer Token Alternative ```json { "mcpServers": { "apify": { "url": "https://mcp.apify.com", "headers": { "Authorization": "Bearer " } } } } ``` ### Related Tools - **[URL Health](https://apify.com/perryay/url-health)** — Check URL status, SSL certificate validity, and response times for individual URLs. - **[HTTP Security Headers Analyzer](https://apify.com/perryay/http-security-headers-analyzer)** — Deep analysis of HTTP security headers for individual URLs with detailed recommendations. - **[Link Quality Analyzer](https://apify.com/perryay/link-quality-analyzer)** — Comprehensive link analysis including status, redirect chains, and security posture. # Actor input Schema ## `urls` (type: `array`): Array of URLs to check HTTP headers for ## `url` (type: `string`): A single URL to check (alternative to urls array) ## `followRedirects` (type: `boolean`): Follow HTTP redirects and report the final destination ## `maxConcurrency` (type: `number`): Maximum simultaneous connections (1-30) ## Actor input object example ```json { "urls": [ "example.com", "httpbin.org/status/301", "google.com" ], "followRedirects": true, "maxConcurrency": 10 } ``` # Actor output Schema ## `results` (type: `string`): HTTP header analysis results in the default dataset # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "urls": [ "example.com", "httpbin.org/status/301", "google.com" ], "url": "" }; // Run the Actor and wait for it to finish const run = await client.actor("perryay/bulk-http-header-checker").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "urls": [ "example.com", "httpbin.org/status/301", "google.com", ], "url": "", } # Run the Actor and wait for it to finish run = client.actor("perryay/bulk-http-header-checker").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "urls": [ "example.com", "httpbin.org/status/301", "google.com" ], "url": "" }' | apify call perryay/bulk-http-header-checker --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=perryay/bulk-http-header-checker", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Bulk HTTP Header Checker — Security & Response Analysis", "description": "Check HTTP response headers for multiple URLs in batch. Analyzes security headers (CSP, HSTS, XFO), redirect chains, response times, and server info.", "version": "1.0", "x-build-id": "YwuYpTNGRoGyWGD5h" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/perryay~bulk-http-header-checker/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-perryay-bulk-http-header-checker", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/perryay~bulk-http-header-checker/runs": { "post": { "operationId": "runs-sync-perryay-bulk-http-header-checker", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/perryay~bulk-http-header-checker/run-sync": { "post": { "operationId": "run-sync-perryay-bulk-http-header-checker", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "urls": { "title": "URLs", "type": "array", "description": "Array of URLs to check HTTP headers for" }, "url": { "title": "Single URL", "type": "string", "description": "A single URL to check (alternative to urls array)" }, "followRedirects": { "title": "Follow Redirects", "type": "boolean", "description": "Follow HTTP redirects and report the final destination", "default": true }, "maxConcurrency": { "title": "Max Concurrency", "minimum": 1, "maximum": 30, "type": "number", "description": "Maximum simultaneous connections (1-30)", "default": 10 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```