HTTP Compression Checker

HTTP Compression Checker

Audit one or more URLs to see whether they serve compressed HTTP responses (Brotli / Gzip / Deflate) and whether they include Vary: Accept-Encoding when serving compressed variants.

This is useful for:

• diagnosing missing compression on HTML/CSS/JS/JSON
• verifying Brotli is enabled on CDNs
• catching cache misconfiguration when Vary: Accept-Encoding is missing

What the actor checks

• Content-Encoding (best encoding: br > gzip > deflate > identity)
• Content-Type (to determine whether the response is likely compressible)
• Vary includes accept-encoding when compression is used

Input

• Start URLs (startUrls) – required
• Max URLs (maxUrls)
• Request timeout (timeoutSecs)
• Follow redirects (followRedirects) and Max redirects (maxRedirects)
• Range header (rangeHeader) – defaults to bytes=0-2047 to reduce download size
• Min Content-Length for warning (minContentLengthForWarning) – avoids warning on tiny responses
• Check Vary: Accept-Encoding (checkVaryAcceptEncoding)
• User-Agent (userAgent)
• Proxy configuration (proxyConfiguration)

Output

• Dataset: one item per URL with fields like usedEncoding, compressionScore, issues, and a small headersSample.
• Key-value store:
  • SUMMARY – aggregate stats
  • REPORT – summary + top issues

Notes

• Some servers may not compress very small responses even if compression is enabled.
• A Range request may change server behavior on some origins/CDNs.

Run locally

npm install
npm start

Deploy

$apify push --no-prompt

Example input

{
  "startUrls": [{ "url": "https://example.com" }],
  "maxUrls": 10,
  "timeoutSecs": 30,
  "followRedirects": true,
  "maxRedirects": 10,
  "rangeHeader": "bytes=0-2047",
  "minContentLengthForWarning": 1024,
  "checkVaryAcceptEncoding": true,
  "maxConcurrency": 10,
  "proxyConfiguration": { "useApifyProxy": false }
}

Quick start

Store page: https://apify.com/scrappy_garden/http-compression-checker

Paste this into Input and click Run:

{
  "startUrls": [
    {
      "url": "https://example.com/"
    }
  ],
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}

Outputs (what you get)

• Dataset: Dataset items typically include fields like: startUrl, finalUrl, statusCode, redirected, usedEncoding, compressionScore, warningCount, errorCount, issues, checkedAt.
• Key-value store: REPORT, SUMMARY

Tips (trust + predictable results)

• Start with 1–3 URLs to validate behavior, then scale up.
• If a target blocks requests, enable Proxy and/or slow down concurrency in Input.
• Use the SUMMARY / REPORT keys (when present) for automation pipelines and monitoring.

Related actors

• broken-link-checker (https://apify.com/scrappy_garden/broken-link-checker)
• canonical-url-checker (https://apify.com/scrappy_garden/canonical-url-checker)
• meta-tag-analyzer (https://apify.com/scrappy_garden/meta-tag-analyzer)
• security-headers-checker (https://apify.com/scrappy_garden/security-headers-checker)

Search keywords

http compression checker, http compression checker - verify brotli/gzip and vary: accept-encoding, website audit, seo