Lighthouse CI Regression Gate

Lighthouse CI performance regression and pass/fail gate for one to ten caller-authorised public URLs. It compares current Lighthouse category scores and Core Web Vitals with explicit caller-supplied baselines, then returns SNAPSHOT, PASS, or REGRESSION per URL.

This is an engineering signal, not a WCAG conformance test, legal opinion, usability study, or guarantee of search/performance outcomes.

Why it exists

One-off scores are noisy and easy to ignore. A release gate answers the narrower question: did this deployment get materially worse than the baseline we chose?

The Actor checks:

• Lighthouse performance, accessibility, best-practices and SEO category scores;
• Largest Contentful Paint, Cumulative Layout Shift and Total Blocking Time deltas;
• optional minimum category scores; and
• configurable regression tolerances so tiny measurement variation does not fail a build.

Use cases

• Release and deployment gates: fail a CI job when a new release crosses the performance or loading-metric tolerances your team selected.
• Performance-budget checks: enforce minimum Lighthouse scores and maximum changes in LCP, CLS and TBT without silently replacing the baseline.
• Client-site maintenance: let an agency recheck a bounded set of authorised public pages after a CMS, theme, tag-manager or dependency release.
• Accessibility change detection: flag a fall in Lighthouse's automated accessibility score so the team can investigate it alongside manual WCAG testing.
• EAA-readiness engineering: use repeatable automated checks as one input to a broader WCAG 2.2 AA / EN 301 549 review. This Actor does not determine legal scope or establish conformance.

Input

The Store form starts with a publisher-owned demonstration URL so Apify's automated quality test can exercise the real browser path without asserting permission on behalf of another site owner. That exact demo is the only URL allowed without confirmPermission: true; replace it with a public URL you control or may audit and affirm permission explicitly.

{
  "startUrls": [{ "url": "https://example.com/" }],
  "confirmPermission": true,
  "device": "mobile",
  "baselines": [
    {
      "url": "https://example.com/",
      "scores": {
        "performance": 92,
        "accessibility": 100,
        "bestPractices": 96,
        "seo": 100
      },
      "metrics": {
        "lcpMs": 1800,
        "cls": 0.02,
        "tbtMs": 100
      }
    }
  ],
  "thresholds": {
    "maxScoreDrop": 5,
    "maxLcpIncreaseMs": 500,
    "maxClsIncrease": 0.05,
    "maxTbtIncreaseMs": 200
  },
  "minimumScores": {
    "performance": 80,
    "accessibility": 90,
    "bestPractices": 80,
    "seo": 80
  },
  "failRunOnRegression": true
}

Run once without baselines to obtain the first snapshot. Copy the returned scores and metrics into the next task/run input. Keeping the baseline explicit prevents a hidden mutable state from silently changing the standard a release is judged against.

Use through Apify MCP

AI agents can discover this Actor through Apify's hosted MCP server at https://mcp.apify.com. Connect the official Apify MCP integration, search the Actor Store for Lighthouse CI Regression Gate, inspect its input schema, and add the Actor as a tool. Running an Actor requires the user's own Apify authentication and uses the same pay-per-event pricing and input safeguards described below.

GitHub Actions release gate

A dedicated GitHub Actions template repository contains the same capped workflow, setup instructions, and a Use this template path.

A complete copy-ready workflow is below and is also kept in examples/github-actions.yml in the source package:

name: Lighthouse release regression

on:
  workflow_dispatch:

permissions:
  contents: read

jobs:
  lighthouse-gate:
    runs-on: ubuntu-latest
    timeout-minutes: 7
    env:
      # Replace with a public URL you control or have permission to test.
      AUDIT_URL: https://hero988.github.io/-five-page-focus/
      APIFY_TOKEN: ${{ secrets.APIFY_TOKEN }}
    steps:
      - name: Run the capped regression gate
        shell: bash
        run: |
          set -euo pipefail
          test -n "${APIFY_TOKEN}"

          # Replace these baseline values with a snapshot from your URL.
          jq -n --arg url "${AUDIT_URL}" '{
            startUrls: [{url: $url}],
            confirmPermission: true,
            device: "desktop",
            baselines: [{
              url: $url,
              scores: {
                performance: 100,
                accessibility: 100,
                bestPractices: 96,
                seo: 100
              },
              metrics: {lcpMs: 251, cls: 0, tbtMs: 0}
            }],
            thresholds: {
              maxScoreDrop: 5,
              maxLcpIncreaseMs: 500,
              maxClsIncrease: 0.05,
              maxTbtIncreaseMs: 200
            },
            minimumScores: {
              performance: 80,
              accessibility: 90,
              bestPractices: 80,
              seo: 80
            },
            failRunOnRegression: true
          }' > actor-input.json

          curl --fail-with-body --silent --show-error --max-time 330 \
            --request POST \
            --header "Authorization: Bearer ${APIFY_TOKEN}" \
            --header "Content-Type: application/json" \
            --data-binary @actor-input.json \
            "https://api.apify.com/v2/acts/hero98~lighthouse-release-regression-gate/run-sync-get-dataset-items?timeout=300&maxTotalChargeUsd=0.05" \
            > actor-result.json

          jq -e 'length == 1 and .[0].verdict == "PASS"' actor-result.json

The workflow:

• keeps the Apify token in a repository secret named APIFY_TOKEN and sends it in an authorization header rather than a URL;
• confirms that the configured public target is authorised;
• supplies explicit baseline scores, metrics and tolerances;
• caps a one-URL run at $0.05; and
• fails the job unless the returned Dataset has exactly one PASS verdict.

Copy the workflow into .github/workflows/lighthouse-regression.yml, replace AUDIT_URL and the example baseline with a snapshot from your own site, and add the token under Settings → Secrets and variables → Actions. Start with workflow_dispatch; once the result is stable, attach the job to the deployment event that should be gated. GitHub does not pass repository secrets to workflows triggered from forks, so do not weaken that boundary or print the token for pull-request tests.

Output and billing

One Dataset item is returned per successfully audited URL. OUTPUT contains aggregate counts and uncharged errors. Invalid, private/local, unresolved, robots-disallowed and failed URLs create no Dataset result and no paid event.

Every completed result uses the configured url-audited event. Invalid, private/local, unresolved, robots-disallowed and failed URLs remain uncharged.

Store pricing:

• event name: url-audited
• title: URL audited
• price: $0.05
• primary event: yes
• platform usage: paid by the user

Set a maximum cost per run when calling the Actor if you need a hard spending cap. A one-URL run can be capped at $0.05; a ten-URL run can be capped at $0.50. The separate Apify platform-usage cost depends on the run resources and is shown by Apify.

Uncharged execution is restricted in code to private CLI, DEVELOPMENT, or TEST runs. If the platform does not supply an active pay-per-event pricing record, normal API, web, scheduler and MCP runs fail closed rather than giving away an unmetered result.

The Actor uses Apify's lightweight official API client rather than importing a crawler stack it does not need. It deliberately stores no credentials, cookies, authenticated pages, page HTML, or screenshots. Apify's normal per-run Dataset and key-value retention still applies to the returned scores, requested/final URLs, regressions and aggregate errors.

Responsible use

• Test only public URLs you control or have permission to audit.
• Do not submit credentials, account/dashboard URLs, private-network hosts or sensitive query strings.
• The Actor rejects local/private DNS targets and respects an applicable robots.txt disallow rule for its declared user agent.
• It does not circumvent access controls, rate limits, anti-bot measures or robots.
• Lighthouse and automated accessibility scores cannot establish WCAG conformance.

Frequently asked questions

Does this replace Lighthouse CI?

No. It is a hosted, bounded pass/fail workflow for teams that want an Apify API, task, scheduler or MCP route with explicit baselines. If the open-source Lighthouse CI stack already fits your infrastructure, keep using it.

Can this certify WCAG 2.2 AA, EN 301 549 or EAA compliance?

No. Automated tools cannot determine accessibility or legal compliance. Use the accessibility score and reported audits as engineering signals, then complete manual keyboard, screen-reader, zoom/reflow and content checks with a qualified human review.

Does the Actor choose or update my baseline?

No. The caller supplies the baseline on every comparison run. A first run without a baseline returns SNAPSHOT; it does not silently create a mutable standard.

What causes a paid event?

Only a successfully completed URL that stores a result creates one url-audited event. Invalid, private, disallowed, unreachable and failed URLs do not create that event. Apify platform usage is separate.

Can I test authenticated or private pages?

No. The Actor accepts only public pages the caller confirms they control or may test. It rejects credential-bearing URLs and local or private-network targets.

Local development

npm ci
npm test
npm run test:integration

The integration test audits the public Five Page Focus demonstration by default. Set TEST_URL to another authorised public page when needed.