Federal Website Compliance Auditor — GSA Site Scanning API

Audit 29,000+ federal .gov websites in one run. This actor turns GSA's official Site Scanning API — the same nightly scan engine behind digital.gov's dashboards — into three ready-to-use datasets:

1. scorecard (default) — one row per federal agency: site count, % compliant on each of 8 pillars (USWDS design system, DAP analytics, HTTPS enforcement, HSTS, IPv6, mobile viewport, sitemap.xml, robots.txt), a composite 0–100 compliance score, and the agency's worst-offender domains.
2. sites — one flat row per website, filterable by agency, bureau, domain substring, and "must-fail" compliance filters (e.g. every site missing both DAP and USWDS).
3. gaps — the govtech lead-gen mode: one row per live site failing a chosen pillar, with agency, bureau, CMS, hosting provider, and traffic — a prospect list of agencies missing exactly the technology you sell.

No scraping, no headless browsers — pure official API, fast and reliable.

Who is this for?

• Govtech vendors & digital agencies — gaps mode with pillar: "dap" or "uswds" is a qualified prospect list: agency, domain, current CMS, current host, and proof they're missing what you sell.
• Federal web / compliance / CIO teams — track your agency's 21st Century IDEA & OMB-mandate posture (USWDS, DAP, HTTPS) against every other agency.
• Auditors & policy analysts — reproducible per-agency compliance scorecards from official government scan data.
• Journalists & researchers — "Which agencies still don't enforce HTTPS?" answered in one run, with receipts.

Quick start

Default run (no key needed — uses the shared DEMO_KEY) produces an agency scorecard for the General Services Administration:

{
  "mode": "scorecard",
  "apiKey": "DEMO_KEY",
  "agencies": ["General Services Administration"],
  "maxResults": 1000
}

Lead-gen example — every live Interior Department site missing DAP analytics:

{
  "mode": "gaps",
  "pillar": "dap",
  "apiKey": "YOUR_API_DATA_GOV_KEY",
  "agencies": ["Department of the Interior"],
  "maxResults": 2000
}

Full-government sweep (needs your own free key): leave agencies empty and set maxResults: 20000.

Get a free API key (30 seconds)

The prefilled DEMO_KEY works for small runs but is shared and rate-limited across all anonymous users. Register a free personal key at https://api.data.gov/signup/ (1,000 requests/hour — enough to sweep all ~29,500 sites three times over) and paste it into apiKey. API docs: https://open.gsa.gov/api/site-scanning-api/. If the shared key hits HTTP 429, the actor logs a warning and emits partial results instead of failing.

Input

Field	Type	Default	Notes
mode	enum	scorecard	scorecard | sites | gaps
apiKey	string	DEMO_KEY	Free key from api.data.gov/signup
agencies	string[]	all	Exact names use the API's server-side filter; partials match client-side
pillar	enum	dap	Gaps mode: dap, uswds, https_enforced, hsts, ipv6, viewport_meta_tag, sitemap_xml, robots_txt
failingPillars	enum[]	—	Sites mode: only export sites failing every listed pillar
domainContains / bureauContains	string	—	Case-insensitive substring filters
includeNonLive	boolean	false	Include dead-DNS / unreachable catalogue entries
maxResults	integer	2000	Caps sites fetched (100/request) and rows pushed; up to 20,000

Output fields

Every row has row_type (agency_scorecard, site, or gap).

• Scorecard rows: agency, total_sites, uswds_pct, dap_pct, https_enforced_pct, hsts_pct, ipv6_pct, viewport_meta_tag_pct, sitemap_xml_pct, robots_txt_pct, compliance_score (0–100 composite), worst_offenders (top-5 domains by failed pillars). Percentages count only sites where the pillar was actually measured.
• Site rows: domain, url, agency, bureau, branch, live, status_code, cms, cloud_provider, uswds, uswds_count, dap, dap_version, https_enforced, hsts, ipv6, viewport_meta_tag, sitemap_xml_detected, robots_txt_detected, third_party_service_count, login_provider, site_search, pageviews, visits, scan_date, failed_pillars.
• Gap rows: pillar, domain, url, agency, bureau, cms, cloud_provider, status_code, pageviews, visits, failed_pillars, scan_date.

Use as an MCP tool

This actor works as a clean, chainable MCP tool for AI agents (Claude, Cursor, custom agents) via mcp.apify.com. The input schema is fully described field-by-field, so an agent can ask things like "score every agency's website compliance" or "list Department of Energy sites without DAP analytics" and get structured rows back — ready to chain into CRM enrichment, report generation, or outreach drafting.

FAQ

Where does the data come from? GSA's Site Scanning program (api.gsa.gov/technology/site-scanning/v1), which rescans every known federal website roughly daily. scan_date on each row shows freshness.

Do I need an API key? No — DEMO_KEY works out of the box. But it's shared by everyone, so for full sweeps register a free key at https://api.data.gov/signup/ (instant, no approval).

What counts as "USWDS detected"? The site exposes usa- CSS classes or scores ≥ 50 on GSA's cumulative USWDS heuristic (uswds_count).

Why don't scorecard percentages include every site? Sites the scanner couldn't measure for a pillar (e.g. DNS failures) are excluded from that pillar's denominator, so scores reflect real measurements, not gaps in coverage. By default only live: true sites are analyzed at all (includeNonLive overrides).

Is there a security.txt pillar? Not currently — the v1 API does not expose a security.txt field. The 8 pillars cover USWDS, DAP, HTTPS, HSTS, IPv6, mobile viewport, sitemap, and robots.txt.

Can it audit state or local government sites? No — the underlying scan covers the federal web presence (~29,500 sites across executive, legislative, and judicial branches).

How is it priced? Pay-per-event: you pay a small fee per dataset row. A scorecard for one agency is a handful of rows; a full-government gap list is a few thousand.