Federal Website Compliance Auditor — GSA Site Scanning API

Audit 29,591 federal .gov websites (count verified live 2026-08-01) in one run. This actor turns GSA's official Site Scanning API — the same nightly scan engine behind digital.gov's dashboards — into three ready-to-use datasets:

1. scorecard (default) — one row per federal agency: site count, % compliant on each of 8 pillars plus the denominator each percentage was computed over, a composite 0–100 compliance score, required-links and accessibility rollups, and the agency's worst-offender domains.
2. sites — one flat row per website: 8 compliance pillars, 9 per-scan status fields, USWDS version detail, required footer links, Section 508 violation counts, Core Web Vitals, third-party and cookie domains, robots/sitemap detail, and the www variant.
3. gaps — the govtech lead-gen mode: one row per live site measurably failing a chosen pillar, with agency, bureau, CMS, hosting provider and traffic — a prospect list of agencies missing exactly the technology you sell.

No scraping, no headless browsers — pure official API, fast and reliable.

What changed in v1.1 (and why it matters to your numbers)

v1.1 is a correctness pass. Three defects were found by re-measuring against the live API, and all three produced plausible wrong answers on a run that reported success:

• Truncated agency scorecards. Several agencies are larger than the actor's own default fetch budget — Health and Human Services has 4,741 sites, Energy 4,508, Defense 3,279, NASA 2,230. A scorecard for any of them was computed over an arbitrary slice (as little as 21%) and published as the agency's score with no warning. Every row now carries sites_available, sites_fetched, coverage_pct and is_complete, and a named agency is fetched in full by default.
• Silently dropped agencies. When several agencies were requested, they shared one fetch budget, so a large first agency consumed it and later agencies produced no rows at all — indistinguishable from "this agency has nothing to report". Each scope now has its own budget, and a scope that could not be measured emits an explicit api_query_status row instead of vanishing.
• Agency names that silently matched nothing. GSA's filter is exact and case-sensitive: "general services administration" and "General Services" both return zero. The actor now resolves what you type against the live list of 146 agency names (abbreviations, casing and unique partial names all work) and reports an explicit no_match with the closest candidates rather than an empty dataset.

Also new: the API's nine per-scan status fields are now on every row, so https_enforced: null is visibly "the security scan did not complete" (which happens on 18.8% of live federal sites) rather than an unexplained blank. And the run now fails loudly — billing nothing — when the API is unreachable, when it has drifted, or when no requested scope could be measured.

Who is this for?

• Govtech vendors & digital agencies — gaps mode with pillar: "dap" or "uswds" is a qualified prospect list: agency, domain, current CMS, current host, front-end tooling and proof they're missing what you sell. Sites that were never measured are excluded, so you don't pitch a false negative.
• Federal web / compliance / CIO teams — track your agency's 21st Century IDEA & OMB M-23-22 posture (USWDS, DAP, HTTPS, required footer links, the .gov banner, Section 508 violations) against every other agency.
• Auditors & policy analysts — reproducible per-agency scorecards with the measurement denominator published alongside every percentage.
• Journalists & researchers — "Which agencies still don't enforce HTTPS?" answered in one run, with receipts and an explicit coverage figure.

Quick start

Default run produces a complete agency scorecard for the General Services Administration:

{
  "mode": "scorecard",
  "apiKey": "DEMO_KEY",
  "agencies": ["General Services Administration"],
  "completeAgencies": true,
  "maxResults": 1000
}

Lead-gen example — every live Interior Department site missing DAP analytics:

{
  "mode": "gaps",
  "pillar": "dap",
  "apiKey": "YOUR_API_DATA_GOV_KEY",
  "agencies": ["Department of the Interior"],
  "maxResults": 2000
}

Full-government sweep (needs your own free key): leave agencies empty and set maxResults: 20000.

Get a free API key (30 seconds)

The prefilled DEMO_KEY is api.data.gov's shared demo key and is capped at 10 requests per hour across every anonymous caller (measured live: X-Ratelimit-Limit: 10) — roughly one page of results. Register a free personal key at https://api.data.gov/signup/ (1,000 requests/hour — enough to sweep all ~29,600 sites three times over) and paste it into apiKey. API docs: https://open.gsa.gov/api/site-scanning-api/.

Input

Field	Type	Default	Notes
mode	enum	scorecard	scorecard | sites | gaps
apiKey	string	DEMO_KEY	Free key from api.data.gov/signup
agencies	string[]	all	Names are resolved against the live 146-name vocabulary — GSA, nasa, Department of Interior and National Aeronautics all work. An unresolvable name produces an explicit no_match row
pillar	enum	dap	Gaps mode: dap, uswds, https_enforced, hsts, ipv6, viewport_meta_tag, sitemap_xml, robots_txt
failingPillars	enum[]	—	Sites mode: only export sites failing every listed pillar (a pillar that was never measured never counts as a fail)
domainContains / bureauContains	string	—	Case-insensitive substring filters
includeNonLive	boolean	false	Include dead-DNS / unreachable catalogue entries
completeAgencies	boolean	true	Fetch each named agency in full so its scorecard describes the whole agency. Does not change what you pay — maxResults still caps output rows
maxResults	integer	2000	Caps rows written to the dataset (and sites fetched on a national sweep); up to 20,000

Output fields

Every row carries row_type (agency_scorecard, site or gap) plus the scope envelope below. Fields not applicable to a row type are null.

Scope & data-quality envelope (on every row)

Field	Meaning
source	Always GSA Site Scanning API v1
scope	The exact GSA agency name queried, or ALL AGENCIES (national sweep)
requested_agency	The agency string you supplied, before resolution
agency_name_resolution	exact, alias, case_insensitive, unique_substring, ambiguous, no_match, national_sweep
api_query_status	ok, partial, no_match or unavailable. Anything but ok means this row is not a complete answer
source_error	Upstream error for this scope. null on a healthy run
sites_available	Sites GSA catalogues in this scope (the API's own meta.totalItems)
sites_fetched	How many were actually retrieved
coverage_pct	sites_fetched / sites_available as a percentage
is_complete	True only when the whole scope was retrieved
data_completeness_note	Plain-English warning whenever this row is not a complete measurement
sites_analyzable	Fetched sites that survived the live/filter gates and your input filters

Agency scorecard rows

agency, total_sites, scan_date, uswds_pct / uswds_measured, dap_pct / dap_measured, https_enforced_pct / https_enforced_measured, hsts_pct / hsts_measured, ipv6_pct / ipv6_measured, viewport_meta_tag_pct / viewport_meta_tag_measured, sitemap_xml_pct / sitemap_xml_measured, robots_txt_pct / robots_txt_measured, compliance_score, worst_offenders, pillars_fully_measured, sites_missing_a_measurement, scan_errors, required_links_pct, required_links_measured, accessibility_clean_pct, accessibility_measured, uswds_banner_pct.

Each *_measured is the denominator its *_pct was computed over. It is often lower than total_sites — GSA's security scan fails on about 19% of live federal sites, so https_enforced_measured and hsts_measured are typically the smallest.

Site rows (and gap rows, which carry the same detail plus pillar)

Identity & hosting: domain, url, base_domain, initial_base_domain, top_level_domain, agency, bureau, branch, live, status_code, redirect, media_type, canonical_link, page_title, page_description, language, cms, cloud_provider, tooling, source_list.

Design system: uswds, uswds_count, uswds_usa_classes, uswds_version, uswds_semantic_version, uswds_publicsans_font, uswds_banner_heres_how, main_element_present.

Analytics: dap, dap_version, dap_agency_parameter, ga_tag_id.

Security & platform pillars: https_enforced, hsts, ipv6, viewport_meta_tag, sitemap_xml_detected, robots_txt_detected, failed_pillars, pillars_measured, pillars_unmeasured.

Per-scan status (the null-vs-false contract): primary_scan_status, dns_scan_status, security_scan_status, robots_txt_scan_status, sitemap_xml_scan_status, accessibility_scan_status, performance_scan_status, www_scan_status, not_found_scan_status.

Required links (21st Century IDEA / OMB M-23-22): required_links_found, required_links_missing, required_links_pct.

Accessibility & performance: accessibility_violations_total, accessibility_violations_detail, largest_contentful_paint_ms, cumulative_layout_shift.

Privacy & supply chain: third_party_service_count, third_party_service_domains, cookie_domains, hyperlink_domain_count.

Identity & search: login_provider, login_form_evidence, site_search, search_dot_gov.

Crawl surface: robots_txt_url, robots_txt_status_code, robots_txt_crawl_delay, robots_txt_sitemap_locations, sitemap_xml_url, sitemap_xml_status_code, sitemap_xml_count, sitemap_xml_pdf_count, sitemap_xml_lastmod, www_url, www_status_code, www_same, not_found_test_passed.

Traffic & freshness: pageviews, visits, scan_date.

Reading null correctly

null means not measured. false means measured and negative. They are different answers and this actor keeps them apart:

• https_enforced: null with security_scan_status: "unknown_error" means GSA could not test HTTPS on that site. It does not mean HTTPS is unenforced, and such a site never appears in an https_enforced gap list.
• required_links_missing: null means GSA did not scan the footer (about a third of live sites). It does not mean all nine links are missing.
• accessibility_violations_total: null means the accessibility scan did not complete; 0 means it completed and found nothing.
• search_dot_gov is true or null upstream — it is never false, so a null there is "not detected", not a verified negative.

Fields that populate only on certain inputs: source_error and data_completeness_note populate when a scope is partial, unavailable or unresolved (try agencies: ["Ministry of Magic"]); pillar populates in gaps mode; all *_pct / *_measured / total_sites / compliance_score / worst_offenders populate in scorecard mode only.

Use as an MCP tool

This actor works as a clean, chainable MCP tool for AI agents (Claude, Cursor, custom agents) via mcp.apify.com. Every input and output field is described, so an agent can ask "score every agency's website compliance" or "list Department of Energy sites without DAP analytics" and get structured rows back — with api_query_status and coverage_pct on each one, so the agent can tell a real answer from an incomplete one before chaining it into CRM enrichment, report generation or outreach drafting.

FAQ

Where does the data come from? GSA's Site Scanning program (api.gsa.gov/technology/site-scanning/v1), which rescans every known federal website nightly. The whole corpus checked on 2026-08-01 had been scanned within the previous 24 hours. scan_date on each row shows freshness, and the run fails if the nightly scan goes stale.

Do I need an API key? DEMO_KEY works, but it allows only 10 requests/hour shared across every anonymous user, which is not enough for a real audit. Register a free key at https://api.data.gov/signup/ (instant, no approval) for 1,000/hour.

What counts as "USWDS detected"? The site scores above zero on GSA's weighted usa- CSS class signal, or scores >= 50 on the cumulative USWDS heuristic (uswds_count). Both raw signals are published (uswds_usa_classes, uswds_count) so you can apply your own threshold.

Why is https_enforced_measured lower than total_sites? Because GSA's security scan does not complete on every site — 18.8% of live federal sites nationally. Those sites are excluded from the HTTPS and HSTS percentages rather than being counted as failures, and the denominator is published so you can see it.

Is there a security.txt pillar? No — the v1 API does not expose a security.txt field. The 8 pillars cover USWDS, DAP, HTTPS, HSTS, IPv6, mobile viewport, sitemap and robots.txt; required footer links and Section 508 violations are reported alongside them as additional signals.

Can it audit state or local government sites? No — the underlying scan covers the federal web presence across the Executive, Legislative, Judicial and Federal branches.

What happens if the API is down or has changed? The run fails with an explanatory message and bills nothing. Before any row is written, the actor checks the national catalogue size, a positive canary agency, a negative control (a nonexistent agency that must return zero — this catches the API silently ignoring the agency filter), the record schema, the types of every pillar-bearing field, and scan freshness.

How is it priced? Pay-per-event: a small fee per dataset row. A scorecard for one agency is a handful of rows; a full-government gap list is a few thousand. A failed run emits no rows.

Related actors

Every screening tool in this portfolio is a screening tool, not a determination. If you audit federal digital compliance you may also want GSA-adjacent lead sources such as the SAM/registry and licensing monitors in the same publisher's catalogue.