GOV.UK Content Sitemap Scraper

Turn GOV.UK sitemaps into a clean, searchable dataset of official pages, titles, document types, publishing organisations, and update dates. Use it for UK policy monitoring, compliance research, public-content inventories, and AI/RAG ingestion.

Start in 30 seconds

1. Keep the default GOV.UK sitemap.
2. Add a topic such as employment or tax.
3. Run with 25-50 URLs, then inspect the Dataset tab.

{
  "startUrls": [{ "url": "https://www.gov.uk/sitemap.xml" }],
  "keywords": ["employment"],
  "fetchContentApi": true,
  "maxUrls": 25
}

Expected result: up to 25 official GOV.UK page rows with source URLs and public Content API metadata where available. Empty topic matches return zero rows rather than diagnostic records.

Why this actor

GOV.UK is a major source for UK policy, compliance, tax, employment, immigration, procurement, business-support, and public-service information. Organizations often need to discover and monitor official pages by topic, but raw sitemaps and content metadata are awkward to join manually. This actor creates structured rows that work for UK compliance dashboards, public-sector research, site inventories, and RAG pipelines.

SEO keywords covered by this Store page include GOV.UK scraper, GOV.UK content API scraper, UK government website scraper, government guidance monitor UK, GOV.UK sitemap scraper, UK compliance content scraper, UK policy monitoring, and GOV.UK RAG dataset.

Output

• GOV.UK page URL and base path
• Title and description
• Document type and schema name
• Public update date and first published date
• Publishing app, organisations, and topical links
• Raw GOV.UK Content API payload when available

Common Use Cases

• Monitor regulatory, tax, immigration, employment, or policy guidance
• Build GOV.UK content inventories for search and AI retrieval
• Track public-sector information updates by topic
• Feed official UK government pages into compliance dashboards
• Discover pages for downstream sitemap or content-change monitors
• Trigger alerts when pages matching a policy, tax, employment, or immigration topic are discovered or updated

Agent and automation workflows

Schedule the actor for topic lists, send rows into Airtable, Slack, Make, n8n, a warehouse, or a vector database, and let agents summarize new guidance, route pages to policy owners, or prepare compliance briefs from official source URLs.

Responsible use

This actor extracts public GOV.UK sitemap and Content API metadata. Use it for monitoring, compliance support, public research, content inventory, and AI retrieval workflows. Verify legal, policy, or compliance decisions against the official page and qualified advice.

Pricing

Pay $2 per 1,000 returned content-page rows. Content API enrichment is included when enabled. Filtered URLs, duplicate URLs, failed enrichment requests, and diagnostics are not returned as billable rows.