GOV.UK Content Search Scraper

GOV.UK Content Search Scraper

Scrape GOV.UK content pages by keyword, format, organisation, or taxonomy, up to a million per run. Every page returns its title, description, URL, publication date, and document type. No API key or registration. Export to CSV, JSON, Excel, or XML.

GOV.UK publishes thousands of official documents, announcements, and guidance pages across hundreds of organisations and formats, but there is no public API that lets you run a free-text search and pull the results into a spreadsheet. This Actor queries the public GOV.UK content search directly, applies your filters for document format, publishing department, topic, or date range, and returns each matching page as a flat row.\n\nYou get the full catalogue without writing a single line of code.

Who uses it	What they scrape GOV.UK for
Policy analysts	Monitor new guidance, impact assessments, and consultations from a specific department.
Journalists and researchers	Track FOI releases, national statistics, and transparency data across government.
Compliance officers	Collect regulatory notices, statutory instruments, and official decisions for an industry.
Data scientists	Build a corpus of UK government text for topic modelling or trend analysis.

What it does

This Actor collects GOV.UK content pages by keyword, document format, organisation, or taxonomy filter and returns each one as a flat row with its title, description, URL, publication date, and content type.

• 🔍 Free-text search: query the full GOV.UK catalogue by any keyword and get matching titles, descriptions, and body text.
• 📄 Document format filter: restrict results to a single type, such as press releases, FOI releases, official statistics, or statutory guidance.
• 🏛️ Organisation filter: pull content from one specific department or agency, from the Cabinet Office to the DVLA.
• 🌍 World location filter: narrow results to pages tagged with a specific country, such as Ukraine or France.
• 📅 Date range: set a published-after and published-before window to capture only recent updates or a historical period.
• 🗂️ Taxonomy drill-down: filter by GOV.UK topic taxon, topical event, person, or manual slug for precise retrieval.
• 📊 Flexible output: sort by relevance, newest, oldest, title, or popularity, and export up to a million rows.

Results export to CSV, JSON, Excel, or XML, or straight from the API.

What you can do with GOV.UK data

📈 Monitor policy announcements.

A think tank sets a keyword for 'net zero' and the organisation filter to 'Department for Energy Security & Net Zero' to collect every new policy paper and press release for a weekly digest.

📰 Track FOI releases across government.

A journalist runs the Actor with the format set to 'FOI Release' and a date range for the last quarter to find newly disclosed documents from all departments.

⚖️ Collect regulatory decisions for an industry.

A compliance team filters by 'CMA Case' format and a keyword for 'pharmaceuticals' to gather all Competition and Markets Authority rulings relevant to their sector.

🌍 Gather country-specific travel and trade guidance.

A business analyst uses the world location filter for 'india' and the format 'Guidance' to pull all current official advice for operating in that market.

Why choose this scraper

	What you get
No API key	Queries the public GOV.UK content search with no registration or authentication.
All formats	Covers over 100 document types, from news stories and speeches to tribunal decisions and statutory instruments.
Full taxonomy	Filter by organisation, topic, world location, person, topical event, or manual.
Date control	Set a precise publication window with published-after and published-before fields.
Scalable	Collect up to 1,000,000 pages in a single run.

How it compares

No other Store actor targets GOV.UK the same way, so the honest comparison is with the alternatives teams actually weigh.

	GOV.UK Content Search Scraper	Build it in-house	By hand
Setup	Run it now, zero config	Days of engineering	None, but hours per pull
When GOV.UK changes	Maintained for you	You fix it	You re-learn the page
Proxies, retries, anti-bot	Built in	Your problem	Browser only
Output	Fixed JSON schema, CSV/Excel export	Whatever you build	Copy-paste
Cost	Pay per result	Engineering time	Analyst hours

Configure the run

Drive the Actor from a keyword query, document format, and organisation slug, alone or together, and filters run as each page is read so only matches reach your dataset. The Input tab lists every parameter.

A first run with the defaults:

{
  "maxItems": 10
}

A larger pull:

{
  "maxItems": 200
}

Pricing

Pay-per-result: $0.0305 per result collected. You pay only for the results written to your dataset.

Results collected	Approximate cost
100 results	$3.05
1,000 results	$30.50
10,000 results	$305.00

New Apify accounts start with $5 in free credit.

Free users

Free-plan runs return up to 10 results as a preview. Upgrade your Apify plan to collect up to 1,000,000 results per run.

Run it

1. Create a free Apify account with $5 in credit.
2. Open the GOV.UK Content Search Scraper.
3. Set your inputs and any filters, then click Start.
4. Export the results as CSV, Excel, JSON, or XML from the Dataset tab.

Run it programmatically through the Apify API (run-sync-get-dataset-items) or the ApifyClient for JavaScript and Python.

Use with AI agents (MCP)

Give an AI agent live access to GOV.UK through the Model Context Protocol. Add the Actor to Claude, Cursor, or any MCP client:

$claude mcp add --transport http apify "https://mcp.apify.com?tools=parseforge/govuk-content-search-scraper"

Then prompt it in plain language to run the scraper and read back the results.

Troubleshooting

Why am I getting no results?

Check that your keyword, format, and organisation filters are not too restrictive. Try removing one filter at a time to see which is eliminating all matches. Also verify that your date range does not exclude all content.

The Actor returns fewer pages than my maxItems setting.

This is expected when the total number of matching pages on GOV.UK is smaller than your maximum. The Actor collects every available match and stops when there are no more pages.

My topical event or person slug is not working.

Slugs are case-sensitive and must match the GOV.UK URL exactly. Visit the topical event or person page on GOV.UK and copy the slug directly from the address bar.

The run is taking a long time.

Large queries with a high maxItems value will take longer. Reduce the maximum pages or narrow your filters to speed up the run.

I see duplicate pages in my dataset.

GOV.UK may return the same page under different search contexts. Use the deduplication features in your export tool or process the dataset by URL to remove duplicates.

FAQ

Question	Answer
What is the difference between a keyword query and a format filter?	The keyword query searches the full text of titles, descriptions, and body content. The format filter restricts results to a specific GOV.UK document type, such as 'Press Release' or 'FOI Release'. You can use them together or separately.
How do I find the correct organisation slug?	The input editor provides a dropdown list of all available organisation slugs and their display names. Pick one from the list; you do not need to look up slugs manually.
Can I scrape all content from a single department without a keyword?	Yes. Leave the keyword field empty, select an organisation from the dropdown, and set your maximum pages. The Actor will return all content pages from that department in the order you choose.
What is a topical event slug and where do I find it?	A topical event slug is the URL path for a government event page, such as 'spring-budget-2024'. You can find these slugs by browsing gov.uk/government/topical-events and copying the last part of the URL.
How do I filter by a specific topic like 'Business and industry'?	Use the taxon slug field. Enter the slug for the topic, for example 'business-and-industry'. You can find taxon slugs by browsing the topic hierarchy on GOV.UK.
Does this Actor handle pagination automatically?	Yes. The Actor follows the GOV.UK content search pagination and collects every matching page up to the maximum you set.
Can I get the full body text of each page?	The Actor returns the title, description, URL, publication date, and content type for each page. It does not fetch the full body text from the detail page.
What export formats are supported?	You can export your dataset in CSV, JSON, Excel, or XML from the Apify platform.
Is there a rate limit or do I need an API key?	No API key or registration is required. The Actor queries the public GOV.UK content search and respects standard request intervals.
Can I filter by multiple organisations at once?	The input currently accepts a single organisation slug per run. To collect from multiple organisations, run the Actor once per organisation or leave the field empty to search across all of them.

Related actors

Browse the full ParseForge collection for more scrapers.

🆘 Need help? Email parseforge@protonmail.com with your run ID, your input, and what you expected.

⚠️ Disclaimer. This Actor is unofficial and is not affiliated with, endorsed by, or sponsored by His Majesty's Government. It collects only publicly available data. You are responsible for using the collected data in compliance with the source's terms of service and applicable data-protection laws, including GDPR, CCPA, and PIPL. Do not use it to collect personal data unlawfully.