UK Planning Register Scraper

Live, per-council planning application records from UK councils — not just the weekly national bulletin digest. Pulls directly from each council's own public planning register (the Idox Public Access platform, used by roughly 60-70% of England, Scotland, and Wales's 425 local planning authorities).

Why this instead of the bulletin scrapers already on Apify

Existing UK planning actors on the store scrape the weekly national bulletin — a lagging, low-detail digest. This actor queries each council's live register directly, returning:

• The full application record (not just a headline), including applicant name, case officer (where the council publishes it), decision, and decision date
• Records as soon as the council validates them — no waiting for the weekly bulletin cycle
• A direct link back to the council's own portal page for every record

Construction firms, architects, planning consultants, and property developers currently pay £50-300+/month for commercial planning-intelligence SaaS (Glenigan, Barbour ABI, Radius Data) to get this. This actor gets the same underlying public data at pay-per-record pricing.

What it does

For each selected council, the actor:

1. Opens the council's Idox Public Access advanced-search form and collects a session CSRF token (required by every Idox deployment).
2. Submits a search filtered by validated-date range, and optionally by keyword and application status.
3. Paginates through every result page.
4. For each application, fetches the summary, details, and dates tabs from the council's own detail page and merges them into one structured record.
5. Emits one dataset item per planning application.

Supported councils (this release)

13 councils confirmed live and reachable via their Idox Public Access advanced-search endpoint at build time (2026-07-02):

Key	Council
leeds	Leeds City Council
lambeth	London Borough of Lambeth
edinburgh	City of Edinburgh Council
oxford	Oxford City Council
york	City of York Council
bristol	Bristol City Council
sheffield	Sheffield City Council
nottingham	Nottingham City Council
westminster	Westminster City Council
southwark	London Borough of Southwark
glasgow	Glasgow City Council
plymouth	Plymouth City Council
aberdeen	Aberdeen City Council

Use "councils": ["all"] to scrape all supported councils in one run. More councils will be added over time — Idox Public Access is a shared platform template, so adding a council is a one-line config addition once its base URL is verified, not new code.

Not supported yet: councils running other portal platforms (Northgate, Civica, bespoke in-house portals) — roughly 30-40% of the remaining councils. Some Idox-confirmed councils were deliberately excluded from this seed list because their public endpoint returned a blocking response (403/406/503) during verification — see Known limitations below.

Input

Field	Type	Required	Description
councils	array of strings	No	Council keys to scrape, or ["all"]. Defaults to all supported councils.
dateFrom	string (YYYY-MM-DD)	No	Earliest validated date to include. Defaults to 7 days before today.
dateTo	string (YYYY-MM-DD)	No	Latest validated date to include. Defaults to today. Cannot be in the future.
keyword	string	No	Free-text filter matched against the application description.
applicationStatus	string	No	One of Current, Decided, Unknown. Empty = all statuses.
maxRecordsPerCouncil	integer	No	Safety cap on records fetched per council in this run.

Date range is capped at 365 days (dateTo - dateFrom) — wider ranges are rejected with a validation error rather than silently truncated. For ranges beyond ~30 days, set maxRecordsPerCouncil explicitly: a wide date range on a busy council can return thousands of applications, and each one costs a handful of per-application HTTP requests to enrich — an unbounded wide-range run risks a very long runtime and correspondingly high PPE cost.

Example input

{
  "councils": ["leeds", "york"],
  "dateFrom": "2026-06-25",
  "dateTo": "2026-07-02",
  "keyword": "extension",
  "applicationStatus": "",
  "maxRecordsPerCouncil": 50
}

Output

One dataset item per planning application:

{
  "council": "Leeds City Council",
  "reference": "26/03750/TR",
  "address": "13 North Park Avenue Lidgett Park Leeds LS8 1DN",
  "proposal": "T1 - Birch - Remove to ground level.",
  "application_type": "Tree Works",
  "status": "Current",
  "date_received": null,
  "date_validated": "Thu 02 Jul 2026",
  "decision": null,
  "decision_date": null,
  "applicant_name": "Mrs Lyn Prollins",
  "case_officer": null,
  "portal_url": "https://publicaccess.leeds.gov.uk/online-applications/applicationDetails.do?keyVal=THJV35JB3BO00&activeTab=summary"
}

Field notes:

• date_received and case_officer are populated only when the council chooses to publish them — this varies by council. A missing field is null, never an error.
• decision / decision_date are null for applications still under consideration.
• portal_url always links back to the live record on the council's own site.

Pricing (suggested)

Pay-per-event: $0.015 per application record returned. Justified against the $5-7/1,000 pricing norm for thinner open-data-API wrappers (company registries, procurement) given this actor returns much richer structured fields (applicant, case officer, decision, portal link) scraped from theme-customized HTML across many independent council deployments, and against the £50-300+/month commercial SaaS incumbents this directly undercuts.

Publisher setup (required before listing with pay-per-event pricing)

This actor charges one application-record event per dataset item pushed — the event name is hardcoded in src/main.py (APPLICATION_RECORD_EVENT) and passed as charged_event_name to Actor.push_data(). For billing to actually take effect once published, the event must be wired up on the Apify Console side:

1. In the Actor's Apify Console page, open the Publication → Pricing tab.
2. Select Pay per event as the pricing model.
3. Add a pricing event with the exact name application-record (must match APPLICATION_RECORD_EVENT in src/main.py verbatim — Apify does not validate this at build time, a mismatch means push_data calls succeed but no charge is recorded).
4. Set the price per event to the suggested $0.015 (or your chosen rate).
5. Save and publish. Verify with a small paid test run that a charge event appears in the run's Usage tab matching the number of dataset items pushed.

Local runs (python -m src.main without APIFY_IS_AT_HOME set) never call Actor.push_data() — they write directly to sample_output.json via plain json.dump() — so no charge event fires and local testing never incurs cost, on or off the platform.

Politeness and reliability

• Each council is rate-limited to roughly 1 request per 1.5 seconds — under the 1-2 req/s ceiling generally considered safe for public council infrastructure.
• Every request retries up to 4 times with exponential backoff on timeouts and 5xx errors. A 429 (rate limited) response gets an additional fixed cooldown on top of the normal backoff, since it signals the council's WAF has already flagged the client.
• One council's failure (blocked, unreachable, malformed response) never aborts a multi-council run — it's caught, logged, and the run continues with the remaining councils.
• If a specific application's detail-tab enrichment fails after retries, the actor still emits a record built from the search-results listing (reference, address, proposal, status) rather than dropping the application entirely.
• A circuit breaker stops a council's enrichment early after 5 consecutive per-application failures (instead of retry-storming every remaining application against a WAF that has already started blocking the run). Records collected before the breaker tripped are still returned; the remaining applications for that council are simply not enriched in that run.

Known limitations

• Case officer and applicant name are not universally published. Some councils (e.g. Leeds) omit these fields from their public detail pages entirely; others (e.g. York, Westminster) publish both. This is a per-council editorial choice on the council's side, not a scraping gap — the actor surfaces whatever the council makes public and returns null otherwise.
• date_received vs date_validated. Idox's advanced-search form only filters on validated date, not received date, so the date-range filter always applies to validation date. Received date is still captured in the output when the council's detail page exposes it.
• A handful of large councils were excluded from this seed list despite running Idox, because their public endpoint returned a blocking response during verification (Camden and Wandsworth: HTTP 403; Manchester: HTTP 503; Newcastle: HTTP 406; Liverpool: inconsistent 406/connection resets; Leicester and Brighton: search flow required a session step not yet implemented). These may be addressable with council-specific header/cookie handling in a future release, but were dropped rather than shipped unreliable.
• Idox WAF cooldowns are real and per-IP. Under sustained load within a short window, an individual council's Idox instance may return 429 for several minutes even at the actor's default throttle. The retry/backoff logic handles this within a single run; back-to-back runs against the same council in quick succession may still see elevated 429 rates.

Local testing (no Apify account required)

pip install -r requirements.txt
python -m src.main

This reads test_input.json from the project root and writes results to sample_output.json. No Apify platform, API key, or account is needed for local testing — src/main.py detects the Apify runtime via the APIFY_IS_AT_HOME environment variable and falls back to local file I/O when it's absent.

Legal basis

UK local planning authorities are statutorily required to maintain a public register of planning applications (Town and Country Planning Act 1990, s.69). This actor reads only that public register through each council's own public-facing search interface — the same data and the same access path available to any member of the public via a web browser.