# UK Planning Register Scraper (`vagile_airliner/uk-planning-register-scraper`) Actor Returns UK planning applications from 13 Idox council portals (Leeds, Edinburgh, Oxford, York, Bristol, Sheffield, Westminster, Glasgow, more) as structured records: reference, address, proposal, status, dates, applicant, decision, portal link. Live per-application data, not the weekly bulletin. - **URL**: https://apify.com/vagile\_airliner/uk-planning-register-scraper.md - **Developed by:** [milan Pecka](https://apify.com/vagile_airliner) (community) - **Categories:** Lead generation, Real estate - **Stats:** 2 total users, 1 monthly users, 0.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $15.00 / 1,000 planning application records This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## UK Planning Register Scraper Live, per-council planning application records from UK councils — not just the weekly national bulletin digest. Pulls directly from each council's own public planning register (the **Idox Public Access** platform, used by roughly 60-70% of England, Scotland, and Wales's 425 local planning authorities). ### Why this instead of the bulletin scrapers already on Apify Existing UK planning actors on the store scrape the weekly national bulletin — a lagging, low-detail digest. This actor queries each council's live register directly, returning: - The full application record (not just a headline), including applicant name, case officer (where the council publishes it), decision, and decision date - Records as soon as the council validates them — no waiting for the weekly bulletin cycle - A direct link back to the council's own portal page for every record Construction firms, architects, planning consultants, and property developers currently pay £50-300+/month for commercial planning-intelligence SaaS (Glenigan, Barbour ABI, Radius Data) to get this. This actor gets the same underlying public data at pay-per-record pricing. ### What it does For each selected council, the actor: 1. Opens the council's Idox Public Access advanced-search form and collects a session CSRF token (required by every Idox deployment). 2. Submits a search filtered by validated-date range, and optionally by keyword and application status. 3. Paginates through every result page. 4. For each application, fetches the summary, details, and dates tabs from the council's own detail page and merges them into one structured record. 5. Emits one dataset item per planning application. ### Supported councils (this release) 13 councils confirmed live and reachable via their Idox Public Access advanced-search endpoint at build time (2026-07-02): | Key | Council | |---|---| | `leeds` | Leeds City Council | | `lambeth` | London Borough of Lambeth | | `edinburgh` | City of Edinburgh Council | | `oxford` | Oxford City Council | | `york` | City of York Council | | `bristol` | Bristol City Council | | `sheffield` | Sheffield City Council | | `nottingham` | Nottingham City Council | | `westminster` | Westminster City Council | | `southwark` | London Borough of Southwark | | `glasgow` | Glasgow City Council | | `plymouth` | Plymouth City Council | | `aberdeen` | Aberdeen City Council | Use `"councils": ["all"]` to scrape all supported councils in one run. More councils will be added over time — Idox Public Access is a shared platform template, so adding a council is a one-line config addition once its base URL is verified, not new code. **Not supported yet:** councils running other portal platforms (Northgate, Civica, bespoke in-house portals) — roughly 30-40% of the remaining councils. Some Idox-confirmed councils were deliberately excluded from this seed list because their public endpoint returned a blocking response (403/406/503) during verification — see Known limitations below. ### Input | Field | Type | Required | Description | |---|---|---|---| | `councils` | array of strings | No | Council keys to scrape, or `["all"]`. Defaults to all supported councils. | | `dateFrom` | string (`YYYY-MM-DD`) | No | Earliest validated date to include. Defaults to 7 days before today. | | `dateTo` | string (`YYYY-MM-DD`) | No | Latest validated date to include. Defaults to today. Cannot be in the future. | | `keyword` | string | No | Free-text filter matched against the application description. | | `applicationStatus` | string | No | One of `Current`, `Decided`, `Unknown`. Empty = all statuses. | | `maxRecordsPerCouncil` | integer | No | Safety cap on records fetched per council in this run. | Date range is capped at 365 days (`dateTo - dateFrom`) — wider ranges are rejected with a validation error rather than silently truncated. For ranges beyond ~30 days, set `maxRecordsPerCouncil` explicitly: a wide date range on a busy council can return thousands of applications, and each one costs a handful of per-application HTTP requests to enrich — an unbounded wide-range run risks a very long runtime and correspondingly high PPE cost. #### Example input ```json { "councils": ["leeds", "york"], "dateFrom": "2026-06-25", "dateTo": "2026-07-02", "keyword": "extension", "applicationStatus": "", "maxRecordsPerCouncil": 50 } ```` ### Output One dataset item per planning application: ```json { "council": "Leeds City Council", "reference": "26/03750/TR", "address": "13 North Park Avenue Lidgett Park Leeds LS8 1DN", "proposal": "T1 - Birch - Remove to ground level.", "application_type": "Tree Works", "status": "Current", "date_received": null, "date_validated": "Thu 02 Jul 2026", "decision": null, "decision_date": null, "applicant_name": "Mrs Lyn Prollins", "case_officer": null, "portal_url": "https://publicaccess.leeds.gov.uk/online-applications/applicationDetails.do?keyVal=THJV35JB3BO00&activeTab=summary" } ``` Field notes: - `date_received` and `case_officer` are populated only when the council chooses to publish them — this varies by council. A missing field is `null`, never an error. - `decision` / `decision_date` are `null` for applications still under consideration. - `portal_url` always links back to the live record on the council's own site. ### Pricing (suggested) Pay-per-event: **$0.015 per application record** returned. Justified against the $5-7/1,000 pricing norm for thinner open-data-API wrappers (company registries, procurement) given this actor returns much richer structured fields (applicant, case officer, decision, portal link) scraped from theme-customized HTML across many independent council deployments, and against the £50-300+/month commercial SaaS incumbents this directly undercuts. #### Publisher setup (required before listing with pay-per-event pricing) This actor charges one `application-record` event per dataset item pushed — the event name is hardcoded in `src/main.py` (`APPLICATION_RECORD_EVENT`) and passed as `charged_event_name` to `Actor.push_data()`. For billing to actually take effect once published, the event must be wired up on the Apify Console side: 1. In the Actor's **Apify Console** page, open the **Publication → Pricing** tab. 2. Select **Pay per event** as the pricing model. 3. Add a pricing event with the exact name `application-record` (must match `APPLICATION_RECORD_EVENT` in `src/main.py` verbatim — Apify does not validate this at build time, a mismatch means `push_data` calls succeed but no charge is recorded). 4. Set the price per event to the suggested $0.015 (or your chosen rate). 5. Save and publish. Verify with a small paid test run that a charge event appears in the run's **Usage** tab matching the number of dataset items pushed. Local runs (`python -m src.main` without `APIFY_IS_AT_HOME` set) never call `Actor.push_data()` — they write directly to `sample_output.json` via plain `json.dump()` — so no charge event fires and local testing never incurs cost, on or off the platform. ### Politeness and reliability - Each council is rate-limited to roughly 1 request per 1.5 seconds — under the 1-2 req/s ceiling generally considered safe for public council infrastructure. - Every request retries up to 4 times with exponential backoff on timeouts and 5xx errors. A 429 (rate limited) response gets an additional fixed cooldown on top of the normal backoff, since it signals the council's WAF has already flagged the client. - One council's failure (blocked, unreachable, malformed response) never aborts a multi-council run — it's caught, logged, and the run continues with the remaining councils. - If a specific application's detail-tab enrichment fails after retries, the actor still emits a record built from the search-results listing (reference, address, proposal, status) rather than dropping the application entirely. - A circuit breaker stops a council's enrichment early after 5 consecutive per-application failures (instead of retry-storming every remaining application against a WAF that has already started blocking the run). Records collected before the breaker tripped are still returned; the remaining applications for that council are simply not enriched in that run. ### Known limitations - **Case officer and applicant name are not universally published.** Some councils (e.g. Leeds) omit these fields from their public detail pages entirely; others (e.g. York, Westminster) publish both. This is a per-council editorial choice on the council's side, not a scraping gap — the actor surfaces whatever the council makes public and returns `null` otherwise. - **`date_received` vs `date_validated`.** Idox's advanced-search form only filters on validated date, not received date, so the date-range filter always applies to validation date. Received date is still captured in the output when the council's detail page exposes it. - **A handful of large councils were excluded from this seed list** despite running Idox, because their public endpoint returned a blocking response during verification (Camden and Wandsworth: HTTP 403; Manchester: HTTP 503; Newcastle: HTTP 406; Liverpool: inconsistent 406/connection resets; Leicester and Brighton: search flow required a session step not yet implemented). These may be addressable with council-specific header/cookie handling in a future release, but were dropped rather than shipped unreliable. - **Idox WAF cooldowns are real and per-IP.** Under sustained load within a short window, an individual council's Idox instance may return 429 for several minutes even at the actor's default throttle. The retry/backoff logic handles this within a single run; back-to-back runs against the same council in quick succession may still see elevated 429 rates. ### Local testing (no Apify account required) ```bash pip install -r requirements.txt python -m src.main ``` This reads `test_input.json` from the project root and writes results to `sample_output.json`. No Apify platform, API key, or account is needed for local testing — `src/main.py` detects the Apify runtime via the `APIFY_IS_AT_HOME` environment variable and falls back to local file I/O when it's absent. ### Legal basis UK local planning authorities are statutorily required to maintain a public register of planning applications (Town and Country Planning Act 1990, s.69). This actor reads only that public register through each council's own public-facing search interface — the same data and the same access path available to any member of the public via a web browser. # Actor input Schema ## `councils` (type: `array`): Which councils to scrape. Use \["all"] to scrape every supported council. Valid keys: all, leeds, lambeth, edinburgh, oxford, york, bristol, sheffield, nottingham, westminster, southwark, glasgow, plymouth, aberdeen. Unknown keys are rejected at run time with an error listing the valid set. See README for the full supported-council list. ## `dateFrom` (type: `string`): Earliest application-validated date to include, ISO format YYYY-MM-DD. Defaults to 7 days before today if omitted. ## `dateTo` (type: `string`): Latest application-validated date to include, ISO format YYYY-MM-DD. Defaults to today if omitted. Cannot be in the future. ## `keyword` (type: `string`): Optional free-text filter matched against the application description/proposal (e.g. "extension", "loft conversion", "solar"). ## `applicationStatus` (type: `string`): Optional filter by application status. ## `maxRecordsPerCouncil` (type: `integer`): Safety cap on how many applications to fetch per council in this run. Leave empty for no cap (bounded only by the date range results). ## Actor input object example ```json { "councils": [ "all" ], "keyword": "", "applicationStatus": "" } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "councils": [ "all" ] }; // Run the Actor and wait for it to finish const run = await client.actor("vagile_airliner/uk-planning-register-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "councils": ["all"] } # Run the Actor and wait for it to finish run = client.actor("vagile_airliner/uk-planning-register-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "councils": [ "all" ] }' | apify call vagile_airliner/uk-planning-register-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=vagile_airliner/uk-planning-register-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "UK Planning Register Scraper", "description": "Returns UK planning applications from 13 Idox council portals (Leeds, Edinburgh, Oxford, York, Bristol, Sheffield, Westminster, Glasgow, more) as structured records: reference, address, proposal, status, dates, applicant, decision, portal link. Live per-application data, not the weekly bulletin.", "version": "0.1", "x-build-id": "zfrF0tLC0JPu3cPfL" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/vagile_airliner~uk-planning-register-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-vagile_airliner-uk-planning-register-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/vagile_airliner~uk-planning-register-scraper/runs": { "post": { "operationId": "runs-sync-vagile_airliner-uk-planning-register-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/vagile_airliner~uk-planning-register-scraper/run-sync": { "post": { "operationId": "run-sync-vagile_airliner-uk-planning-register-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "councils": { "title": "Councils", "type": "array", "description": "Which councils to scrape. Use [\"all\"] to scrape every supported council. Valid keys: all, leeds, lambeth, edinburgh, oxford, york, bristol, sheffield, nottingham, westminster, southwark, glasgow, plymouth, aberdeen. Unknown keys are rejected at run time with an error listing the valid set. See README for the full supported-council list.", "items": { "type": "string" }, "default": [ "all" ] }, "dateFrom": { "title": "Date from", "type": "string", "description": "Earliest application-validated date to include, ISO format YYYY-MM-DD. Defaults to 7 days before today if omitted." }, "dateTo": { "title": "Date to", "type": "string", "description": "Latest application-validated date to include, ISO format YYYY-MM-DD. Defaults to today if omitted. Cannot be in the future." }, "keyword": { "title": "Keyword filter", "type": "string", "description": "Optional free-text filter matched against the application description/proposal (e.g. \"extension\", \"loft conversion\", \"solar\").", "default": "" }, "applicationStatus": { "title": "Application status", "enum": [ "", "Current", "Decided", "Unknown" ], "type": "string", "description": "Optional filter by application status.", "default": "" }, "maxRecordsPerCouncil": { "title": "Max records per council", "minimum": 1, "type": "integer", "description": "Safety cap on how many applications to fetch per council in this run. Leave empty for no cap (bounded only by the date range results)." } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```