UK Biodiversity Net Gain (BNG) Register Scraper

Structured, machine-readable feed of the UK Biodiversity Net Gain (BNG) sites register — every registered biodiversity gain site, its habitat units, local planning authority (LPA) allocations, and responsible body, pulled from the official Defra Biodiversity Net Gain data API. Free and public.

Keywords: biodiversity net gain, BNG, BNG credits, habitat bank, habitat banking, gain site register, biodiversity units, habitat units, Natural England, Defra, NSIP, Nationally Significant Infrastructure Projects, UK planning, local planning authority, LPA, statutory biodiversity metric, Town and Country Planning Act, environmental due diligence.

Why this actor exists

England's Environment Act 2021 makes mandatory 10% biodiversity net gain a condition of most planning permissions (since February 2024 for major developments, April 2024 for small sites) and — from 2 November 2026 — for Nationally Significant Infrastructure Projects (NSIPs) too. Every registered gain site and every habitat-unit credit sale is recorded on the government's "biodiversity gain sites register," but that register is published only as a manual search UI on GOV.UK: https://www.gov.uk/guidance/search-the-biodiversity-gain-sites-register — searchable one site at a time, no bulk export, no documented API.

This actor queries the register's own underlying JSON API directly and returns the full dataset (or a filtered slice) as structured records — one item per gain site.

Who this is for: planning consultants and ecologists tracking BNG delivery, habitat bank operators and credit sellers monitoring competitor supply, developers and their environmental DD teams checking a site or LPA's registered gain capacity, and AI agents / research tools that need current UK BNG register data without screen-scraping a JavaScript search box by hand.

Data source

The GOV.UK search page is a Next.js single-page app. Its page bundle embeds "apiUrl":"https://environment.data.gov.uk/bng" in a __NEXT_DATA__ JSON payload, and that host exposes a public Swagger UI at https://environment.data.gov.uk/bng/swagger (spec: https://environment.data.gov.uk/bng/swagger/v1/swagger.json) describing a "Biodiversity Net Gain API" with a paginated GET /bng/api/all endpoint. This actor calls that endpoint directly — no API key or authentication is required, and it is the same public data the GOV.UK search page itself reads. No scraping of rendered HTML is involved.

Input

Field	Type	Required	Description
localPlanningAuthority	string	No	Free-text filter, case-insensitive substring match against the LPA name(s) recorded on the site's habitat-unit allocations (e.g. "Leeds", "Buckinghamshire"). Empty = all authorities.
habitatType	string	No	Free-text filter, case-insensitive substring match against habitat type names on the site (e.g. "Lowland meadows", "Woodland"). Empty = all habitat types.
dateFrom	string (YYYY-MM-DD)	No	Earliest gain-site start (registration) date to include. Empty = no lower bound.
dateTo	string (YYYY-MM-DD)	No	Latest gain-site start (registration) date to include. Empty = no upper bound. Cannot be in the future.
maxRecords	integer	No	Safety cap on the number of records returned, applied after filtering. Empty = no cap (the register currently holds under 1,000 sites).

Example input

{
  "localPlanningAuthority": "Leeds",
  "habitatType": "",
  "dateFrom": "2024-01-01",
  "dateTo": "",
  "maxRecords": 50
}

Output

One dataset item per biodiversity gain site:

{
  "reference_number": "BGS-060324001",
  "site_size_hectares": 19.74,
  "grid_reference": "SP6532018711",
  "gain_site_status": "Registered",
  "start_date": "2023-08-10T00:00:00",
  "end_date": "2053-08-10T00:00:00",
  "removal_date": null,
  "extract_date": "2026-07-03T13:35:33.1287007",
  "responsible_bodies": ["Buckinghamshire LPA"],
  "local_planning_authorities": ["Buckinghamshire LPA"],
  "planning_references": ["PL/24/2989/FA"],
  "habitat_types": [
    "Grassland - Lowland meadows",
    "Grassland - Other neutral grassland"
  ],
  "land_boundary_url": "https://manage-biodiversity-gains.defra.gov.uk/public-documents/BGS-060324001.jpg",
  "allocation_count": 1,
  "amendment_count": 0,
  "raw": { "...": "full unflattened API record, including per-habitat-parcel area/condition/unit detail and per-allocation developer references" }
}

Field notes:

• reference_number is the official Biodiversity Gain Site (BGS) reference.
• site_size_hectares is the total registered site area.
• local_planning_authorities / planning_references are drawn from the site's allocations — a site with no allocations recorded yet (habitat units registered but not yet sold/assigned to a specific planning permission) returns empty arrays here, not an error.
• habitat_types lists every distinct habitat type recorded on the site (baseline and created/enhanced combined) — see raw.habitats for the full baseline vs. created-or-enhanced breakdown with area, condition, and unit counts per parcel.
• raw preserves the complete, unflattened API record for anyone who needs the nested detail (hedgerow/watercourse units, per-parcel intervention type, developer references, amendment history).

Pricing

Free. This is a market-validation probe: BNG is newly mandatory for NSIPs from 2 November 2026 and no bulk API product for this register currently exists on Apify Store. This actor is published free to test demand before any paid tier is considered.

Known limitations / fragility

• The underlying Defra API is an unversioned public beta, not a documented, stable contract. It was reverse-engineered from a Next.js page bundle and a discoverable Swagger UI, not from official API documentation — Defra could change or remove it without notice.
• Record offsets roughly 90-120 in the register reproducibly fail on the Defra server side. Verified across multiple independent page-size/offset combinations (pageSize=15 page 7, pageSize=20 page 6, pageSize=10 page 10) — every request whose window falls in that range either times out (45-60s) or returns HTTP 500, regardless of page size. This points to a server-side issue with a specific record (or small run of records) in that part of the register, not a client pagination-parameter problem. This actor retries each page individually with exponential backoff (up to 4 attempts), but if a page keeps failing, the run fails with a clear error rather than silently returning a truncated dataset. Set maxRecords to 90 or fewer (the default) to reliably avoid this range entirely.
• Large page sizes hang. pageSize=100 was observed to time out consistently during testing; this actor defaults to pageSize=15 to keep each request small and each retry cheap.
• localPlanningAuthority / habitatType filters are substring matches against whatever free-text values the register itself contains — they are not validated against a fixed enum, because the register's own LPA and habitat-type values are not exposed as a stable fixed list by the source API.

Local testing (no Apify account required)

pip install -r requirements.txt
python -m src.main

Reads test_input.json from the project root and writes results to sample_output.json. No Apify platform, API key, or account is needed — src/main.py detects the Apify runtime via the APIFY_IS_AT_HOME environment variable and falls back to local file I/O when it's absent.

Legal basis

The biodiversity gain sites register is a statutory public register maintained by Natural England under the Environment Act 2021 and the Town and Country Planning Act 1990 (as amended). This actor reads only that public register through its own public API — the same data and access path available to any member of the public via the GOV.UK search page, at machine scale.