Defra UK Wholesale Produce Prices

A bounded Apify Actor for the official Defra wholesale horticultural price CSV. It discovers the current machine-readable CSV from the official GOV.UK dataset page on each run; no dated download URL is hardcoded into production code.

What the observations mean

These are national average wholesale prices for selected home-grown horticultural produce in England and Wales. The source describes averages of the most usual prices charged in Birmingham, Bristol, Manchester and a London market (New Spitalfields or Western International). They are not individual-market prices, retail prices or live spot quotations. Current source publication cadence is fortnightly; historical observations may be weekly.

Verified source snapshot on 8 October 2026: 17,702 CSV observations, 3 November 2017 through 28 September 2026; 53 observations at the latest source date. Latest Bramley seedling apple observation: GBP 1.23/kg. GBP is confirmed by the accompanying official current-week ODS, whose table uses £ headings. “Latest” always means the maximum source observation date, never today's date. Each row reports latestSourceDate, observedAt and sourceAgeDays; the Actor warns if the newest observation is more than 21 days old. This threshold is a warning heuristic, not a publication guarantee.

Inputs

Default: latestOnly=true and maxItems=100. The newest date is selected globally before applying filters, so an item absent from that date returns zero rows rather than silently returning an older price.

• latestOnly: true for the newest published source date; false for bounded history
• startDate/endDate: inclusive YYYY-MM-DD, both required for history, forbidden in latest mode
• categories: case-insensitive exact source categories, e.g. fruit, vegetable, cut_flowers, pot_plants
• items: case-insensitive exact source items, e.g. apples
• maxItems: 1–10,000, deterministic newest-first output cap; default 100

Empty filter lists mean all. No custom URLs, proxies, credentials or browser are accepted. History requires both dates and cannot exceed the actual CSV coverage. Sample small inputs are in smoke/latest.json and smoke/history.json.

Output and precision

One observation per (date, category, item, variety, unit), with deterministic SHA-256 id. Different varieties and units remain separate. Identical repeated observations collapse; conflicting prices for the same key fail the run. Output is sorted descending by date, category, item, variety, unit. Price is a decimal string (for example "1.23") to avoid binary floating-point rounding; currency is GBP. Units are preserved verbatim (observed: kg, head, twin, stem, unit). Zero prices are preserved. Missing, negative or malformed prices, dates, or required columns fail closed rather than becoming zero/null.

Every output includes official source URL, discovered CSV URL, attribution, OGL license link, source and geographic aggregation. All rows are validated against .actor/dataset_schema.json fields before any dataset push. RUN_SUMMARY includes matched/output/source counts, truncation, latest source date, source age, observation time, CSV URL and SHA-256. Zero matches are a successful empty dataset with a warning and summary. Network or source failures fail the run; they do not create invented rows or error rows.

Safe source access

Fixed HTTPS host allowlist: www.gov.uk and assets.publishing.service.gov.uk. Every request and redirect target is checked. The relevant origin's robots.txt is read before content, and disallow rules stop access. An explicit robots 404 means absent rules; other HTTP errors fail. Cross-origin robots redirects are refused. Requests are serial, with at least one second between requests and longer robots Crawl-delay respected. Bounds: 20 total HTTP requests, 5 redirects per fetch, 30-second socket timeout, 2 MiB page, 256 KiB robots, 8 MiB CSV and 100,000 parsed data rows. The full bounded CSV is read once even for small output caps; maxItems is an output limit, not an upstream byte limit.

HTTP 401/403/429 and CAPTCHA/challenge markers stop immediately. There is no retry, proxy rotation, login or challenge bypass. CSV discovery fails if zero or multiple distinct matching official CSV links appear. Schema drift and malformed rows stop processing rather than silently skipping observations.

Build and local tests

Python 3.12. Install requirements.txt into a clean virtual environment, then run:

python -m pip install -r requirements.txt
python -m unittest discover -s tests -v
python -m src

Runtime dependencies are pinned: Apify 4.0.2, Crawlee 1.10.4, Pydantic 2.13.5, jsonschema 4.26.0. The Actor uses Apify for lifecycle, input, dataset and key-value output, and standard-library HTTP/CSV parsing. Crawlee is pinned for SDK compatibility, not a claim of browser crawling.

The self-contained tests use 12 genuine bounded source observations, a sanitized actual official attachment link, robots fixture, and mocks for transport failures. They verify both official Apify metaschemas, every fixture output row against dataset fields, decimals/zero/null behavior, history/latest semantics, filters/caps, duplicates, malformed input/CSV, URL/redirect/robots/challenge gates and an isolated real SDK entrypoint. Official Apify schema snapshots and Apache license notice are under tests/metaschemas. No Apify account, spend or paid run is required. Local SDK tests clear platform environment variables and use a temporary local storage directory.

Source and licensing

Official source: https://www.gov.uk/government/statistical-data-sets/wholesale-fruit-and-vegetable-prices-weekly-average

The official dataset page explicitly declares Open Government Licence v3.0 (except where otherwise stated): https://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/

Contains public sector information licensed under the Open Government Licence v3.0. Crown copyright. The license destination itself returned HTTP 403 during research; no attempt was made to bypass it. The licensing attribution here is based on the official source page's explicit license marker, not a claim that the full license text was downloaded. Sanitized fixtures contain no user data, tokens, cookies or personal information. See tests/fixtures/provenance.json and THIRD_PARTY_NOTICES.md.