UK Council Spending Scraper (Local Authority Payments)

Extract UK council payments over GBP 500 from public open-data portals. Built by Steadfetch: web data that does not break.

What it does

UK councils must publish every payment over GBP 500. Each council publishes a different CSV file with different column names, so comparing them normally means downloading and reshaping one file per council. This Actor normalizes those files into one stable schema:

• councils: Camden and Hounslow in v1 (camden, hounslow);
• one row per payment: council, supplier organisation, amount, payment date, service area, expense type and transaction reference;
• filters by council, payment-date range and minimum amount;
• English field names and ISO dates (YYYY-MM-DD), validated against the published dataset schema before anything is saved;
• organisation-focused output: rows whose payee cannot be confidently identified as an organisation are skipped by a precision-first filter. It is a name-based check, so it strongly reduces personal-name records but cannot guarantee a 100% match.

The data is downloaded as plain public CSV over HTTP. There is no browser, no login, no paywall, no CAPTCHA bypass and no API key. The Camden portal asks crawlers for a one-second delay, which the Actor respects before each request.

Input

Field	Type	Default	Description
councils	array of camden, hounslow	both	Councils to extract.
dateFrom	date (YYYY-MM-DD)	-	Optional inclusive lower bound for the payment date.
dateTo	date (YYYY-MM-DD)	-	Optional inclusive upper bound for the payment date.
minAmount	number	500	Only payments at or above this amount are emitted.
limit	integer (1-100000)	100	Maximum results delivered per selected council.

Minimal input (every field is optional):

{}

With {} the Actor reads both councils and delivers up to 100 payments per council (200 in total, about 0.20 USD), which keeps a default run well inside the Apify free-plan credit. Raise limit for larger extracts. dateFrom/dateTo narrow the payment-date range, and minAmount raises the GBP 500 floor.

Example output

{
    "council": "camden",
    "vendorName": "ACME Facilities Ltd",
    "amountGbp": 12450.00,
    "paymentDate": "2026-09-29",
    "expenseArea": "Housing",
    "expenseType": "Services",
    "transactionRef": "PL-2026-04431",
    "sourceUrl": "https://opendata.camden.gov.uk/api/v3/views/3ixw-qvb8",
    "url": "https://opendata.camden.gov.uk/api/v3/views/3ixw-qvb8",
    "title": "ACME Facilities Ltd - 2026-09-29",
    "scrapedAt": "2026-10-23T23:30:00.000Z"
}

url and scrapedAt are the common Steadfetch envelope. Every item links to the public dataset it was read from.

Output fields

Field	Type	Required	Source
council	string	yes	selected council id (camden, hounslow)
vendorName	string	yes	Beneficiary Name (Camden) / BeneficiaryName (Hounslow)
amountGbp	number	yes	Amount GBP (Camden) / Amount (Hounslow)
paymentDate	date	yes	Payment Date / PaymentDate, normalized from DD/MM/YYYY
expenseArea	string or null	no	Organisational Unit (Camden) / ServiceCategoryLabel (Hounslow)
expenseType	string or null	no	Purpose (both councils)
transactionRef	string or null	no	Unique Identifier (Camden) / TransactionNumber (Hounslow)
sourceUrl	URI	yes	public dataset page
url	URI	yes	same as sourceUrl (envelope)
title	string	no	supplier and payment date
scrapedAt	date-time	yes	run timestamp (envelope)

Amounts are numeric GBP values. Dates are normalized to ISO dates so results from both councils can be sorted and compared in one table.

Pricing

Pay per event (PPE), charged automatically by Apify:

Event	Charged when	Price
apify-actor-start	once per run start (synthetic)	0.00005 USD
apify-default-dataset-item	once per validated payment saved to the dataset (synthetic)	0.001 USD = 1.00 USD per 1,000 payments

There is no custom per-item event and no Actor.charge() call: the synthetic dataset event already bills each result, so users are never billed twice. Rows skipped by the organisation filter, the amount filter or the date filter are not pushed and are not charged. Errors and empty runs only incur the start event.

Works on the Apify free plan - about 4,999 payments for the $5 monthly credit ((5.00 USD - 0.00005 USD run-start) / 0.001 USD per payment).

Cost per 1,000 results

Measured on 24 October 2026 in local fixture runs, without spending Apify credit:

• streaming CSV parsing, field normalization and schema validation of 1,000 payments: 70 ms of local CPU (measured on the real Camden export, 24 October 2026);
• one CSV download per selected council (the Camden export snapshot was about 110 MB uncompressed on 24 October 2026; it is streamed and reading stops at limit. Hounslow's monthly file is about 400 KB);
• a run makes at most three HTTP requests (Camden CSV, the Hounslow dataset page and the newest Hounslow CSV) and waits one second between them;
• at 256 MB and Apify's compute price of 0.20 USD per GB-hour, extraction and validation cost approximately 0.000001 USD per 1,000 payments, plus the 0.00005 USD actor start.

Cost per 1,000 payments is therefore well below 0.01 USD (the spec ceiling was 0.10 USD). At 1.00 USD per 1,000 the gross margin is above 99%. No proxy is used and the CSV downloads are free, so there is no per-request fee.

Data sources and coverage

Council	Portal	Format	Licence
Camden	opendata.camden.gov.uk (Socrata, dataset 3ixw-qvb8)	CSV export, one row per payment	UK Open Government Licence v3
Hounslow	data.hounslow.gov.uk (PortalJS / Datopian)	one CSV per month; the Actor discovers and reads only the newest file on every run	UK Open Government Licence v3

Hounslow keeps only its newest monthly file on the portal, and the Actor reads that file only. A dateFrom/dateTo window that ends before that month returns zero Hounslow rows (Camden is not affected, because its export carries the full history). For older Hounslow months, download them from the portal directly.

Both files are published as "spending over 500" transaction lists. The Actor reads the columns it needs and fails loudly if a council renames a required column, instead of silently emitting partial records.

GDPR and data

• Only organisation-level records are emitted: supplier name, amount, date, service area, expense type and transaction reference.
• No contact data (emails, phone numbers, addresses) is requested or produced.
• Rows whose beneficiary cannot be confidently identified as an organisation (for example a sole trader's personal name) are skipped by a precision-first filter. This can also skip a few unusual organisation names. The check is name-based, so a surname that happens to match an organisation marker can still slip through; the Actor never adds or infers personal data beyond the payee name the council already publishes.
• The underlying data is published by UK councils under the Open Government Licence v3. See the source link on every item for attribution.

Development and tests

Everything below runs offline against saved real CSV fixtures and spends no Apify credit:

npm install --include=dev
npm test                # lint + typecheck + Vitest (CSV, extraction, GDPR filter, schema, pricing, publish gate)
npm run fixture:run     # extract + validate both council fixtures and print timing
npm run publish:check   # Steadfetch pre-publication gate
npm run build           # tsc production build (dist/main.js)

The tests cover quoted CSV fields and embedded newlines, both councils' column layouts, GBP amount and DD/MM/YYYY date parsing, organisation filtering, date/amount filters, schema acceptance and rejection, retry behaviour and the 1.00 USD per 1,000 price.

FAQ

Do I need an API key or a paid subscription to the council portals? No. The CSV files are public open data, published under the UK Open Government Licence v3.

Does it scrape council websites with a browser? No. It downloads the published CSV files over plain HTTP. No browser, login or CAPTCHA bypass is involved.

Why are some payments missing? Two reasons: the payment is below minAmount or outside dateFrom/dateTo, or the payee could not be confidently identified as an organisation. The second rule is deliberate: Steadfetch filters out payees that look like individuals.

Why does Hounslow return no rows for an older date range? Hounslow publishes one CSV per month and only the newest file stays on the portal. The Actor reads that newest file, so a date range that ends before that month has no Hounslow rows to return. Camden is unaffected: its single export contains the full history.

Can I compare councils in one dataset? Yes. Every item shares the same fields and ISO date format, and council tells you where it came from. Run with both councils selected and analyse the default dataset as one table.

Can I add another council? The architecture is multi-council by design. Each council needs its own CSV URL and column mapping in src/councils.ts; v1 ships Camden and Hounslow.

Is the output schema stable? Yes. New fields are added additively and required fields never change.

Maintenance

This Actor is tested daily against its minimal input. Tests keep both council CSV endpoints reachable and validate the dataset against the published schema. Broken runs are reported to the Steadfetch catalog QA job and repaired before new work.