# UK Council Spending Scraper (Local Authority Payments) (`steadfetch/uk-council-spending-extractor`) Actor

UK council payments over GBP 500 from official open-data portals (Camden, Hounslow). One JSON item per payment: council, vendorName, amountGbp, paymentDate, expenseArea, expenseType, transactionRef. For supplier research and spend transparency. No login, no paywall, no API key.

- **URL**: https://apify.com/steadfetch/uk-council-spending-extractor.md
- **Developed by:** [Steadfetch](https://apify.com/steadfetch) (community)
- **Categories:** Lead generation, Business
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.80 / 1,000 payment delivereds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## UK Council Spending Scraper (Local Authority Payments)

Extract UK council payments over GBP 500 from public open-data portals. Built by [Steadfetch](https://apify.com/steadfetch): web data that does not break.

### What it does

UK councils must publish every payment over GBP 500. Each council publishes a different CSV file with different column names, so comparing them normally means downloading and reshaping one file per council. This Actor normalizes those files into one stable schema:

- **councils**: Camden and Hounslow in v1 (`camden`, `hounslow`);
- **one row per payment**: council, supplier organisation, amount, payment date, service area, expense type and transaction reference;
- **filters** by council, payment-date range and minimum amount;
- **English field names and ISO dates** (`YYYY-MM-DD`), validated against the published dataset schema before anything is saved;
- **organisation-focused output**: rows whose payee cannot be confidently identified as an organisation are skipped by a precision-first filter. It is a name-based check, so it strongly reduces personal-name records but cannot guarantee a 100% match.

The data is downloaded as plain public CSV over HTTP. There is no browser, no login, no paywall, no CAPTCHA bypass and no API key. The Camden portal asks crawlers for a one-second delay, which the Actor respects before each request.

### Input

| Field | Type | Default | Description |
|---|---|---|---|
| `councils` | array of `camden`, `hounslow` | both | Councils to extract. |
| `dateFrom` | date (`YYYY-MM-DD`) | - | Optional inclusive lower bound for the payment date. |
| `dateTo` | date (`YYYY-MM-DD`) | - | Optional inclusive upper bound for the payment date. |
| `minAmount` | number | `500` | Only payments at or above this amount are emitted. |
| `limit` | integer (1-100000) | `100` | Maximum results delivered **per selected council**. |

Minimal input (every field is optional):

```json
{}
```

With `{}` the Actor reads both councils and delivers up to 100 payments per council (200 in total, about 0.20 USD), which keeps a default run well inside the Apify free-plan credit. Raise `limit` for larger extracts. `dateFrom`/`dateTo` narrow the payment-date range, and `minAmount` raises the GBP 500 floor.

### Example output

```json
{
    "council": "camden",
    "vendorName": "ACME Facilities Ltd",
    "amountGbp": 12450.00,
    "paymentDate": "2026-09-29",
    "expenseArea": "Housing",
    "expenseType": "Services",
    "transactionRef": "PL-2026-04431",
    "sourceUrl": "https://opendata.camden.gov.uk/api/v3/views/3ixw-qvb8",
    "url": "https://opendata.camden.gov.uk/api/v3/views/3ixw-qvb8",
    "title": "ACME Facilities Ltd - 2026-09-29",
    "scrapedAt": "2026-10-23T23:30:00.000Z"
}
```

`url` and `scrapedAt` are the common Steadfetch envelope. Every item links to the public dataset it was read from.

### Output fields

| Field | Type | Required | Source |
|---|---|---|---|
| `council` | string | yes | selected council id (`camden`, `hounslow`) |
| `vendorName` | string | yes | `Beneficiary Name` (Camden) / `BeneficiaryName` (Hounslow) |
| `amountGbp` | number | yes | `Amount GBP` (Camden) / `Amount` (Hounslow) |
| `paymentDate` | date | yes | `Payment Date` / `PaymentDate`, normalized from DD/MM/YYYY |
| `expenseArea` | string or null | no | `Organisational Unit` (Camden) / `ServiceCategoryLabel` (Hounslow) |
| `expenseType` | string or null | no | `Purpose` (both councils) |
| `transactionRef` | string or null | no | `Unique Identifier` (Camden) / `TransactionNumber` (Hounslow) |
| `sourceUrl` | URI | yes | public dataset page |
| `url` | URI | yes | same as `sourceUrl` (envelope) |
| `title` | string | no | supplier and payment date |
| `scrapedAt` | date-time | yes | run timestamp (envelope) |

Amounts are numeric GBP values. Dates are normalized to ISO dates so results from both councils can be sorted and compared in one table.

### Pricing

Pay per event (PPE), charged automatically by Apify:

| Event | Charged when | Price |
|---|---|---|
| `apify-actor-start` | once per run start (synthetic) | 0.00005 USD |
| `apify-default-dataset-item` | once per validated payment saved to the dataset (synthetic) | 0.001 USD = **1.00 USD per 1,000 payments** |

There is no custom per-item event and no `Actor.charge()` call: the synthetic dataset event already bills each result, so users are never billed twice. Rows skipped by the organisation filter, the amount filter or the date filter are not pushed and are not charged. Errors and empty runs only incur the start event.

**Works on the Apify free plan** - about 4,999 payments for the $5 monthly credit ((5.00 USD - 0.00005 USD run-start) / 0.001 USD per payment).

### Cost per 1,000 results

Measured on 24 October 2026 in local fixture runs, without spending Apify credit:

- streaming CSV parsing, field normalization and schema validation of 1,000 payments: **70 ms** of local CPU (measured on the real Camden export, 24 October 2026);
- one CSV download per selected council (the Camden export snapshot was about 110 MB uncompressed on 24 October 2026; it is streamed and reading stops at `limit`. Hounslow's monthly file is about 400 KB);
- a run makes at most three HTTP requests (Camden CSV, the Hounslow dataset page and the newest Hounslow CSV) and waits one second between them;
- at 256 MB and Apify's compute price of 0.20 USD per GB-hour, extraction and validation cost approximately **0.000001 USD per 1,000 payments**, plus the 0.00005 USD actor start.

Cost per 1,000 payments is therefore well below **0.01 USD** (the spec ceiling was 0.10 USD). At 1.00 USD per 1,000 the gross margin is above 99%. No proxy is used and the CSV downloads are free, so there is no per-request fee.

### Data sources and coverage

| Council | Portal | Format | Licence |
|---|---|---|---|
| Camden | `opendata.camden.gov.uk` (Socrata, dataset `3ixw-qvb8`) | CSV export, one row per payment | UK Open Government Licence v3 |
| Hounslow | `data.hounslow.gov.uk` (PortalJS / Datopian) | one CSV per month; the Actor discovers and reads only the newest file on every run | UK Open Government Licence v3 |

Hounslow keeps only its newest monthly file on the portal, and the Actor reads that file only. A `dateFrom`/`dateTo` window that ends before that month returns **zero Hounslow rows** (Camden is not affected, because its export carries the full history). For older Hounslow months, download them from the portal directly.

Both files are published as "spending over 500" transaction lists. The Actor reads the columns it needs and fails loudly if a council renames a required column, instead of silently emitting partial records.

### GDPR and data

- Only organisation-level records are emitted: supplier name, amount, date, service area, expense type and transaction reference.
- No contact data (emails, phone numbers, addresses) is requested or produced.
- Rows whose beneficiary cannot be confidently identified as an organisation (for example a sole trader's personal name) are skipped by a precision-first filter. This can also skip a few unusual organisation names. The check is name-based, so a surname that happens to match an organisation marker can still slip through; the Actor never adds or infers personal data beyond the payee name the council already publishes.
- The underlying data is published by UK councils under the Open Government Licence v3. See the source link on every item for attribution.

### Development and tests

Everything below runs offline against saved real CSV fixtures and spends no Apify credit:

```bash
npm install --include=dev
npm test                # lint + typecheck + Vitest (CSV, extraction, GDPR filter, schema, pricing, publish gate)
npm run fixture:run     # extract + validate both council fixtures and print timing
npm run publish:check   # Steadfetch pre-publication gate
npm run build           # tsc production build (dist/main.js)
```

The tests cover quoted CSV fields and embedded newlines, both councils' column layouts, GBP amount and DD/MM/YYYY date parsing, organisation filtering, date/amount filters, schema acceptance and rejection, retry behaviour and the 1.00 USD per 1,000 price.

### FAQ

**Do I need an API key or a paid subscription to the council portals?**
No. The CSV files are public open data, published under the UK Open Government Licence v3.

**Does it scrape council websites with a browser?**
No. It downloads the published CSV files over plain HTTP. No browser, login or CAPTCHA bypass is involved.

**Why are some payments missing?**
Two reasons: the payment is below `minAmount` or outside `dateFrom`/`dateTo`, or the payee could not be confidently identified as an organisation. The second rule is deliberate: Steadfetch filters out payees that look like individuals.

**Why does Hounslow return no rows for an older date range?**
Hounslow publishes one CSV per month and only the newest file stays on the portal. The Actor reads that newest file, so a date range that ends before that month has no Hounslow rows to return. Camden is unaffected: its single export contains the full history.

**Can I compare councils in one dataset?**
Yes. Every item shares the same fields and ISO date format, and `council` tells you where it came from. Run with both councils selected and analyse the default dataset as one table.

**Can I add another council?**
The architecture is multi-council by design. Each council needs its own CSV URL and column mapping in `src/councils.ts`; v1 ships Camden and Hounslow.

**Is the output schema stable?**
Yes. New fields are added additively and `required` fields never change.

### Maintenance

This Actor is **tested daily** against its minimal input. Tests keep both council CSV endpoints reachable and validate the dataset against the published schema. Broken runs are reported to the Steadfetch catalog QA job and repaired before new work.

# Actor input Schema

## `councils` (type: `array`):

Councils to extract. Valid values: camden, hounslow. Defaults to both.

## `dateFrom` (type: `string`):

Optional inclusive lower bound for the payment date (YYYY-MM-DD). Camden covers its full published history; Hounslow only publishes its newest monthly CSV, so dates outside that month return no Hounslow rows.

## `dateTo` (type: `string`):

Optional inclusive upper bound for the payment date (YYYY-MM-DD). Hounslow only publishes its newest monthly CSV: a range that ends before that month returns no Hounslow rows (Camden is not affected).

## `minAmount` (type: `number`):

Only payments at or above this amount are emitted. These datasets are published as spending over GBP 500.

## `limit` (type: `integer`):

Maximum number of payments delivered for each selected council. Defaults to 100 so a run with no other input stays cheap (200 payments for both councils); raise it for larger extracts.

## Actor input object example

```json
{
  "councils": [
    "camden",
    "hounslow"
  ],
  "dateFrom": "2026-09-01",
  "dateTo": "2026-09-30",
  "minAmount": 500,
  "limit": 10
}
```

# Actor output Schema

## `results` (type: `string`):

Dataset items validated against the item schema in dataset_schema.json (fields).

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "councils": [
        "camden",
        "hounslow"
    ],
    "minAmount": 500,
    "limit": 10
};

// Run the Actor and wait for it to finish
const run = await client.actor("steadfetch/uk-council-spending-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "councils": [
        "camden",
        "hounslow",
    ],
    "minAmount": 500,
    "limit": 10,
}

# Run the Actor and wait for it to finish
run = client.actor("steadfetch/uk-council-spending-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "councils": [
    "camden",
    "hounslow"
  ],
  "minAmount": 500,
  "limit": 10
}' |
apify call steadfetch/uk-council-spending-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,steadfetch/uk-council-spending-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/pQSW4b2sIgef7JfeA/builds/8xX8tJ3q1tGMWvkLC/openapi.json