# Socrata Open Data API Scraper (`devilscrapes/socrata-open-data-scraper`) Actor

Pull raw records from any Socrata Open Data (SODA) portal -- NYC, Chicago, and hundreds more US city, county, and state portals. Give us portal domains and dataset IDs; we page through results and retry on throttling, returning typed rows for 311 complaints, crime data, inspections, or any dataset.

- **URL**: https://apify.com/devilscrapes/socrata-open-data-scraper.md
- **Developed by:** [DevilScrapes](https://apify.com/devilscrapes) (community)
- **Categories:** Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

<div align="center">
  <img src=".actor/icon.svg" width="160" alt="Devil Scrapes mark" />

## Socrata Open Data API Scraper

**💰 $2.20 / 1 000 results**  ·  pay only for results  ·  no credit card to try

*We do the dirty work so your dataset stays clean.* 😈

Pull raw records from any Socrata Open Data (SODA) portal in one run — NYC, Chicago, and hundreds more US city, county, and state portals. Give us portal domains and dataset IDs; we page through results and retry on throttling, returning typed rows for 311 complaints, crime data, inspections, or any dataset a portal publishes.

</div>

***

### 🎯 What this scrapes

Several hundred US (and some EU) city, county, and state governments publish
their open data through the **Socrata Open Data API (SODA)** — a standard
`/resource/<dataset-id>.json` REST interface with a SQL-like query language
(SoQL). Every portal exposes a different dataset: NYC's 311 complaints,
Chicago's crime reports, Seattle's permits, LA's budget lines — same
protocol, completely different columns.

This Actor is a **generic, cross-portal harvester**, not a single-vertical
scraper. You supply a list of `{portal_domain, dataset_id}` pairs (plus
optional SoQL filters), and we return every matching row from every target
in one run — whatever shape that dataset happens to be.

### 🔥 Features — what we handle for you

- 🔁 **We retry with exponential backoff** on `429` and `5xx` responses —
  up to 5 attempts per page, honouring `Retry-After` when a portal sends it.
- 📄 **We page every target for you** — `$limit`/`$offset` walked
  automatically until the dataset is exhausted or your cap is hit.
- 🧩 **We fan out across portals in one run** — mix a 311 feed and a crime
  feed and get valid, disjoint-column rows for both, no schema conflicts.
- 🧱 **We isolate failures per target** — an unknown or private dataset
  is logged and skipped; your other targets still complete.
- 🧊 **We keep the dataset clean** — Pydantic-validated rows, ISO-8601
  timestamps, a stable `record_id` when the dataset provides one.
- 💰 **You pay only for results that land.** No data → no charge beyond the
  small warm-up fee.

### 💡 Use cases

- **Civic analytics** — pull a city's 311 or crime feed on a schedule and
  track trends over time.
- **Journalism / research** — cross-reference multiple portals' datasets
  without writing a client for each one.
- **Proptech / real-estate diligence** — pull permit, inspection, or code
  violation datasets for a market you're evaluating.
- **Academic / policy research** — bulk-export any Socrata dataset into a
  typed, analysis-ready format.
- **Compliance monitoring** — watch an inspections or recalls dataset for
  new entries.

### ⚙️ How to use it

1. Click **Try for free** at the top of the page.
2. Add one or more `targets` — each is `{portal_domain, dataset_id}` plus
   optional `soql_where` / `soql_select` / `soql_order` / `label`. The
   prefilled example already scrapes NYC 311 (NYPD-filtered) and Chicago
   crime so you can see multi-portal fan-out on the first run.
3. Click **Start**. Output streams into the run's dataset as each target
   finishes paging.
4. Export from **Storage → Dataset** as JSON, CSV, or Excel — or fetch via
   the API.

### 📥 Input

| Field | Type | Required | Default | Notes |
|---|---|:--:|---|---|
| `targets` | `array` | **yes** | 2-item NYC 311 + Chicago crime prefill | List of `{portal_domain, dataset_id, soql_where?, soql_select?, soql_order?, label?}` |
| `page_size` | `integer` | no | `1000` | Internal `$limit` per page request (1–2000) |
| `max_items_per_target` | `integer` | no | `5000` | Row cap per target; `0` = unlimited |
| `proxy_configuration` | `object` | no | `{"useApifyProxy": true}` | Apify Proxy spec |

#### Example input

```json
{
  "targets": [
    {
      "portal_domain": "data.cityofnewyork.us",
      "dataset_id": "erm2-nwe9",
      "soql_where": "agency='NYPD'",
      "label": "NYC 311"
    },
    {
      "portal_domain": "data.cityofchicago.org",
      "dataset_id": "ijzp-q8t2",
      "label": "Chicago crime"
    }
  ],
  "page_size": 1000,
  "max_items_per_target": 20,
  "proxy_configuration": { "useApifyProxy": true }
}
```

### 📤 Output

Every row is one dataset item. `data` carries the raw Socrata row exactly
as returned — untyped by design, since the target dataset's columns are
unknown at build time and change from portal to portal.

| Field | Type | Notes |
|---|---|---|
| `portal_domain` | `string` | Socrata hostname the row came from |
| `dataset_id` | `string` | Socrata 4x4 dataset id |
| `target_label` | `string \| null` | Echoed from the target's `label` input field |
| `record_id` | `string \| null` | The row's `:id`, when the dataset provides one |
| `scraped_at` | `string` | ISO-8601 UTC fetch timestamp |
| `data` | `object` | The raw Socrata row, untouched |

#### Example output

```json
{
  "portal_domain": "data.cityofnewyork.us",
  "dataset_id": "erm2-nwe9",
  "target_label": "NYC 311",
  "record_id": "46882077",
  "scraped_at": "2026-08-31T09:15:03.000Z",
  "data": {
    "unique_key": "46882077",
    "agency": "NYPD",
    "complaint_type": "Noise - Vehicle",
    "status": "Closed"
  }
}
```

### 💰 Pricing

Pay-Per-Event — you pay only when these events fire. **Pricing is not yet
live** on this listing; the figures below are the provisional target,
matching our existing Socrata Actor (`us-building-permit-leads-scraper`),
and will be confirmed in the Apify Console before this Actor is published.

| Event | USD | What it is |
|---|---:|---|
| `actor-start` | $0.20 | One-off warm-up charge per run |
| `result` | $0.002 | Per dataset row emitted |

Example: 1 000 results at the rates above ≈ **$2.20**. No subscription, no
minimum, no card to start — Apify gives every new account free credit.

### 🚧 Limitations

Only the `/resource/<dataset-id>.json` SODA2 endpoint is supported —
metadata/catalog APIs and CKAN portals (a different protocol some cities
use instead of Socrata) are out of scope. `record_id` is `null` whenever
the source dataset has no `:id` column, which is common and expected, not
a bug. This Actor does not normalize columns across datasets — `data` is
the raw row for whatever dataset you point it at.

### ❓ FAQ

**Do I need a Socrata account or API key?**

No. Public Socrata datasets return data with no `$app_token` — this Actor
runs entirely keyless.

**What happens if I give it a bad `dataset_id`?**

That target is logged and skipped; the run still finishes successfully and
your other targets complete normally.

**What if my `soql_where` filter matches nothing?**

The run finishes successfully with zero rows for that target — an empty
result is not an error.

**Can I scrape any Socrata portal, or only NYC/Chicago?**

Any Socrata portal — several hundred US city, county, and state
governments run one. The prefill uses NYC and Chicago only to demonstrate
multi-portal fan-out in a single run.

**Why is `data` a raw untyped object instead of fixed columns?**

Because the dataset is unknown at build time — a 311 feed and a crime feed
share zero columns. Forcing them into one fixed schema would mean dropping
real data; `data` keeps every column the source portal returned.

### 💬 Your feedback

Spotted a bug, hit a weird edge case, or need a new field? Open an issue
on the Actor's **Issues** tab on Apify Console — we ship fixes weekly and
we read every report.

***

<div align="center">

Built by **[Devil Scrapes](https://apify.com/DevilScrapes)** 😈 — a small fleet of
opinionated public-data Actors. Honest pricing, real engineering, zero fine print.

</div>

# Actor input Schema

## `targets` (type: `array`):

List of Socrata portal + dataset pairs to scrape. Each entry: <code>{"portal\_domain": "data.cityofnewyork.us", "dataset\_id": "erm2-nwe9", "soql\_where": "agency='NYPD'", "soql\_select": null, "soql\_order": null, "label": null}</code>. Only <code>portal\_domain</code> and <code>dataset\_id</code> are required per target.

## `page_size` (type: `integer`):

Internal <code>$limit</code> per page request to the Socrata API. Capped well under Socrata's documented ceiling, defensively.

## `max_items_per_target` (type: `integer`):

Stop paging one target after this many rows. <code>0</code> means unlimited. Kept small in the prefill for a fast QA demo run -- raise it for a real run.

## `proxy_configuration` (type: `object`):

Apify Proxy spec. Socrata portals are public APIs with no anti-bot wall, but Apify Proxy still absorbs any per-IP throttling under concurrent runs.

## Actor input object example

```json
{
  "targets": [
    {
      "portal_domain": "data.cityofnewyork.us",
      "dataset_id": "erm2-nwe9",
      "soql_where": "agency='NYPD'",
      "label": "NYC 311"
    },
    {
      "portal_domain": "data.cityofchicago.org",
      "dataset_id": "ijzp-q8t2",
      "label": "Chicago crime"
    }
  ],
  "page_size": 1000,
  "max_items_per_target": 20,
  "proxy_configuration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `datasetItems` (type: `string`):

All dataset items as JSON.

## `datasetItemsCsv` (type: `string`):

Same data exported to CSV.

## `datasetView` (type: `string`):

Open the run dataset in the Console.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "targets": [
        {
            "portal_domain": "data.cityofnewyork.us",
            "dataset_id": "erm2-nwe9",
            "soql_where": "agency='NYPD'",
            "label": "NYC 311"
        },
        {
            "portal_domain": "data.cityofchicago.org",
            "dataset_id": "ijzp-q8t2",
            "label": "Chicago crime"
        }
    ],
    "page_size": 1000,
    "max_items_per_target": 20,
    "proxy_configuration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("devilscrapes/socrata-open-data-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "targets": [
        {
            "portal_domain": "data.cityofnewyork.us",
            "dataset_id": "erm2-nwe9",
            "soql_where": "agency='NYPD'",
            "label": "NYC 311",
        },
        {
            "portal_domain": "data.cityofchicago.org",
            "dataset_id": "ijzp-q8t2",
            "label": "Chicago crime",
        },
    ],
    "page_size": 1000,
    "max_items_per_target": 20,
    "proxy_configuration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("devilscrapes/socrata-open-data-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "targets": [
    {
      "portal_domain": "data.cityofnewyork.us",
      "dataset_id": "erm2-nwe9",
      "soql_where": "agency='\''NYPD'\''",
      "label": "NYC 311"
    },
    {
      "portal_domain": "data.cityofchicago.org",
      "dataset_id": "ijzp-q8t2",
      "label": "Chicago crime"
    }
  ],
  "page_size": 1000,
  "max_items_per_target": 20,
  "proxy_configuration": {
    "useApifyProxy": true
  }
}' |
apify call devilscrapes/socrata-open-data-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,devilscrapes/socrata-open-data-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/M1zQH1ScdKLuwYMmS/builds/GB7aCXYHC7sil2hrb/openapi.json