New York Open Data Scraper

New York Open Data Scraper

Scrape any public dataset from data.ny.gov, up to a million rows per run. Filter by column, full-text search, or SQL-style WHERE clause, and get back only the columns you need. Export to CSV, JSON, Excel, or XML.

New York State publishes thousands of public datasets on data.ny.gov, from driver licenses and liquor licenses to campaign finance and environmental records. The portal's web interface limits you to browsing one page at a time and downloading static snapshots. This Actor reads the live Socrata API behind the portal, so you can pull exactly the rows and columns you need with full-text search, column filters, and SQL-like expressions, all in one automated run.

Who uses it	What they scrape New York Open Data for
Journalists and newsrooms	Pull the latest liquor license applications or campaign finance filings for a story on local business or politics.
Compliance officers	Monitor active driver license records or professional certifications for expiration and renewal tracking.
Data analysts	Feed clean, filtered New York State open data into dashboards and statistical models without manual downloads.
Academic researchers	Collect multi-year environmental or health datasets for longitudinal studies and policy analysis.
Real estate developers	Gather property tax, building permit, and zoning datasets to evaluate sites and market conditions.

What it does

This Actor collects rows from a single data.ny.gov dataset identified by its Resource ID and returns each row as a flat record with the columns you select.

• 🔍 Full-text search: search across every column in the dataset with a single query term.
• 📋 Column filters: pass a JSON object of exact-match filters like {"county": "BRONX"} to narrow rows server-side.
• 🗂️ Select columns: return only the columns you name, reducing payload size and post-processing time.
• 📊 Aggregation support: group rows and apply HAVING clauses for summary statistics like counts by county.
• ⚙️ SQL-style WHERE clause: write advanced filter expressions with operators and date comparisons.

Results export to CSV, JSON, Excel, or XML, or straight from the API.

What you can do with New York Open Data data

📰 Investigate liquor license activity.

A journalist pulls the Liquor Authority dataset filtered by county and license type to map new applications and renewals in a specific borough.

📈 Track driver license counts.

A compliance analyst groups the Active Driver License dataset by county and license class to report on credential distribution across the state.

🏗️ Evaluate building permits.

A developer searches the Building Permits dataset for a keyword and filters by issue date to assess construction activity in a target zip code.

🌿 Monitor environmental records.

A researcher collects multi-year air quality readings with a WHERE clause on date and a sort order to feed a time-series model.

💰 Audit campaign finance.

A watchdog group pulls campaign finance filings filtered by candidate and contribution amount to identify large donors in a state election cycle.

Why choose this scraper

	What you get
Live API access	Reads the Socrata API directly, so you always get the current data, not a stale export.
Column selection	Request only the columns you need to keep datasets small and processing fast.
Server-side filtering	Filters, search, and WHERE clauses run on the server, so you download only matching rows.
Large volume support	Collect up to a million rows per run, enough for the largest state datasets.
Structured output	Every row arrives in a consistent flat schema ready for CSV, JSON, Excel, or XML export.

How it compares

The competitors below are popular Apify scrapers for different sources. None of them target data.ny.gov or the Socrata open data API, so this Actor is the only one built specifically for New York State public datasets.

Feature	ParseForge	Google Maps Scraper	Instagram Scraper	Website Content Crawler
Targets data.ny.gov datasets	Yes	Not listed	Not listed	Not listed
Reads Socrata open data API	Yes	Not listed	Not listed	Not listed
Column selection	Yes	Not listed	Not listed	Not listed
SQL-style WHERE clause filtering	Yes	Not listed	Not listed	Not listed
Aggregation with GROUP BY and HAVING	Yes	Not listed	Not listed	Not listed
No API key or login required	Yes	Not listed	Not listed	Not listed

Configure the run

Drive the Actor with a Resource ID from data.ny.gov, then narrow results with full-text search, column filters, a WHERE clause, and a sort order so only matching rows reach your dataset. The Input tab lists every parameter.

A first run with the defaults:

{
 "maxItems": 10,
 "resourceId": "9a8c-vfzj"
}

A larger pull:

{
 "maxItems": 200,
 "resourceId": "9a8c-vfzj"
}

Pricing

Pay-per-result: $0.01733 per result collected. You pay only for the results written to your dataset.

Results collected	Approximate cost
100 results	$1.73
1,000 results	$17.33
10,000 results	$173.30

New Apify accounts start with $5 in free credit.

Free users

Free-plan runs return up to 10 results as a preview. Upgrade your Apify plan to collect up to 1,000,000 results per run.

Run it

1. Create a free Apify account with $5 in credit.
2. Open the New York Open Data Scraper.
3. Set your inputs and any filters, then click Start.
4. Export the results as CSV, Excel, JSON, or XML from the Dataset tab.

Run it programmatically through the Apify API (run-sync-get-dataset-items) or the ApifyClient for JavaScript and Python.

Use with AI agents (MCP)

Give an AI agent live access to New York Open Data through the Model Context Protocol. Add the Actor to Claude, Cursor, or any MCP client:

$claude mcp add --transport http apify "https://mcp.apify.com?tools=parseforge/data-ny-gov-new-york-open-data-scraper"

Then prompt it in plain language to run the scraper and read back the results.

Troubleshooting

Why am I getting no results?

Check that your Resource ID is correct by visiting the dataset page on data.ny.gov. Also verify that your filter values match the exact casing and spelling used in the dataset, and that your WHERE clause syntax is valid.

Why does my run fail with an error about the Resource ID?

The Resource ID must be the exact identifier from the dataset URL on data.ny.gov. It is case-sensitive. Copy it directly from the URL bar after opening the dataset.

Why are some columns missing from my output?

If you used the Select Columns field, only those columns are returned. Leave the field blank to get all columns. Also check that the column names match exactly, including underscores and capitalization.

Why is my WHERE clause not working?

Wrap string values in single quotes and use correct Socrata SQL syntax. For example: "county = 'BRONX' AND date_created > '2024-01-01'". Column names with spaces need double quotes around them.

Why does the run time out on a large dataset?

Use the Select Columns field to reduce the payload, add filters to narrow the result set, or increase the run timeout in your Apify actor settings.

Why do I get a 'column not found' error?

Column names are case-sensitive and must match the dataset schema exactly. Visit the dataset page on data.ny.gov, click 'Export', and check the column names in the API documentation or the column listing.

FAQ

Question	Answer
How do I find the Resource ID for a dataset?	Visit data.ny.gov, open the dataset you want, and look at the URL. The Resource ID is the last segment, for example '9a8c-vfzj' in 'data.ny.gov/Transportation/Active-Driver-License-Information/9a8c-vfzj'.
Can I scrape more than one dataset in a single run?	Each run targets one Resource ID. To collect multiple datasets, run the Actor once per Resource ID, either sequentially or in parallel using separate runs.
What format does the output come in?	The Actor returns data in the Apify dataset format, which you can export to CSV, JSON, Excel, XML, or other formats from the Apify console or API.
Does this Actor require an API key or login?	No. It reads the public Socrata API that powers data.ny.gov, which is open and does not require authentication.
How do I filter rows by a specific column value?	Use the Column Filters input as a JSON object, for example {"county": "BRONX"}, or write a WHERE clause like "county = 'BRONX'" for more complex conditions.
Can I get only certain columns instead of the whole dataset?	Yes. List the columns you want in the Select Columns field as a comma-separated string, for example 'license_number, city, county'.
How many rows can I collect in one run?	You can set the Maximum datasets input up to 1,000,000 rows. The actual number returned depends on how many rows match your filters.
Does the Actor support sorting?	Yes. Use the Sort Order field with a column name and direction, for example 'license_number DESC' or 'city ASC'.
Can I do aggregations like COUNT or SUM?	Yes. Use the Group By field to group rows and include aggregate functions like count(*) in the Select Columns field. Add a Having Clause to filter on the aggregated values.
What if my dataset has thousands of columns?	Use the Select Columns field to request only the columns you need. This keeps the payload small and the run fast.
Is this Actor suitable for daily scheduled runs?	Yes. You can schedule it on Apify to run daily, weekly, or on any cron schedule, and it will pull the latest data from the live API each time.
What happens if the dataset is empty or my filters match nothing?	The run completes successfully and returns an empty dataset. Check your filter values and Resource ID if you expected results.

Related actors

Browse the full ParseForge collection for more scrapers.

🆘 Need help? Email parseforge@protonmail.com with your run ID, your input, and what you expected.

⚠️ Disclaimer. This Actor is unofficial and is not affiliated with, endorsed by, or sponsored by New York State. It collects only publicly available data. You are responsible for using the collected data in compliance with the source's terms of service and applicable data-protection laws, including GDPR, CCPA, and PIPL. Do not use it to collect personal data unlawfully.