# City Data Scraper – Population, Crime & Demographics (`staticxviper/city-data-scraper`) Actor

city-data.com can be used to track important data for real estate investments, or general market/demographic data.

- **URL**: https://apify.com/staticxviper/city-data-scraper.md
- **Developed by:** [CJ Moulton](https://apify.com/staticxviper) (community)
- **Categories:** Lead generation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $5.00 / 1,000 city profiles

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## City-Data City Profiles

Playwright Actor that reads [city-data.com](https://www.city-data.com/) city profile pages and optional crime pages, then pushes one JSON object per city. It wraps `scripts/city_data/city_data_scraper.py` (issue #99). It does not call a paid API and it does not post results to Lovable.

Low-volume research. Keep `delaySeconds` at 1.5 or higher.

### What it does

For each city the Actor opens one Chromium browser, loads `https://www.city-data.com/city/{City}-{State}.html`, and parses population, income, housing, cost of living, and education with the same BeautifulSoup parsers as the local CLI. When `crime` is requested it also loads `/crime/crime-{City}-{State}.html`. Images, fonts, media, and common ad hosts are blocked by the existing session. A missing city page sets `ok: false` and the run continues with the next city.

Sex-offender lists are not scraped.

### Input

`fixtures/INPUT.json`:

```json
{
  "city": "Clementon",
  "state": "NJ",
  "fields": ["population", "crime"],
  "delaySeconds": 1.5,
  "timeoutMs": 30000,
  "headless": true,
  "maxCities": 1,
  "maxSpendUsd": 5,
  "maxProxyCostUsd": 0.5,
  "allowCaptcha": false,
  "proxyConfiguration": {"useApifyProxy": false}
}
```

A list works too: `cities: [{"city": "Clementon", "state": "NJ", "slug": ""}]`. Optional `slug` overrides the path token. `fields` is a multi-select of the six groups. `headless` defaults to true. `maxSpendUsd` defaults to 5, `maxProxyCostUsd` to 0.50, and `allowCaptcha` to false. The proxy group is used only after a block. The default run timeout is 600 seconds at 2048 MB.

### Output

One dataset item per city, plus key-value `SUMMARY` with pushed / blocked / error counts.

```json
{
  "city": "Clementon",
  "state": "NJ",
  "urls": {
    "city": "https://www.city-data.com/city/Clementon-New-Jersey.html",
    "crime": "https://www.city-data.com/crime/crime-Clementon-New-Jersey.html"
  },
  "ok": true,
  "sourceStatus": "ok",
  "scrapedAt": "2026-09-04T19:00:00+00:00",
  "population": {"year": 2024, "total": 5600},
  "crime": {"index": 120.5, "index_year": 2024, "by_year": []}
}
```

The sample numbers match a live Clementon fetch. A later page will differ. Missing parser keys are omitted. `sourceStatus` is `ok`, `error`, or `robots_disallowed`. Overview shows the city and status. Detail shows `urls`, `population`, `income`, `housing`, `cost_of_living`, `education`, and `crime`.

#### Field glossary

| Group | Page | Keys |
|-------|------|------|
| `population` | City | year, total, urban/rural %, change since 2000, median age, males, females |
| `income` | City | year, median household, per capita, poverty rate |
| `housing` | City | median home value, median gross rent, renter % |
| `cost_of_living` | City | index, year |
| `education` | City (age 25+) | high school or higher %, bachelor's or higher % |
| `crime` | Crime page, city table if the crime URL 404s | index, year, vs US average, year-over-year %, homicides, violent/property rates, officers per 1,000, `by_year` |

### Limitations

- One browser for the run. Delay default is 1.5 seconds between requests, same as the CLI.
- The session retries a failed navigation once. After a block (HTTP 401, 403, 429, or a challenge page) the Actor may open Apify Proxy once and retry that URL. It does not add another retry loop on top of that.
- A disallowed path is skipped. The row records `robots_disallowed` and no request is sent. That skip is not retried through a proxy.
- `maxSpendUsd` defaults to 5 and `maxProxyCostUsd` defaults to 0.50. `allowCaptcha` defaults to false. A block with no proxy budget, with CAPTCHA solving off, or with no solver configured pushes `sourceStatus: "blocked"` and later cities are not requested. This Actor does not ship a solver.
- The proxy group is `proxyConfiguration` or `proxyGroups`. An empty group uses datacenter, and only after a block. The first page stays direct. No stealth plugin is added.
- This Actor does not geocode addresses and does not ingest rows into another app.

### Robots

Checked against the live `https://www.city-data.com/robots.txt` while packaging. For `User-agent: *`, `/city/{Name}.html` and `/crime/crime-{Name}.html` are not disallowed. The file does disallow paths such as `/so/`, `/city/js/getBoxes.php*`, forums search, and several script endpoints. Named AI crawlers are `Disallow: /`. This Actor sends the same Chrome user agent as the local scraper and evaluates the `*` group with the longest-match rules in `crm_enrich_robots.py` before every URL. If the site later disallows `/city/` or `/crime/`, those fetches stop.

### Pricing

Pay per event. The primary event is `result` at $0.01, charged only when `ok` is true. A robots skip, a missing page, a blocked row, and any other error are stored and not billed. `proxy` is $0.01, charged only when a blocked page is retried through Apify Proxy. `captcha` is $1.00, charged only when a solver returns a token. Direct page loads are free. Dataset items are not a charge event. The platform still applies its start event. Prices are the starting Store rates in `.actor/pay_per_event.json`. Memory default is 2048 MB and the default timeout is 600 seconds. City-data terms restrict automated collection; keep runs small and leave the delay at 1.5 seconds or higher.

### Categories

Set in the Apify Console, not in `actor.json`. Suggested: Business, Real estate.

### Local run

```bash
cd scripts/apify-scripts/city-data
mkdir -p storage/key_value_stores/default
cp fixtures/INPUT.json storage/key_value_stores/default/INPUT.json
apify run --purge
```

Publish with `APIFY_TOKEN` (`apify push`). That token is not `APIFY_API_KEY`. See `scripts/apify-scripts/README.md`.

# Changelog

This Actor's version history is a separate document: https://apify.com/staticxviper/city-data-scraper/changelog.md

# Actor input Schema

## `cities` (type: `array`):

Objects with city, state, and optional slug. Overrides city + state when set.

## `city` (type: `string`):

Single city name, used when cities is empty.

## `state` (type: `string`):

Two-letter state code for the single-city form.

## `slug` (type: `string`):

Optional city-data path token for the single-city form.

## `fields` (type: `array`):

Groups to parse. Crime also fetches the crime page.

## `delaySeconds` (type: `number`):

Pause between page loads. Default 1.5, same as the local CLI.

## `timeoutMs` (type: `integer`):

Playwright navigation timeout.

## `maxCities` (type: `integer`):

Optional cap on how many cities from the list are scraped.

## `headless` (type: `boolean`):

Run Chromium headless. Default true.

## `maxSpendUsd` (type: `number`):

Stop when the next result, proxy, or CAPTCHA event would pass this amount. Default 5, maximum 100. Later rows are saved and are not billed.

## `maxProxyCostUsd` (type: `number`):

Cap on proxy events alone. Default 0.50, clamped to maxSpendUsd. The first city page stays direct. A proxy retry happens only after a block, and only when this cap and maxSpendUsd both still have room. 0 skips the proxy retry.

## `allowCaptcha` (type: `boolean`):

Default false. When false, a challenge after the proxy retry pushes a blocked row and exits without a solve charge. When true, a solve is attempted only if that proxy retry is still a challenge and maxSpendUsd covers the captcha event. This Actor does not ship a solver, so an unconfigured solve pushes blocked and is not billed.

## `proxyConfiguration` (type: `object`):

Apify Proxy group used only after a direct page is blocked. Leave Apify Proxy off so the first attempt stays direct. Empty groups use datacenter.

## `proxyGroups` (type: `array`):

Optional group names, such as DATACENTER or RESIDENTIAL. Used after a block when proxyConfiguration has no apifyProxyGroups.

## Actor input object example

```json
{
  "cities": [
    {
      "city": "Clementon",
      "state": "NJ"
    }
  ],
  "city": "Clementon",
  "state": "NJ",
  "fields": [
    "population",
    "income",
    "housing",
    "cost_of_living",
    "education",
    "crime"
  ],
  "delaySeconds": 1.5,
  "timeoutMs": 30000,
  "headless": true,
  "maxSpendUsd": 5,
  "maxProxyCostUsd": 0.5,
  "allowCaptcha": false,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `results` (type: `string`):

Dataset containing all scraped items

## `files` (type: `string`):

Key-value store containing the SUMMARY record for this run

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "cities": [
        {
            "city": "Clementon",
            "state": "NJ"
        }
    ],
    "city": "Clementon",
    "state": "NJ"
};

// Run the Actor and wait for it to finish
const run = await client.actor("staticxviper/city-data-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "cities": [{
            "city": "Clementon",
            "state": "NJ",
        }],
    "city": "Clementon",
    "state": "NJ",
}

# Run the Actor and wait for it to finish
run = client.actor("staticxviper/city-data-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "cities": [
    {
      "city": "Clementon",
      "state": "NJ"
    }
  ],
  "city": "Clementon",
  "state": "NJ"
}' |
apify call staticxviper/city-data-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,staticxviper/city-data-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/bEPZLDU9fdBBbMjuK/builds/WlS1hvi9IvOfBVox7/openapi.json