City-Data City Profiles

Playwright Actor that reads city-data.com city profile pages and optional crime pages, then pushes one JSON object per city. It wraps scripts/city_data/city_data_scraper.py (issue #99). It does not call a paid API and it does not post results to Lovable.

Low-volume research. Keep delaySeconds at 1.5 or higher.

What it does

For each city the Actor opens one Chromium browser, loads https://www.city-data.com/city/{City}-{State}.html, and parses population, income, housing, cost of living, and education with the same BeautifulSoup parsers as the local CLI. When crime is requested it also loads /crime/crime-{City}-{State}.html. Images, fonts, media, and common ad hosts are blocked by the existing session. A missing city page sets ok: false and the run continues with the next city.

Sex-offender lists are not scraped.

Input

fixtures/INPUT.json:

{
  "city": "Clementon",
  "state": "NJ",
  "fields": ["population", "crime"],
  "delaySeconds": 1.5,
  "timeoutMs": 30000,
  "headless": true,
  "maxCities": 1,
  "maxSpendUsd": 5,
  "maxProxyCostUsd": 0.5,
  "allowCaptcha": false,
  "proxyConfiguration": {"useApifyProxy": false}
}

A list works too: cities: [{"city": "Clementon", "state": "NJ", "slug": ""}]. Optional slug overrides the path token. fields is a multi-select of the six groups. headless defaults to true. maxSpendUsd defaults to 5, maxProxyCostUsd to 0.50, and allowCaptcha to false. The proxy group is used only after a block. The default run timeout is 600 seconds at 2048 MB.

Output

One dataset item per city, plus key-value SUMMARY with pushed / blocked / error counts.

{
  "city": "Clementon",
  "state": "NJ",
  "urls": {
    "city": "https://www.city-data.com/city/Clementon-New-Jersey.html",
    "crime": "https://www.city-data.com/crime/crime-Clementon-New-Jersey.html"
  },
  "ok": true,
  "sourceStatus": "ok",
  "scrapedAt": "2026-09-04T19:00:00+00:00",
  "population": {"year": 2024, "total": 5600},
  "crime": {"index": 120.5, "index_year": 2024, "by_year": []}
}

The sample numbers match a live Clementon fetch. A later page will differ. Missing parser keys are omitted. sourceStatus is ok, error, or robots_disallowed. Overview shows the city and status. Detail shows urls, population, income, housing, cost_of_living, education, and crime.

Field glossary

Group	Page	Keys
population	City	year, total, urban/rural %, change since 2000, median age, males, females
income	City	year, median household, per capita, poverty rate
housing	City	median home value, median gross rent, renter %
cost_of_living	City	index, year
education	City (age 25+)	high school or higher %, bachelor's or higher %
crime	Crime page, city table if the crime URL 404s	index, year, vs US average, year-over-year %, homicides, violent/property rates, officers per 1,000, by_year

Limitations

• One browser for the run. Delay default is 1.5 seconds between requests, same as the CLI.
• The session retries a failed navigation once. After a block (HTTP 401, 403, 429, or a challenge page) the Actor may open Apify Proxy once and retry that URL. It does not add another retry loop on top of that.
• A disallowed path is skipped. The row records robots_disallowed and no request is sent. That skip is not retried through a proxy.
• maxSpendUsd defaults to 5 and maxProxyCostUsd defaults to 0.50. allowCaptcha defaults to false. A block with no proxy budget, with CAPTCHA solving off, or with no solver configured pushes sourceStatus: "blocked" and later cities are not requested. This Actor does not ship a solver.
• The proxy group is proxyConfiguration or proxyGroups. An empty group uses datacenter, and only after a block. The first page stays direct. No stealth plugin is added.
• This Actor does not geocode addresses and does not ingest rows into another app.

Robots

Checked against the live https://www.city-data.com/robots.txt while packaging. For User-agent: *, /city/{Name}.html and /crime/crime-{Name}.html are not disallowed. The file does disallow paths such as /so/, /city/js/getBoxes.php*, forums search, and several script endpoints. Named AI crawlers are Disallow: /. This Actor sends the same Chrome user agent as the local scraper and evaluates the * group with the longest-match rules in crm_enrich_robots.py before every URL. If the site later disallows /city/ or /crime/, those fetches stop.

Pricing

Pay per event. The primary event is result at $0.01, charged only when ok is true. A robots skip, a missing page, a blocked row, and any other error are stored and not billed. proxy is $0.01, charged only when a blocked page is retried through Apify Proxy. captcha is $1.00, charged only when a solver returns a token. Direct page loads are free. Dataset items are not a charge event. The platform still applies its start event. Prices are the starting Store rates in .actor/pay_per_event.json. Memory default is 2048 MB and the default timeout is 600 seconds. City-data terms restrict automated collection; keep runs small and leave the delay at 1.5 seconds or higher.

Categories

Set in the Apify Console, not in actor.json. Suggested: Business, Real estate.

Local run

cd scripts/apify-scripts/city-data
mkdir -p storage/key_value_stores/default
cp fixtures/INPUT.json storage/key_value_stores/default/INPUT.json
apify run --purge

Publish with APIFY_TOKEN (apify push). That token is not APIFY_API_KEY. See scripts/apify-scripts/README.md.