United States Data Center Development Monitor

A Python Actor that returns evidence-backed data center development events from a documented set of official public sources. It reports proposed development, zoning approval, permit issuance, construction start, expansion, and operational opening when source wording supports that stage.

This is a supported-source monitor, not an exhaustive inventory of United States data centers. Enabled sources are CyrusOne's worldwide newsroom, Google's Global Network RSS summaries, and Loudoun County planning records. Operator material is filtered to evidenced US projects; the county supplement covers only Loudoun, Virginia. Source coverage, access restrictions, and missing history are part of every run's output. An empty dataset is not proof that no projects changed.

Supported sources and access

The authoritative machine-readable registry is docs/source-registry.json, with its review date, discovery URLs, policy evidence, access status, and coverage limits. Runtime behavior is defined in src/dc_monitor/sources.py.

Source	Current status	Coverage and restriction
CyrusOne press releases	Enabled, subject to runtime robots and access checks	Bounded first listing page; only US project evidence. No claim to all CyrusOne projects or all US operators.
Google Global Network RSS	Enabled, subject to runtime robots and access checks	Official recent-20-item summary feed, worldwide with evidenced US filtering; no full article ingestion or historical backfill.
Loudoun County planning records	Enabled, subject to runtime robots and access checks	Virginia county planning records, latest 100 queried data center matches; no contacts, geometries, applicant plans, or attachments. A truncated service response is partial.
Vantage	Disabled; permission review required	Restrictive reuse terms. Robots permission alone is insufficient.
Digital Realty investor and corporate newsrooms	Disabled	Investor robots returned 403; restrictive site terms. No bypass.
Equinix US newsroom	Disabled; permission review required	Restrictive reuse terms and unresolved robots response.
QTS	Disabled; written permission required	Legal notices restrict AI/tool use and reuse.
STACK	Disabled; appropriate permission required	Terms grant personal, noncommercial access.
EdgeConneX	Disabled	Public access returned 403.
Chicago building-permit open data	Deferred; disabled	Linked terms returned 403, and issue_date can represent a fee-due stage; matching records did not establish unconditional paid/issued milestones. No permit coverage is claimed.

Disabled sources can be selected to obtain an explicit coverage receipt; input cannot enable them. No arbitrary source URLs, credentials, proxies, private portals, applicant-plan downloads, or contact harvesting are accepted. Source availability is not a content license. Review policy changes before future releases; robots and access denials are respected on each run.

No live building-permit issuance feed is enabled in this version. Permit issuance is supported by deterministic extraction and fixtures when an enabled official source explicitly evidences issuance. Loudoun planning or engineering review approval is not treated as a building permit. Permit-feed expansion remains a documented coverage gap.

Run in Apify Console

Select snapshot mode for a first run. Keep the supported-source defaults and review the displayed pricing and maximum charge. The configured defaults are 256 MB memory, a 90-second platform timeout, and a $0.05 maximum charge. This bounded example fits within those defaults:

{
  "mode": "snapshot",
  "maxRequests": 20,
  "maxResults": 10,
  "maxArticlesPerSource": 25,
  "timeoutSeconds": 8,
  "maxRetries": 0,
  "minDelaySeconds": 2,
  "maxRunSeconds": 60,
  "includeUncertain": true
}

Click Start, then open the Project events dataset and Coverage receipt output. Read COVERAGE before interpreting missing rows: an empty or partial result is not evidence that no US development changed. At the listed rates, this example has at most $0.025 in start and delivered-row charges. Changes mode additionally requires a named store and a genuinely serialized writer; the input confirmation does not create a lock. Source-code access is not needed to use the Console.

Local quick start

The core uses only Python's standard library and requires Python 3.11 or newer. A plain local run requires no Apify account or token.

PYTHONPATH=src python3 -m dc_monitor \
  --input examples/input-default.json \
  --output output/snapshot

This makes a bounded public crawl: at most 20 HTTP requests, 25 articles or records per source, 50 event rows, and 60 seconds. The 25-item window can cover the current CyrusOne listing and Google's recent 20-item RSS feed without truncating them at five items. Requests for robots policies, redirects, and retries consume the same request budget. The source interval is at least two seconds; stricter robots/source rules take priority. Each response is capped at two million bytes. Defaults may still leave a source partial; see the receipt.

Use a fresh output directory for each run. Before crawling, the CLI rejects an output directory containing any existing events.json, coverage.json, baseline.json, observations.json, or run.json; this prevents a stale completion report from being mistaken for a new successful run. It writes those files through atomic per-file replacement. The directory is not a multi-file transaction. A process failure may leave some completed output files; only a completed run.json confirms the CLI finished.

To compare a later crawl with the previous safe candidate:

PYTHONPATH=src python3 -m dc_monitor \
  --input examples/input-changes-local.json \
  --baseline output/snapshot/baseline.json \
  --observations output/snapshot/observations.json \
  --output output/changes

The supplied baseline and observation ledger remain read only. The next candidates are written under the new output directory. Their states, maxArticlesPerSource, includeUncertain, and extractor semantics version must match the run; use separate state when any of these change. This corrected build uses extractor semantics version 3 and deliberately rejects earlier state, which may retain incorrect milestones. Archive the old state and use a new baseline key; no implicit migration or reset is performed. The changes examples select the two operator sources; the separate observation ledger also supports repeat detection for partial county feeds. Pass both files on every local comparison to preserve delivered partial observations.

Inputs and bounds

The complete Actor UI schema is .actor/input_schema.json. Unknown fields, invalid identifiers, duplicate filters, booleans supplied as numeric bounds, and out-of-range values are rejected.

Field	Default	Allowed value
mode	snapshot	snapshot or changes
sourceIds	Enabled registry sources	Supported source IDs; no empty selection
states	All supported US locations	Uppercase USPS state abbreviations plus DC; empty means no state filter
maxRequests	20	1–200
maxResults	50	1–1,000
maxArticlesPerSource	25	1–500
timeoutSeconds	15	1–60
maxRetries	1	0–3
minDelaySeconds	2	1–60
maxRunSeconds	60	5–1,800
maxResponseBytes	2,000,000	10,000–5,000,000
includeUncertain	true	Boolean
baselineStoreName	Omitted	Explicit named store; Actor changes mode only
baselineKey	baseline-v1	Record key within the named store
singleWriterConfirmed	false	Required true for Actor changes persistence

A state filter cannot turn a vague location into a known location. Records without a matching evidenced state do not pass a state filter. Filtering uncertain records does not improve source completeness.

Set the run's maximum charge in Apify Console or the API's maxTotalChargeUsd option. The configured $0.05 cap permits at most 22 delivered rows after the $0.005 start charge, even though the input's maxResults default is 50. Allowing all 50 rows requires a cap of at least $0.105 at the listed prices. maxResults bounds output, while maxRunSeconds, maxRequests, memory, and the platform timeout bound resource use. The default 60-second fetch deadline leaves persistence time within the 90-second platform timeout; increase the platform timeout if you raise that deadline. Never overlap changes-mode runs sharing a store/key.

Event records and evidence

The strict dataset schema is .actor/dataset_schema.json. Each output row includes:

• project_id, project_name, campus_name, phase_name, and scope, preserving campus versus phase/building identity.
• developer and operator, assigned only when the article or structured public record supports that role. The host company and an applicant name alone do not establish a role.
• location with country, state, city, and address. Unknown scalar facts remain null.
• event_type, source_date, UTC first_seen, UTC last_checked, source_urls, and short evidence excerpts with supported field names.
• capacity_mw as separate observations with numeric value, basis (it_load, utility_load, or unknown), and scope (campus, phase, building, or unknown). An empty array means no supported capacity observation. Unknown numbers are never invented; unqualified numbers do not become IT load. Ranges, approximate quantities and upper bounds are omitted from exact numeric assertions and flagged as uncertain; negated capacity is rejected. Campus and phase figures are never added together.
• uncertainty and conflicts, preserving unresolved interpretation and competing values.
• Stable event_id, changed_fields, and change_type (new, updated, or unchanged). Snapshot mode may return unchanged rows; changes mode emits new or materially changed evidence.

examples/output-fixture.json contains six synthetic example records, generated through the actual extractor and reconciliation code, one per supported stage. They demonstrate the output contract and are not real project intelligence; see examples/README.md.

source_date preserves the source publication date or timezone-qualified ISO timestamp; unknown remains null. It is not a claim about the precise date construction or operation started. first_seen is the earliest retained observation in the supplied healthy baseline or delivered-observation ledger, not the project's original announcement date. last_checked reflects this crawl's observation. A previously stored record absent from the current window retains its previous check date.

For an observation from a partial source, the delivered-observation ledger preserves first_seen across retained-state runs while the healthy source baseline stays unchanged. If no ledger is retained or supplied, partial-only observations may repeat with a new first_seen. Records already present in a healthy baseline retain their original first_seen during later partial crawls.

Extraction uses bounded text rules and structured fields, without an LLM or executing source content. It rejects generic substation news, onsite visitor/training/sales facilities misattributed to the data center, unsupported future openings, permit applications represented as permit issuance, and foreign projects whose only US connection is another asset or a headquarters address. Opening actions must refer to the computing asset as their subject or object; a campus used only as an auxiliary facility's location does not qualify. Geography is bound to the event asset, with contextual protection for US state phrases such as New Mexico. Future milestone dates are compared at month/day resolution to the publication date when known, so reading a forecast later does not establish completion. Ambiguous multi-campus announcements may produce an uncertain aggregate with no attributed capacity, or no event at all. Short evidence is retained; entire articles and plans are not republished.

Identity matching is conservative: supported suffix aliases can merge when location and campus/phase identity agree. Separate named campuses stay separate. A unique named asset with the same source URL, stage, date, phase and compatible location retains its ID when a city or address is added. Conflicting locations or competing matches remain separate. Insufficient identity falls back to source-specific evidence; uncertain duplicates may remain. conflicts preserve disagreement instead of silently overwriting facts.

Coverage receipts and safe changes

Every selected source receives a receipt containing its status, discovery URLs, bounded article counts, request use, warnings, errors, and whether its baseline advanced. Missing pagination and historical backfill are disclosed. A completed bounded source window is not a full history scan.

Healthy source state advances independently; failed, disabled, filtered, and partial source state is retained. Output truncation freezes baseline advancement so events are not acknowledged before delivery. Records are append-preserved: a project disappearing from a listing never creates a removal, closure, cancellation, or demolition event. No such event type exists in this version.

Loudoun's latest-100-record query can return an upstream exceededTransferLimit flag because older matching history remains outside that response. Its healthy source baseline stays frozen, even if maxArticlesPerSource is increased to 100 or more. A separate delivered-observation ledger preserves first_seen and suppresses unchanged partial observations in changes mode. It records only delivered results, plus later checks of already delivered unchanged results; unseen rows excluded by maxResults are not acknowledged. Without retaining and supplying that ledger, partial-only observations may repeat and restart first_seen. The ledger does not establish a complete crawl or authorize removal events. Raising the article cap cannot override upstream truncation or the request/result/time bounds.

The Actor writes events to its default dataset and stores these key-value records:

Record	Purpose
COVERAGE	Source receipts, bounds, and delivery/commit status
SNAPSHOT	Safe candidate baseline; retains prior state if dataset delivery is capped or fails
OBSERVATIONS	Delivered-observation candidate; retains prior state if dataset delivery is capped or fails
OUTPUT	Delivered and attempted event counts, charge-limit status, and output references

Actor changes mode requires an explicit baselineStoreName and singleWriterConfirmed: true; see examples/input-changes-actor.json. Configure one non-overlapping writer for the same store/key. The named record at baselineKey contains {state_schema_version: 1, baseline, observations}. The wrapper writes both candidates together after successful dataset delivery, rereads the whole prior state before committing, and stops if another write is detected. Apify key-value storage is not a transactional compare-and-swap lock; concurrent writers remain unsupported. Do not set the confirmation flag unless serialization is actually configured.

Dataset delivery occurs before the combined state commit. Pay-per-event mode requires a positive flat native row price; invalid configuration fails before crawling. If the remaining allowance cannot pay for one row, the Actor skips source fetching, preserves state, and reports COVERAGE.status: skipped with billing_limit_prevented_crawl: true. Otherwise the wrapper checks the SDK's accepted-item count. If a charge cap saves only a prefix of the rows, OUTPUT.event_count reports that delivered prefix, attempted_event_count reports the full attempted batch, and both named state and the SNAPSHOT/OBSERVATIONS recovery records retain prior state. COVERAGE.delivery.events_persisted is false for incomplete delivery. A cap reached exactly after all requested rows is a complete delivery and can commit state.

If dataset writing fails or the priced delivery receipt is invalid, state is not committed. A retry performs a fresh fetch; it does not replay a saved pending batch. Records can change or disappear from the rolling source window before a retry, so recovery of every original row is not guaranteed. A retry after partial delivery or interruption between delivery and commit may also return and charge for previously delivered rows with the same event_id; consumers must deduplicate on that key. Increase the cap enough to deliver the full bounded batch, or narrow the requested source/window and use separate state. The inspected recovery candidates remain distinct from committed named state.

Apify packaging and verification

The Actor metadata, input schema, output links, dataset schema, Dockerfile, and pinned SDK requirements are included. The wrapper uses apify==4.1.0 and its Actor lifecycle and storage API. requirements-lock.txt constrains runtime dependency versions. The Docker image is python:3.14.8-slim-bookworm; its digest is not pinned. Build 1.0.2 completed bounded hosted billing and delivery checks on October 15, 2026.

python3 -m venv .venv
.venv/bin/python -m pip install -r requirements-dev.txt
PYTHONPATH=src .venv/bin/python -m unittest discover -s tests -v

Core tests do not contact live sources. Optional schema tests require jsonschema; the SDK integration test uses temporary local storage with inherited environment variables removed and socket connections disabled. Tests cover evidence semantics, identity, failure and partial results, bounds, repeated runs, schema conformance, dataset failure, and baseline commit ordering. Synthetic fixtures are labeled and do not represent verified real projects.

The existing schemas passed the official @apify/input_schema 4.0.6 and @apify/json_schemas 1.0.1 validators. Hosted checks on October 15, 2026 returned five event rows with five native row events. A $0.007-capped run delivered one of five attempted rows, recorded one native row event, and left named state uncommitted. A subsequent fresh-source retry delivered all five rows, recorded one start and five row events, and committed both the healthy baseline and observation ledger. Its overall coverage remained partial because Loudoun's bounded history is incomplete; delivery itself completed successfully. These checks validate the observed billing and state behavior, not exhaustive coverage or guaranteed future source availability.

Offline release tests also exercise the pinned SDK's native dataset-item pricing with temporary local storage, including caps allowing zero, one, or both of two synthetic event rows after a simulated start charge. Those tests incur no platform charges. Historical public-source checks and their exact limits are recorded separately; fixture success does not establish live recall.

When Docker is available, this is the local build command:

docker build -t us-data-center-monitor:local .

The container runs the Apify entry point and does not contain secrets or a token. This repository does not schedule or deploy itself. Changes to billing or state handling should repeat both offline regressions and bounded hosted receipt checks.

Maintenance and billing

To support a new source, document its official ownership, terms/access review, robots behavior, bounded discovery route, event fields and geographic coverage; add a dedicated adapter and labeled fixtures; then run failure and schema tests. Do not enable a source merely because its page is public. A 403, authentication demand, CAPTCHA, disallowed path, or uncertain policy stops remaining requests for that source. No browser/proxy workarounds are used. DNS resolution is bounded by the shared deadline; validated public addresses are pinned for TCP connection while TLS verifies the original official hostname. A timed-out operating-system DNS worker cannot be canceled, but is a daemon that performs no connection and does not delay process exit.

Billing uses Apify's native synthetic events, without manual Actor.charge calls:

Event	Price	Charged unit
apify-actor-start	$0.005	One run start at up to 1 GB memory
apify-default-dataset-item	$0.002 ($2 per 1,000)	One project-event row delivered to the default dataset

A row is a development-stage/evidence observation, not a unique campus or a guarantee of complete source history. Uncertain or partial-source rows are billed when delivered; includeUncertain: false excludes them from output. Snapshot runs can return the same evidence repeatedly and each delivered copy is billed. Changes mode suppresses unchanged observations after a successful state commit; crash or cap retries can repeat and charge rows. COVERAGE, SNAPSHOT, OBSERVATIONS, and OUTPUT are key-value records and are not billed as dataset rows. Empty-result runs still incur the start event. Consult the displayed Store pricing and the run's maximum charge before starting. docs/billing.md records delivery semantics and the October 15 hosted validation results.

Reproducible release checks

Run PYTHONPATH=src .venv/bin/python scripts/verify.py for the offline QA receipt, and python3 scripts/package.py --output ../dist to create the deterministic ZIP and source manifest. The packager includes only source, schemas, documentation, fixtures and examples; it rejects sensitive filenames and high-confidence credential patterns. scripts/live_check.py --live performs the documented bounded public-source check and an offline replay, saving short extracted evidence rather than fetched articles. Treat returned source-derived strings as untrusted text when rendering or exporting them.

Coverage review and next adapters

The first one-event live sample exposed recall gaps as well as a narrow supported set. The operator audit found an Alabama expansion announcement and a Texas campus explicitly under construction inside the existing default window. Those exact meanings are regression fixtures; historical source-check results are recorded in docs/live-check.json and docs/live-events.json. Increasing the article cap alone cannot fix missed wording. Google remains a recent-20 summary feed; CyrusOne remains one listing page. Its observed older-page link is documented but automatic pagination is not enabled.

docs/coverage-audit-operators.json records the bounded diagnostic reads, including an interrupted follow-up with unknown results. docs/coverage-audit-government.json distinguishes implementation gaps from actual access denials. Prince William DAPS was previously unimplemented; its current robots endpoint returned 403, so that host was stopped. Texas TDLR TABS robots, attribution policy, search page and one known project detail were readable. TABS registrations can support proposal evidence, not permits, starts or openings; search/pagination mechanics still need the documented bounded validation before an adapter is enabled. Other research candidates remain unimplemented or policy/access constrained. The 29-entry research file is a candidate inventory, not a coverage claim.

The externally supplied independent regression script is preserved unchanged at scripts/independent_actor_regressions.py; its artifact provenance and SHA-256 are recorded in docs/independent-review-provenance.json. Run it locally with .venv/bin/python scripts/independent_actor_regressions.py . --json-output docs/independent-regressions.json. It disables sockets and DNS and does not require an Apify account.