CDC NNDSS Public Health Surveillance Harvester

🩺 CDC NNDSS Public Health Surveillance Harvester is a powerful Apify Actor designed to collect disease surveillance records from the CDC's National Notifiable Diseases Surveillance System (NNDSS) open data. This tool provides Public Health Surveillance Harvester intelligence including disease indicator, reporting area, MMWR week-ending date, reported value, and a source link for every record. Whether you're tracking a disease over time, building a public health dashboard, or researching outbreak trends, the Public Health Surveillance Harvester delivers structured results efficiently.

With keyword search, reporting-area and year filters, a disease-name listing mode, automatic retries, and reliable Apify Dataset delivery, this Public Health Surveillance Harvester actor turns the CDC's open data into clean, ready-to-use records. It focuses on key signals including indicator name, location, week date, and reported value, making Public Health Surveillance Harvester a practical tool for epidemiology research and public health monitoring.

---

📋 Table of Contents

• Features
• Data Source
• How It Works
• Input
• Output
• Technical Stack
• Data Fields
• Use Cases
• Quick Start
• Configuration
• Performance
• Important Notes
• License & Legal

---

🔥 Features

• CDC Surveillance Data – Pulls records from the CDC's official open data portal (data.cdc.gov). This is the core of Public Health Surveillance Harvester.
• Keyword Search – Case-insensitive search across disease and indicator names (default: measles).
• Reporting Area Filter – Narrow results to a state or reporting area with the state input.
• Year Filter – Limit results to a single year with the year input.
• Disease Name Listing Mode – Set listOnly to get a list of the disease names that match your keyword, so you can pick the right search term.
• MMWR Week-Ending Dates – Converts each record's MMWR year and week into the Saturday that ends that week (YYYY-MM-DD).
• Newest First – Records are ordered by year and week, most recent first.
• Empty Weeks Skipped – Records without a reported numeric value are excluded.
• Automatic Retries – Retries failed requests up to 3 times with increasing wait between attempts.
• Apify Proxy Support – Optional Apify Proxy for reliable Public Health Surveillance Harvester access.
• Proxy Fallback Handling – Retries without a proxy if a proxied request fails with an error.
• Single-Batch Dataset Push – Pushes all records to the Apify Dataset together.
• Detailed Logging – Logs proxy status, search settings, and results for every run.

---

📊 Data Source

CDC Open Data — NNDSS (data.cdc.gov)

• Authority: US Centers for Disease Control and Prevention (CDC), National Notifiable Diseases Surveillance System
• Endpoint: https://data.cdc.gov/resource/x9gk-5huc.json (Socrata Open Data API, queried with SoQL)
• Access Method: Browser-impersonated HTTP requests via curl_cffi (Chrome 124 fingerprint), parsing the JSON response
• Coverage: Disease indicators and reporting areas available in this CDC dataset
• Data: Indicator label, reporting area, year, MMWR week, and reported value
• Access: Public API, no API key required
• Update Frequency: Reflects the dataset's contents at time of run

---

⚙️ How It Works

The Public Health Surveillance Harvester accepts a keyword and optional area and year filters. It builds a SoQL query, requests matching records from the CDC API, converts each record's MMWR year and week into a calendar date, and pushes the cleaned records to the Apify Dataset as one batch. In listOnly mode it instead returns the distinct disease names that match your keyword.

Key Processing Steps:

1. Input Parsing – Read keyword, state, year, maxResults, and listOnly
2. Proxy Setup – Configure Apify Proxy if enabled, with automatic direct-connection fallback
3. Query Building – Build the filter: records with a reported value, keyword match, optional area match, optional year
4. API Request – Fetch records ordered by year and week (newest first), up to maxResults
5. Date Conversion – Convert MMWR year and week into the week-ending (Saturday) date
6. Normalization – Keep records with a numeric value and build one clean record each
7. Dataset Push – Push all Public Health Surveillance Harvester records at once
8. Summary Logging – Report how many records were saved

Key Benefits:

• Search CDC surveillance data without writing SoQL queries
• Get consistent records with real calendar dates instead of raw MMWR week numbers
• Find the exact disease names available with listOnly before running a full search
• Automate recurring Public Health Surveillance Harvester runs to track new weekly data

---

📥 Input

Field	Type	Default	Description
keyword	string	"measles"	Text to match (case-insensitive, partial match) against the indicator/disease name.
state	string	(none)	Optional reporting area filter (case-insensitive, partial match).
year	integer	(none)	Optional year filter.
maxResults	integer	100	Maximum records to fetch (capped at 50,000).
listOnly	boolean	false	If true, returns only the matching disease names instead of data records.
proxyConfiguration	object	{"useApifyProxy": true}	Apify Proxy configuration.

Example Input:

{
  "keyword": "measles",
  "year": 2025,
  "maxResults": 100
}

Filter by Reporting Area:

{
  "keyword": "pertussis",
  "state": "California",
  "maxResults": 200
}

List Matching Disease Names:

{
  "keyword": "hepatitis",
  "listOnly": true
}

---

📤 Output

Data records (default mode)

Field	Type	Description
indicator	string	Disease or indicator name from the CDC dataset
location	string	Reporting area
date	string	MMWR week-ending date (YYYY-MM-DD), or YYYY-W<week> if the date cannot be calculated
value	number	Reported value for that indicator, area, and week
link	string	API link that returns the source record from the CDC dataset

Example Public Health Surveillance Harvester Record:

{
  "indicator": "Measles, Total",
  "location": "TEXAS",
  "date": "2025-03-29",
  "value": 12,
  "link": "https://data.cdc.gov/resource/x9gk-5huc.json?label=Measles%2C+Total&states=TEXAS&year=2025&week=13"
}

Disease name records (listOnly mode)

Field	Type	Description
disease_name	string	A distinct indicator name matching your keyword (up to 500)

---

🧰 Technical Stack

• HTTP Client: curl_cffi with Chrome 124 TLS fingerprint impersonation
• Source: CDC Socrata Open Data API (SoQL queries)
• Date Handling: MMWR week calculation (Sunday–Saturday weeks, week 1 is the first week with at least 4 days in the year), with timezone-aware UTC
• Async: asyncio with executor-run blocking requests
• Proxy: Apify Proxy, optional via useApifyProxy, with automatic no-proxy fallback on failure
• Logging: Apify Actor logging system
• Platform: Apify Actor serverless environment

---

📊 Data Fields Explained

What

• indicator: The disease or condition the value refers to

Where

• location: The reporting area for the value

When

• date: The Saturday that ends the MMWR week of the record

How Much

• value: The reported numeric value from the dataset

Source

• link: API link to the source record in the CDC dataset

---

🎯 Use Cases

• Disease Trend Tracking – Follow a disease over weeks and years
• Public Health Dashboards – Feed charts and dashboards with clean time-series data
• Epidemiology Research – Collect surveillance data for analysis
• Regional Comparison – Compare reporting areas for the same indicator
• Journalism & Reporting – Gather public figures for data-driven stories
• Alerting & Monitoring – Run on a schedule to track newly published weeks

---

🚀 Quick Start

1. Prepare Input

Go to Apify Console and enter:

{
  "keyword": "measles",
  "year": 2025,
  "maxResults": 100
}

2. Run the Actor

Click Start. The Actor will:

• Connect through Apify Proxy if enabled
• Query the CDC API with your filters
• Convert MMWR weeks into dates
• Push all results to the Dataset as one batch

3. Monitor Progress

Console shows:

🛡️ Apify Proxy connected.
🔎 Fetching CDC NNDSS data for 'measles' (year: 2025) (up to 100)...
✅ Successfully harvested 100 CDC surveillance records to dataset!
🎉 Complete!

4. View & Download Results

• Results Tab: All Public Health Surveillance Harvester records
• Export: JSON, CSV, Excel, or HTML
• API Access: Available via the Apify API

---

⚙️ Configuration

Default run:

{
  "keyword": "measles",
  "maxResults": 100
}

Find the right disease name first:

{
  "keyword": "influenza",
  "listOnly": true
}

Direct connection without proxy:

{
  "keyword": "mumps",
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}

---

📈 Performance

Processing Speed

• A single API request per run (plus retries if needed)
• Up to 50,000 records can be requested in one call

Resource Usage

• Memory: Low, since no browser instance is launched
• Network: One lightweight request per run
• Proxy: Optional Apify Proxy with automatic no-proxy fallback on failure

---

⚠️ Important Notes

Legal & Compliance

• Public Data Only: Retrieves only publicly available CDC open data
• Fair Use: Uses a single request per run to keep load on the source minimal
• Not Medical Advice: This data is for research and information purposes, not for diagnosis or treatment decisions
• Legal: Not legal advice

Data Quality

• Source Figures: Values are passed through exactly as published in the CDC dataset. Surveillance counts can be provisional and may be revised by the CDC, so confirm figures with the CDC before official or published use.
• Keyword Matching: Matching is a partial, case-insensitive text match, so broad keywords can return several related indicators. Use listOnly to see the exact names.
• Reporting Area Matching: The state filter is also a partial match against the dataset's reporting area names.
• Skipped Records: Records without a numeric value are left out.
• Date Conversion: date is calculated from the MMWR year and week. If that fails, the value is shown as YYYY-W<week>.
• Link Field: link is a JSON API link to the matching record, not a human-readable web page.
• Empty Results: If nothing matches, the dataset is empty and a warning is logged. In listOnly mode, nothing is pushed when no names match.
• Failures: If the CDC API cannot be reached after 3 attempts, the run ends with an error.

Best Practices

• Run with listOnly first to find the exact disease name you want
• Combine keyword, state, and year to keep results focused
• Keep maxResults close to what you need
• Run on a schedule to pick up newly published weeks
• Verify important figures on the CDC website before publishing

---

📄 License & Legal

Terms of Use:

• Use for legitimate research, reporting, and public health analysis
• Respect the terms of use of the CDC open data portal
• Use data responsibly and in compliance with applicable laws

Disclaimer: Public Health Surveillance Harvester is provided as-is for research and informational purposes. It is not affiliated with or endorsed by the CDC. It does not provide medical advice. Users are responsible for compliance with all applicable laws and the data source's terms. This is not legal advice.

---

⚖️ Public Health Surveillance Harvester Excellence

This Actor is optimized for Public Health Surveillance Harvester research with:

• ✅ Official CDC open data
• ✅ Keyword, reporting-area, and year filters
• ✅ Disease name listing mode
• ✅ MMWR week-to-date conversion
• ✅ Automatic retries and proxy fallback
• ✅ Production-ready code