US Higher-Ed Intelligence — Rankings + Scorecard + IPEDS

One school, one row, three verified sources. Pulls US News Best Graduate School rankings (9 categories), College Scorecard (earnings / completion / aid), and official IPEDS data files (directory, tuition, graduation rates, faculty salaries, enrollment), joins them by IPEDS UNITID, and emits clean, typed records with full raw-record preservation and a per-component run_summary.

Why

Existing store actors scrape single tables. This is the only actor that produces a cross-source school profile: US News rank + Scorecard earnings + IPEDS tuition/graduation/salary/enrollment in one row.

Input

Field	Type	Default	Notes
datasets	select (multiple)	all three	usnews_grad_rankings, scorecard, ipeds
gradCategories	select (multiple)	all 9	law / medical / business / engineering / nursing / education / computer-science / public-affairs / health
ipedsTables	select (multiple)	all 5	directory, tuition, graduation, salaries, enrollment
stateFilter	string	(off)	2-letter code, e.g. CA
typeFilter	select	(off)	public / private (IPEDS CONTROL)
maxResults	select	20	Output cap (records; or schools in joined mode)
outputMode	select	joined	joined / normalized / raw / summary
scorecardApiKey	secret	DEMO_KEY	free key at api.data.gov; needed for large pulls (DEMO_KEY is heavily rate limited)

Join model

• Join key: IPEDS UNITID (native in Scorecard id and IPEDS rows).
• US News items carry no UNITID; it is resolved via school slug and name matching against the IPEDS directory (tiers: slug+state → name+state → name → loose name variants → unique name prefix). Unmatched records keep ipeds_unitid: null with match_method: "unmatched" — never invented.
• In joined mode the first selected source acts as anchor (priority: usnews → scorecard → ipeds); the other sources are then fetched targeted by UNITID (Scorecard id= batches, IPEDS row filtering), so every output row carries all selected sources.

Output

outputMode: joined — one record per school:

{
  "ipeds_unitid": "243744",
  "school_name": "Stanford University", "city": "Stanford", "state": "CA",
  "sources_present": ["usnews_grad", "scorecard", "ipeds"],
  "match_methods": {"usnews_grad": "slug_state", "scorecard": "native", "ipeds": "native"},
  "usnews_grad": {"top-law-schools": {"rank_display": "1", "rank_numeric": 1, "tuition_ft": 79707,
       "enrolled": 565, "test_score": 173, "program_url": "..."}},
  "scorecard": {"student_size": 7554, "admission_rate": 0.0361, "avg_net_price": 13807,
       "completion_rate": 0.9192, "earnings_10y_median": 124080, "pell_grant_rate": 0.1916},
  "ipeds": {
    "directory": {"control": "private", "locale": 11, "...": "..."},
    "tuition": {"tuition_in_state": 61731, "total_in_state": 62484, "...": "..."},
    "graduation": {"derived": {"graduation_rate_150pct": 0.9282},
                   "by_key": {"1": {"label": "cohort_total", "total": 1698}, "3": {"label": "completers_within_150pct_normal_time", "total": 1578}}},
    "salaries": {"primary": {"label": "all_ranks", "headcount_total": 1554, "avg_salary_9m_total": 217546},
                 "by_key": {"1": {"label": "professor", "...": "..."}}},
    "enrollment": {"enrollment_total": 18446, "enrollment_white": 4583, "...": "..."}
  }
}

outputMode: normalized — one record per source record (same envelope as above per source, with raw_record, source_url/source_file, record_hash, fetched_at). raw keeps only the identity envelope + raw_record. summary pushes no records and only writes run_summary.

Envelope fields (all sources)

source, source_component, ipeds_unitid, school_name, city, state, control, raw_record (complete source payload), record_hash (sha256 of raw), fetched_at. US News adds match_method; IPEDS adds table, school_year.

run_summary (key-value store, key run_summary)

status (success / partial / failed), record_count, failed_components, truncated, truncation_by_component, components (per entity-prefixed component: status, record_count, truncated, warnings, error), config_echo (input echoed, api key masked).

Notes & limits

• US News: ranking year is the live API edition; rank_display may be Tier 1 etc. (rank_numeric null).
• IPEDS: 2023-24 release files. graduation is the IPEDS 150%-normal-time survey; graduation_rate_150pct is derived (completers / adjusted cohort) and labeled as derived.
• Scorecard ownership filter maps public=1, private=2..3.
• A component failing (429/4xx) does not fail the run; see run_summary.failed_components.
• Invalid input fails the run with the offending field name in the error.

Pricing

Pay per event: $0.003 per dataset item ($3 per 1,000). First 20 records free per run where the platform pricing tiers allow; start event free.