University Course Catalog Scraper

Scrapes complete course catalogs from US universities in a single run — every course with its code, title, credit hours, full description and prerequisites — with a filter to pick which universities you want.

These catalogs share a common underlying publishing system, which is what lets one scraper cover many of them. None of them offer a bulk download; the data exists only as HTML, one subject page at a time.

What you get

One row per course, with institution and institutionName on every row so a multi-university dataset stays sortable.

Field	Example
institution, institutionName	mit, Massachusetts Institute of Technology
courseKey	6.1000
subject, catalogNumber	6, 1000
title	Introduction to Programming and Computer Science
credits	4-0-8 units
description, descriptionWordCount	full catalog text, 64
prerequisites	MATH 151 with a grade of C or better
sourceUrl	link back to the subject page

Input

{
  "institutions": ["mit", "northeastern"],
  "subjects": ["CS"],
  "maxItemsPerInstitution": 500
}

Field	Default	Notes
institutions	all	keys from the Universities picker in the input; an unknown key fails immediately
subjects	all	codes differ between institutions
maxItemsPerInstitution	0 (no limit)	cap per university
requestDelayMs	600	politeness delay

Leaving institutions empty scrapes every available catalog, which is a multi-hour run. Pick a handful, or set maxItemsPerInstitution for a trial.

Uses

• Cross-institution curriculum comparison — the reason this exists
• Prerequisite-chain analysis across universities
• Transfer-credit and articulation mapping between institutions
• Program benchmarking — what peers teach that you do not
• Catalog change tracking between academic years

Reliability

Each institution is harvested independently, so one failing cannot lose the others — a failure is logged and recorded in the run summary rather than aborting the run. Individual subject pages that fail are skipped with a warning. 429 and 5xx get bounded exponential backoff. Courses are de-duplicated per institution on courseKey.

A per-institution breakdown is written to the key-value store under RUN_SUMMARY, including any institution that returned zero courses — so a layout this parser has not seen shows up as a fact in the summary rather than as a quietly short dataset.

A note on coverage

Every catalog offered in the Universities picker was fetched and parsed before being listed — an institution whose pages came back empty was dropped rather than shipped as a name that quietly returns nothing. Institutions not in the picker may still publish their catalog the same way.

The course markup is broadly shared across these catalogs, but the index path is not — it differs per institution, and there are several distinct title layouts and course-code conventions in the wild (CS 101, ZACM-1000, ANTH101, NYU's ACCT-GB 2103, MIT's 6.1000). This actor handles all of them. Some institutions split undergraduate and graduate catalogs across two indexes, and both are crawled.

---

Not affiliated with or endorsed by any institution listed. All data is public information published by each university.