edX Course Catalog Scraper

Scrapes edX's online course catalog by subject — computer science, business, artificial intelligence, medicine, and 521 more.

Returns name, description, skills taxonomy, provider/university, upgrade price, rating, historical enrollment count, and every past and upcoming run with instructors and schedule dates.

---

What you get

One row per course, plus a SUBJECT_SUMMARY row per subject queried.

From the sitemap (always)

courseSlug · courseUrl · subjectQueried · rank

From the course page (includeCourseDetails, on by default)

name · courseCode · description · providers · isAccessibleForFree · paidPrice / paidPriceCurrency (the verified-track upgrade price, when one exists) · ratingValue / ratingCount · totalHistoricalEnrollment · educationalLevel · timeRequired (ISO 8601, e.g. "P4W" = 4 weeks) · skills (each tagged against the Lightcast skills taxonomy) · courseMode / startDate / endDate / instructors / instanceCount (from the most recent run, when a course has run more than once) — plus the complete raw JSON-LD in courseJsonLd.

---

Input

{
  "subjects": ["artificial-intelligence", "computer-science"],
  "maxItems": 50,
  "includeCourseDetails": true
}

Why "subject" instead of a search box

edX's course search does not exist as something this actor can call over HTTP. Three routes were traced:

1. courses.edx.org's public API — real and unauthenticated, but its search_term parameter is completely inert: "python", "history" and a nonsense string all return the exact same 37,754-course total and the exact same course-id list. The data itself is also raw LMS platform noise (test courses, instructor sandboxes), not the marketing catalog.
2. The /search page — its server-rendered content embeds the same four "featured" courses no matter what you search for (verified by comparing the actual course ids across three different queries — byte-identical every time). Real search runs entirely client-side against Algolia after the page loads, and no API credentials were findable in the site's own JavaScript.
3. discovery.edx.org, the real catalog search backend — requires authentication.

Instead, this actor reads the real subject taxonomy directly off edX's own sitemap: every course page lives at /learn/{subject}/{provider-slug}, and grouping 5,298 such URLs by their first path segment yields exactly the 525 subject categories edX itself uses. subjects is matched against that live list each run — type "machine-learning" or "business-management", not an id.

An unrecognised subject is refused with suggestions, since there's no search fallback here to catch a typo silently.

---

Known limits

No free-text search. See above — this is a structural limit of the target, not something this actor chose to skip.

totalHistoricalEnrollment is cumulative, not current. It's the running total across every time the course has been offered, not a snapshot of active students.

Some courses have run more than once. instanceCount tells you how many runs exist; the flat startDate/endDate/instructors fields reflect the most recently-starting run (not necessarily the first or last entry in the array — verified the array order is not reliably chronological). The full list of every run is in courseJsonLd.hasCourseInstance.

Not every course has a rating. ratingValue/ratingCount are null when a course hasn't collected enough reviews yet — a genuine absence, not a parse failure.

---

Anti-bot posture

None observed. 6/6 TLS profiles clean on the sitemap and every course detail page, across the whole recon session.

Policy

robots.txt disallows /es/learn/* (the Spanish site) and several account/preview/auth paths — none of which this actor touches. The English /learn/* catalog is unrestricted.