# edX Course Catalog Scraper (`scrapyx/edx-courses-scraper`) Actor

Scrapes edX's online course catalog by subject. Returns name, description, skills taxonomy, provider/university, upgrade price, rating, historical enrollment, and every past and upcoming run with instructors and schedule dates.

- **URL**: https://apify.com/scrapyx/edx-courses-scraper.md
- **Developed by:** [Ibnu Adzim](https://apify.com/scrapyx) (community)
- **Categories:** Education
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.40 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## edX Course Catalog Scraper

Scrapes [edX](https://www.edx.org)'s online course catalog by **subject** —
computer science, business, artificial intelligence, medicine, and 521 more.

Returns name, description, skills taxonomy, provider/university, upgrade
price, rating, historical enrollment count, and every past and upcoming run
with instructors and schedule dates.

***

### What you get

One row per course, plus a `SUBJECT_SUMMARY` row per subject queried.

#### From the sitemap (always)

`courseSlug` · `courseUrl` · `subjectQueried` · `rank`

#### From the course page (`includeCourseDetails`, on by default)

`name` · `courseCode` · `description` · `providers` · `isAccessibleForFree` ·
`paidPrice` / `paidPriceCurrency` (the verified-track upgrade price, when one
exists) · `ratingValue` / `ratingCount` · `totalHistoricalEnrollment` ·
`educationalLevel` · `timeRequired` (ISO 8601, e.g. `"P4W"` = 4 weeks) ·
`skills` (each tagged against the Lightcast skills taxonomy) ·
`courseMode` / `startDate` / `endDate` / `instructors` / `instanceCount` (from
the most recent run, when a course has run more than once) — plus the
complete raw JSON-LD in `courseJsonLd`.

***

### Input

```jsonc
{
  "subjects": ["artificial-intelligence", "computer-science"],
  "maxItems": 50,
  "includeCourseDetails": true
}
```

#### Why "subject" instead of a search box

edX's course search does not exist as something this actor can call over
HTTP. Three routes were traced:

1. **`courses.edx.org`'s public API** — real and unauthenticated, but its
   `search_term` parameter is **completely inert**: `"python"`, `"history"`
   and a nonsense string all return the exact same 37,754-course total and
   the exact same course-id list. The data itself is also raw LMS platform
   noise (test courses, instructor sandboxes), not the marketing catalog.
2. **The `/search` page** — its server-rendered content embeds the same four
   "featured" courses no matter what you search for (verified by comparing
   the actual course ids across three different queries — byte-identical
   every time). Real search runs entirely client-side against Algolia after
   the page loads, and no API credentials were findable in the site's own
   JavaScript.
3. **`discovery.edx.org`**, the real catalog search backend — requires
   authentication.

Instead, this actor reads the **real subject taxonomy directly off edX's own
sitemap**: every course page lives at `/learn/{subject}/{provider-slug}`, and
grouping 5,298 such URLs by their first path segment yields exactly the 525
subject categories edX itself uses. `subjects` is matched against that live
list each run — type `"machine-learning"` or `"business-management"`, not an
id.

An unrecognised subject is **refused with suggestions**, since there's no
search fallback here to catch a typo silently.

***

### Known limits

**No free-text search.** See above — this is a structural limit of the
target, not something this actor chose to skip.

**`totalHistoricalEnrollment` is cumulative, not current.** It's the running
total across every time the course has been offered, not a snapshot of active
students.

**Some courses have run more than once.** `instanceCount` tells you how many
runs exist; the flat `startDate`/`endDate`/`instructors` fields reflect the
most recently-starting run (not necessarily the first or last entry in the
array — verified the array order is not reliably chronological). The full
list of every run is in `courseJsonLd.hasCourseInstance`.

**Not every course has a rating.** `ratingValue`/`ratingCount` are `null`
when a course hasn't collected enough reviews yet — a genuine absence, not a
parse failure.

***

### Anti-bot posture

**None observed.** 6/6 TLS profiles clean on the sitemap and every course
detail page, across the whole recon session.

### Policy

`robots.txt` disallows `/es/learn/*` (the Spanish site) and several
account/preview/auth paths — none of which this actor touches. The English
`/learn/*` catalog is unrestricted.

# Actor input Schema

## `subjects` (type: `array`):

Subject names to crawl — 'computer-science', 'business-management', 'artificial-intelligence', 'data-analysis' and 525 more. Resolved live each run against edX's own sitemap, which encodes the real subject taxonomy in its course URL structure (/learn/{subject}/{course-slug}) — this is read directly off the site, not a fixed list that can go stale. Leave empty to crawl the single largest subject as a demo. An unrecognised subject is REFUSED with suggestions rather than silently returning zero rows, since this target has no keyword search to fall back on.

## `maxItems` (type: `integer`):

Stop after this many courses per subject. Set to 0 for unlimited — the sitemap enumeration is exhaustive (525 subjects, 5,298 course URLs total; computer-science alone has 350). Size this to what you actually need.

## `includeCourseDetails` (type: `boolean`):

Fetch each course's detail page: name, description, skills taxonomy, provider/university, education level, language, every past and upcoming run with instructor names and schedule dates, and whether it's free to audit. Turn this off to get just the URL list from the sitemap (fast, but nearly no data — the sitemap carries no titles or metadata of its own).

## `maxConcurrency` (type: `integer`):

How many requests may be in flight at once. This absorbs latency variance; it is NOT the throttle — the crawl rate is set by 'Minimum interval between requests' below.

## `minRequestInterval` (type: `number`):

The actual speed control: the shortest gap between two requests to edX, across all workers. No bot mitigation was observed, which is worth preserving.

## `proxyConfiguration` (type: `object`):

No bot mitigation was observed on the sitemap or course detail pages (6/6 TLS profiles clean). Residential is still the cloud default, since a target being clean during recon is not a guarantee.

## Actor input object example

```json
{
  "subjects": [
    "artificial-intelligence"
  ],
  "maxItems": 50,
  "includeCourseDetails": true,
  "maxConcurrency": 6,
  "minRequestInterval": 0.4,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL"
    ]
  }
}
```

# Actor output Schema

## `items` (type: `string`):

One row per scraped record. See the dataset's default view for field definitions.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "subjects": [
        "artificial-intelligence"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("scrapyx/edx-courses-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "subjects": ["artificial-intelligence"] }

# Run the Actor and wait for it to finish
run = client.actor("scrapyx/edx-courses-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "subjects": [
    "artificial-intelligence"
  ]
}' |
apify call scrapyx/edx-courses-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,scrapyx/edx-courses-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/uCrWwlzGljuf96eKS/builds/Y7JGYVQJNYceybTam/openapi.json