edX Course Scraper

Scrapes public edX course catalog pages and individual course pages. It returns one normalized dataset record per course with the title, URL, description, provider, institution, subject, level, duration, effort, language, course type, image, and source URL.

Inputs

• topic: edX topic slug such as python, data-science, or artificial-intelligence.
• searchQuery: keyword used to derive a public edX topic path when no startUrls are supplied.
• startUrls: public edx.org/learn/... topic or course URLs. Course URLs are scraped directly; topic URLs are used to discover courses.
• maxResults: 1-200, default 25.
• includeDetails: fetch detail pages for enriched fields, default true.
• maxConcurrency: detail request concurrency, default 3.

Example:

{ "topic": "python", "maxResults": 10, "includeDetails": true }

The actor uses public HTML only. It does not log in, bypass challenges, scrape course content behind authentication, or defeat CAPTCHAs. A blocked route is reported in OUTPUT with zero fabricated records. Run diagnostics are saved under the OUTPUT key-value record. The default dataset item is the billable value unit.

What data does edX Course Scraper return?

Field	Type	Description
recordType	string	Value returned as recordType.
courseId	string or null	Value returned as courseId.
title	string	Value returned as title.
url	string	Value returned as url.
description	string or null	Value returned as description.
provider	string or null	Value returned as provider.
institution	string or null	Value returned as institution.
subject	string or null	Value returned as subject.
level	string or null	Value returned as level.
duration	string or null	Value returned as duration.
effort	string or null	Value returned as effort.
language	string or null	Value returned as language.
courseType	string or null	Value returned as courseType.
image	string or null	Value returned as image.
scrapedAt	string	Value returned as scrapedAt.
sourceUrl	string	Value returned as sourceUrl.

Use cases

• Build public prospect lists and qualify organizations or professionals before responsible outreach.
• Schedule repeatable collection and export results to downstream workflows.
• Run a one-off research job and export the structured result as JSON, CSV, Excel, XML, or RSS from Apify.
• Schedule the same input to monitor changes over time and send completed datasets to a webhook or integration.
• Feed schema-shaped records into a database, spreadsheet, BI tool, or AI workflow with the source URL retained for verification.

Output example

{
  "recordType": "Example recordType",
  "courseId": "Example courseId",
  "title": "Example title",
  "url": "Example url",
  "description": "Example description",
  "provider": "Example provider",
  "institution": "Example institution",
  "subject": "Example subject",
  "level": "Example level",
  "duration": "Example duration",
  "effort": "Example effort",
  "language": "Example language"
}

The exact fields depend on the selected input and what the public source exposes. Use the dataset schema as the machine-readable contract and retain source URLs for verification.

Run edX Course Scraper with the Apify API

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('muhammadafzal/edx-course-scraper').call({
  "topic": "python",
  "searchQuery": "",
  "startUrls": [],
  "maxResults": 25,
  "includeDetails": true,
  "maxConcurrency": 3
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

You can also run the Actor from Apify Console, schedules, webhooks, the REST API, Make, Zapier, n8n, or the hosted Apify MCP server.

Pricing and cost control

This Actor uses pay-per-event pricing. Charges follow the live event definitions shown below.

Event	Price (USD)	When it is charged
apify-default-dataset-item	$0.005	One normalized public edX course record delivered to the default dataset.
apify-actor-start	$0.00005	Charged when the Actor starts running. Number of events charged depends on Actor memory (one event per GB, minimum one event).

For example, 100 apify-default-dataset-item events cost $0.5, plus any enabled Actor-start event. Empty or failed work should be checked in the run log and dataset before reuse.

Responsible use

Use this Actor only for data you are authorized to access. Follow the target website's terms, robots and access policies, and applicable privacy, database, copyright, anti-spam, and data-protection laws. Do not use it to bypass authentication or other access controls, collect private data, harass people, or make high-impact decisions without independent verification.

Support

When reporting a problem, include the Actor run ID, a redacted input, the expected result, and a small public example URL when applicable. Do not post API tokens, cookies, credentials, or personal data in an issue.