# University Course Details Scraper (`automation-lab/university-course-details-extractor`) Actor

Extract course codes, titles, descriptions, schedules, units, and prerequisites from supplied public university catalog and program URLs.

- **URL**: https://apify.com/automation-lab/university-course-details-extractor.md
- **Developed by:** [Stas Persiianenko](https://apify.com/automation-lab) (community)
- **Categories:** Education, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.20 / 1,000 item extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## University Course Details Scraper

Extract **university course details** from supplied public catalog and program URLs.
The Actor turns course pages into structured records with course code, title,
description, prerequisites, offering schedule, units or credits, department,
and source links.

Use it to refresh curriculum datasets before each semester or trimester, audit
prerequisite chains, prepare catalog exports, or feed course data into a
spreadsheet, database, search index, or comparison workflow.

### What does University Course Details Scraper do?

1. Accepts one or more public university catalog URLs.
2. Downloads server-rendered catalog pages with bounded retries.
3. recognizes common semantic course blocks used by university catalogs.
4. Extracts useful fields rather than returning raw HTML.
5. Follows explicit same-origin “next page” links when present.
6. Deduplicates courses across pages and supplied URLs.
7. Applies an optional keyword filter.
8. Stops at your course and page limits.
9. Saves normalized rows to the default Apify dataset.

The Actor is HTTP-first. It does not launch a browser for pages whose course
content is already present in HTML, which keeps runs fast and economical.

### Who is it for?

#### Curriculum and academic operations teams

Refresh published descriptions, terms, credits, and prerequisite text before a
new intake period.

#### Education data teams

Build source-attributed course datasets for internal analysis, search, matching,
or catalog normalization.

#### Researchers and analysts

Compare course offerings across departments or capture a reproducible snapshot
of public catalog data.

#### Developers and automation teams

Schedule recurring Actor runs and send JSON or CSV results to a warehouse,
spreadsheet, webhook workflow, or API consumer.

### What university course data is extracted?

| Field | Meaning |
| --- | --- |
| `courseCode` | Published catalog code, when identifiable |
| `title` | Published course title |
| `description` | Course description text |
| `prerequisites` | Prerequisite requirements |
| `schedule` | Term, offering, or schedule text |
| `units` | Credits, units, or contact-hour text |
| `department` | Nearest catalog section or department heading |
| `courseUrl` | Resolved course or catalog link |
| `sourceUrl` | Exact page from which the row was extracted |
| `university` | Catalog site name or source hostname |
| `scrapedAt` | UTC extraction timestamp |

A field can be `null` when the source does not publish it. The Actor does not
invent missing descriptions, schedules, prerequisites, or unit values.

### Why use this Actor?

- **User-supplied sources:** run the same workflow across the public university
  catalog pages relevant to your project.
- **Intake-ready refreshes:** rerun the same saved task before every intake.
- **Source attribution:** every result retains its catalog and course URL.
- **Structured output:** use JSON, CSV, Excel, XML, or RSS exports provided by
  Apify datasets.
- **Respectful crawling:** sequential requests, a configurable delay, page
  limits, and transient-only retries are built in.
- **Clear failure behavior:** an unsupported or empty page does not silently
  become a successful empty dataset.

### Getting started

1. Open the Actor input form.
2. Add public subject, department, program, or course catalog URLs.
3. Choose a sensible `maxItems` value.
4. Keep `maxPages` bounded to the pages you expect.
5. Leave the request delay at 500 ms or increase it for slower sites.
6. Optionally enter a keyword such as `machine learning`.
7. Click **Start**.
8. Open the **Course details** dataset view.
9. Export the data or connect an integration.

A small working input is:

```json
{
  "startUrls": [
    { "url": "https://catalog.mit.edu/subjects/6/" }
  ],
  "maxItems": 10,
  "maxPages": 1,
  "requestDelayMillis": 500
}
```

### Input parameters

#### `startUrls`

Required. Public HTTP or HTTPS pages containing course-detail blocks. Supply
catalog pages rather than a university home page.

#### `maxItems`

Maximum unique course rows to save across all pages. The supported range is 1
to 10,000. The default is 100.

#### `maxPages`

Maximum supplied and discovered pagination pages to request. The supported
range is 1 to 500. The default is 20.

#### `keyword`

Optional case-insensitive filter applied to the code, title, description,
prerequisites, and department. Filtering is applied consistently to every
supplied URL.

#### `requestDelayMillis`

Pause between requests, from 0 to 30,000 milliseconds. The default is 500 ms.
Check the source’s guidance before lowering it.

#### `proxyConfiguration`

Optional Apify Proxy settings. Direct HTTP is the default and is sufficient for
many public catalogs. A proxy may increase infrastructure cost and does not
bypass access rules or authentication requirements.

### Output example

This abbreviated record reflects current extraction from a public MIT catalog:

```json
{
  "courseCode": "6.1000",
  "title": "Introduction to Programming and Computer Science",
  "description": "Develops foundational skills in programming and in computational modeling.",
  "prerequisites": "None",
  "schedule": "U (Fall, Spring)",
  "units": "4-0-8 units. REST",
  "department": "Programming & Software Engineering",
  "courseUrl": "https://catalog.mit.edu/subjects/6/",
  "sourceUrl": "https://catalog.mit.edu/subjects/6/",
  "university": "MIT Course Catalog",
  "scrapedAt": "2026-08-26T12:00:00.000Z"
}
```

Use the source URLs and timestamp when comparing successive runs. Published
catalog wording can change without notice.

### How much does it cost to extract university course details?

The Actor uses pay-per-event pricing:

- a one-time **Start** event for each run;
- an **Item processed** event for each unique course saved.

The current source configuration starts at $0.005 per run. Per-course pricing
uses volume tiers and decreases as usage grows. The active Apify pricing panel
is authoritative; proxy and platform usage may vary by source configuration.

At the BRONZE rate of $0.003668 per course:

- 10 courses: about $0.0417 including the start event;
- 100 courses: about $0.3718 including the start event;
- 1,000 courses: about $3.673 before higher-volume tier reductions.

You are not charged an item event for rejected, duplicate, empty, or failed
records.

### Refresh course catalogs before each intake

Save the input as an Apify Task and schedule it before each semester or
trimester. Keep the dataset from each run, or export it to stable storage, then
compare rows by `university` plus `courseCode`.

Useful comparisons include:

- changed prerequisite wording;
- courses newly offered or no longer listed;
- changed credits or units;
- changed term schedules;
- revised descriptions;
- movement between catalog sections.

This Actor produces current snapshots. It does not itself calculate diffs or
send change alerts.

### Prerequisite audit workflow

Supply one or more department catalog URLs and export these columns:

1. `courseCode`
2. `title`
3. `prerequisites`
4. `department`
5. `sourceUrl`
6. `scrapedAt`

Review null values against the source rather than interpreting them as “no
prerequisite.” A source may omit the field or use an unsupported template.

### Spreadsheet and data-pipeline exports

In the dataset view, choose **Export** and select CSV, Excel, JSON, XML, or RSS.
For recurring pipelines, consume the dataset API after each successful run.

Recommended stable keys are the source hostname plus `courseCode`. If a source
has no machine-readable code, use title and source URL carefully because titles
can change.

### API usage with cURL

```bash
curl -X POST \
  "https://api.apify.com/v2/acts/automation-lab~university-course-details-extractor/runs?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "startUrls": [{"url": "https://catalog.mit.edu/subjects/6/"}],
    "maxItems": 20,
    "maxPages": 1
  }'
```

To wait and receive dataset items directly:

```bash
curl -X POST \
  "https://api.apify.com/v2/acts/automation-lab~university-course-details-extractor/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://catalog.mit.edu/subjects/18/"}],"maxItems":10}'
```

### API usage with JavaScript

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/university-course-details-extractor').call({
  startUrls: [{ url: 'https://catalog.mit.edu/subjects/6/' }],
  maxItems: 20,
  maxPages: 1,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);
```

### API usage with Python

```python
import os
from apify_client import ApifyClient

client = ApifyClient(os.environ['APIFY_TOKEN'])
run = client.actor('automation-lab/university-course-details-extractor').call(
    run_input={
        'startUrls': [{'url': 'https://catalog.mit.edu/subjects/6/'}],
        'maxItems': 20,
        'maxPages': 1,
    }
)
items = client.dataset(run['defaultDatasetId']).list_items().items
print(items)
```

### Use with Apify MCP

Add the Actor to Claude Code:

```bash
claude mcp add --transport http apify \
  "https://mcp.apify.com?tools=automation-lab/university-course-details-extractor"
```

#### Claude Desktop, Cursor, and VS Code setup

Desktop and editor clients can use this MCP configuration:

```json
{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=automation-lab/university-course-details-extractor"
    }
  }
}
```

Example prompts:

- “Extract the first 50 course details from this public department catalog.”
- “Collect published prerequisites and units from these three catalog URLs.”
- “Run my saved intake-refresh input and return the dataset link.”

### Scheduling and integrations

Apify Tasks preserve a reusable input. Add a schedule for the week before each
intake and connect the completed run to Google Sheets, Make, Zapier, a webhook,
or your own API consumer.

For change monitoring, store successive snapshots outside the run dataset and
compare normalized records in your pipeline. Do not overwrite your only prior
snapshot if auditability matters.

### Supported pages and limitations

The Actor recognizes common server-rendered course-block markup, especially
CourseLeaf-style classes such as `courseblocktitle`, `courseblockdesc`,
`courseblockprereq`, `courseblockterms`, and `courseblockhours`.

Limitations:

- JavaScript-only catalogs may not expose records to the HTTP extractor.
- Login-only, CAPTCHA-protected, or private student systems are unsupported.
- A program overview may link to courses without embedding their details.
- Custom layouts may require a future template adapter.
- Pagination is followed only through explicit same-origin next-page links.
- The Actor does not crawl an entire university domain.
- PDF catalogs are not parsed.
- It does not infer prerequisites or schedules from unrelated prose.

### Legality and responsible use

Only collect public data that you are authorized to process. Review the
university’s terms, robots guidance, licensing notices, and applicable law.
Use conservative limits and request delays. Do not use the Actor to access
private student records, authenticated portals, personal data, or systems that
prohibit automated access.

Catalog facts can be copyrighted or licensed. Source attribution does not by
itself grant redistribution rights.

### Troubleshooting

#### The run says no course records were found

Open the supplied URL in a fresh browser session and confirm that complete
course details appear on that exact page. A university home page, search form,
PDF, or JavaScript shell is not a supported catalog page.

#### One field is null

The source may omit that field or express it outside the recognized course
block. Check `sourceUrl` before treating null as a factual “none.”

#### A page returns HTTP 403 or 429

Increase `requestDelayMillis`, reduce page scope, and check the source’s access
policy. If permitted, configure an Apify proxy. Repeated requests do not solve
an authentication or CAPTCHA requirement.

#### The result limit is reached before all pages

Increase `maxItems` and ensure `maxPages` covers the intended pagination. Test a
small bounded run before a large refresh.

#### Duplicate courses are missing

Deduplication uses source hostname plus course code, or title when no code is
found. Cross-listed codes published as different codes remain separate rows.

### Frequently asked questions

#### Can I scrape any university website?

No universal parser can guarantee every custom catalog. The Actor supports
public HTML pages with recognizable course-detail blocks and fails clearly on
unsupported inputs.

#### Does it include faculty or admission requirements?

No. The product focuses on course details: code, title, description,
prerequisites, schedule, units, department, and source context.

#### Does it compare two intakes automatically?

Not yet. Schedule repeated snapshots and compare them in a spreadsheet,
database, or workflow tool.

#### Can I filter a catalog?

Yes. `keyword` checks code, title, description, prerequisites, and department.

#### Are CSV and Excel supported?

Yes. Apify datasets provide CSV, Excel, JSON, XML, and other export formats.

#### Will a failed page be charged as a course?

No item event is emitted for a failed, duplicate, rejected, or empty record.

### Related Automation Lab Actors

For website-level extraction tasks that are not course catalogs, see the
[Flexible HTTP Request Runner](https://apify.com/automation-lab/flexible-http-request-runner).
It returns HTTP response data rather than normalized university course records.

For recurring website quality checks alongside catalog operations, see
[Lighthouse Website Audit](https://apify.com/automation-lab/website-lighthouse-seo-audit).

### Data quality checklist

Before operational use:

- run a small sample;
- compare several records with the source;
- confirm null-field meaning;
- confirm expected departments are represented;
- retain source URLs and timestamps;
- test the exact scheduled input;
- keep historical snapshots when changes matter;
- review source permissions and crawl guidance.

### Changelog

See the Actor’s **Changelog** tab for customer-visible release notes.

# Actor input Schema

## `startUrls` (type: `array`):

Public pages that list course details. Add subject catalogs, departmental catalogs, or program course pages.

## `maxItems` (type: `integer`):

Stop after saving this many unique course records across all supplied URLs.

## `maxPages` (type: `integer`):

Maximum number of supplied and next-page URLs to request.

## `keyword` (type: `string`):

Optional case-insensitive filter applied to course code, title, description, prerequisites, and department.

## `requestDelayMillis` (type: `integer`):

Polite pause between catalog page requests. Keep this at 500 ms or higher unless the source explicitly permits faster crawling.

## `proxyConfiguration` (type: `object`):

Optional Apify Proxy configuration for public catalogs that restrict datacenter traffic.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://catalog.mit.edu/subjects/6/"
    }
  ],
  "maxItems": 20,
  "maxPages": 20,
  "requestDelayMillis": 500
}
```

# Actor output Schema

## `overview` (type: `string`):

Default dataset containing all extracted university course records.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://catalog.mit.edu/subjects/6/"
        }
    ],
    "maxItems": 20
};

// Run the Actor and wait for it to finish
const run = await client.actor("automation-lab/university-course-details-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://catalog.mit.edu/subjects/6/" }],
    "maxItems": 20,
}

# Run the Actor and wait for it to finish
run = client.actor("automation-lab/university-course-details-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://catalog.mit.edu/subjects/6/"
    }
  ],
  "maxItems": 20
}' |
apify call automation-lab/university-course-details-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,automation-lab/university-course-details-extractor"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/jUp094E9k8MpX2rkd/builds/5mZOhIjr4b5AMUXmd/openapi.json