University Course Details Scraper

Extract university course details from supplied public catalog and program URLs. The Actor turns course pages into structured records with course code, title, description, prerequisites, offering schedule, units or credits, department, and source links.

Use it to refresh curriculum datasets before each semester or trimester, audit prerequisite chains, prepare catalog exports, or feed course data into a spreadsheet, database, search index, or comparison workflow.

What does University Course Details Scraper do?

1. Accepts one or more public university catalog URLs.
2. Downloads server-rendered catalog pages with bounded retries.
3. recognizes common semantic course blocks used by university catalogs.
4. Extracts useful fields rather than returning raw HTML.
5. Follows explicit same-origin “next page” links when present.
6. Deduplicates courses across pages and supplied URLs.
7. Applies an optional keyword filter.
8. Stops at your course and page limits.
9. Saves normalized rows to the default Apify dataset.

The Actor is HTTP-first. It does not launch a browser for pages whose course content is already present in HTML, which keeps runs fast and economical.

Who is it for?

Curriculum and academic operations teams

Refresh published descriptions, terms, credits, and prerequisite text before a new intake period.

Education data teams

Build source-attributed course datasets for internal analysis, search, matching, or catalog normalization.

Researchers and analysts

Compare course offerings across departments or capture a reproducible snapshot of public catalog data.

Developers and automation teams

Schedule recurring Actor runs and send JSON or CSV results to a warehouse, spreadsheet, webhook workflow, or API consumer.

What university course data is extracted?

Field	Meaning
courseCode	Published catalog code, when identifiable
title	Published course title
description	Course description text
prerequisites	Prerequisite requirements
schedule	Term, offering, or schedule text
units	Credits, units, or contact-hour text
department	Nearest catalog section or department heading
courseUrl	Resolved course or catalog link
sourceUrl	Exact page from which the row was extracted
university	Catalog site name or source hostname
scrapedAt	UTC extraction timestamp

A field can be null when the source does not publish it. The Actor does not invent missing descriptions, schedules, prerequisites, or unit values.

Why use this Actor?

• User-supplied sources: run the same workflow across the public university catalog pages relevant to your project.
• Intake-ready refreshes: rerun the same saved task before every intake.
• Source attribution: every result retains its catalog and course URL.
• Structured output: use JSON, CSV, Excel, XML, or RSS exports provided by Apify datasets.
• Respectful crawling: sequential requests, a configurable delay, page limits, and transient-only retries are built in.
• Clear failure behavior: an unsupported or empty page does not silently become a successful empty dataset.

Getting started

1. Open the Actor input form.
2. Add public subject, department, program, or course catalog URLs.
3. Choose a sensible maxItems value.
4. Keep maxPages bounded to the pages you expect.
5. Leave the request delay at 500 ms or increase it for slower sites.
6. Optionally enter a keyword such as machine learning.
7. Click Start.
8. Open the Course details dataset view.
9. Export the data or connect an integration.

A small working input is:

{
  "startUrls": [
    { "url": "https://catalog.mit.edu/subjects/6/" }
  ],
  "maxItems": 10,
  "maxPages": 1,
  "requestDelayMillis": 500
}

Input parameters

startUrls

Required. Public HTTP or HTTPS pages containing course-detail blocks. Supply catalog pages rather than a university home page.

maxItems

Maximum unique course rows to save across all pages. The supported range is 1 to 10,000. The default is 100.

maxPages

Maximum supplied and discovered pagination pages to request. The supported range is 1 to 500. The default is 20.

keyword

Optional case-insensitive filter applied to the code, title, description, prerequisites, and department. Filtering is applied consistently to every supplied URL.

requestDelayMillis

Pause between requests, from 0 to 30,000 milliseconds. The default is 500 ms. Check the source’s guidance before lowering it.

proxyConfiguration

Optional Apify Proxy settings. Direct HTTP is the default and is sufficient for many public catalogs. A proxy may increase infrastructure cost and does not bypass access rules or authentication requirements.

Output example

This abbreviated record reflects current extraction from a public MIT catalog:

{
  "courseCode": "6.1000",
  "title": "Introduction to Programming and Computer Science",
  "description": "Develops foundational skills in programming and in computational modeling.",
  "prerequisites": "None",
  "schedule": "U (Fall, Spring)",
  "units": "4-0-8 units. REST",
  "department": "Programming & Software Engineering",
  "courseUrl": "https://catalog.mit.edu/subjects/6/",
  "sourceUrl": "https://catalog.mit.edu/subjects/6/",
  "university": "MIT Course Catalog",
  "scrapedAt": "2026-08-26T12:00:00.000Z"
}

Use the source URLs and timestamp when comparing successive runs. Published catalog wording can change without notice.

How much does it cost to extract university course details?

The Actor uses pay-per-event pricing:

• a one-time Start event for each run;
• an Item processed event for each unique course saved.

The current source configuration starts at $0.005 per run. Per-course pricing uses volume tiers and decreases as usage grows. The active Apify pricing panel is authoritative; proxy and platform usage may vary by source configuration.

At the BRONZE rate of $0.003668 per course:

• 10 courses: about $0.0417 including the start event;
• 100 courses: about $0.3718 including the start event;
• 1,000 courses: about $3.673 before higher-volume tier reductions.

You are not charged an item event for rejected, duplicate, empty, or failed records.

Refresh course catalogs before each intake

Save the input as an Apify Task and schedule it before each semester or trimester. Keep the dataset from each run, or export it to stable storage, then compare rows by university plus courseCode.

Useful comparisons include:

• changed prerequisite wording;
• courses newly offered or no longer listed;
• changed credits or units;
• changed term schedules;
• revised descriptions;
• movement between catalog sections.

This Actor produces current snapshots. It does not itself calculate diffs or send change alerts.

Prerequisite audit workflow

Supply one or more department catalog URLs and export these columns:

1. courseCode
2. title
3. prerequisites
4. department
5. sourceUrl
6. scrapedAt

Review null values against the source rather than interpreting them as “no prerequisite.” A source may omit the field or use an unsupported template.

Spreadsheet and data-pipeline exports

In the dataset view, choose Export and select CSV, Excel, JSON, XML, or RSS. For recurring pipelines, consume the dataset API after each successful run.

Recommended stable keys are the source hostname plus courseCode. If a source has no machine-readable code, use title and source URL carefully because titles can change.

API usage with cURL

curl -X POST \
  "https://api.apify.com/v2/acts/automation-lab~university-course-details-extractor/runs?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "startUrls": [{"url": "https://catalog.mit.edu/subjects/6/"}],
    "maxItems": 20,
    "maxPages": 1
  }'

To wait and receive dataset items directly:

curl -X POST \
  "https://api.apify.com/v2/acts/automation-lab~university-course-details-extractor/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://catalog.mit.edu/subjects/18/"}],"maxItems":10}'

API usage with JavaScript

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/university-course-details-extractor').call({
  startUrls: [{ url: 'https://catalog.mit.edu/subjects/6/' }],
  maxItems: 20,
  maxPages: 1,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

API usage with Python

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ['APIFY_TOKEN'])
run = client.actor('automation-lab/university-course-details-extractor').call(
    run_input={
        'startUrls': [{'url': 'https://catalog.mit.edu/subjects/6/'}],
        'maxItems': 20,
        'maxPages': 1,
    }
)
items = client.dataset(run['defaultDatasetId']).list_items().items
print(items)

Use with Apify MCP

Add the Actor to Claude Code:

claude mcp add --transport http apify \
  "https://mcp.apify.com?tools=automation-lab/university-course-details-extractor"

Claude Desktop, Cursor, and VS Code setup

Desktop and editor clients can use this MCP configuration:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=automation-lab/university-course-details-extractor"
    }
  }
}

Example prompts:

• “Extract the first 50 course details from this public department catalog.”
• “Collect published prerequisites and units from these three catalog URLs.”
• “Run my saved intake-refresh input and return the dataset link.”

Scheduling and integrations

Apify Tasks preserve a reusable input. Add a schedule for the week before each intake and connect the completed run to Google Sheets, Make, Zapier, a webhook, or your own API consumer.

For change monitoring, store successive snapshots outside the run dataset and compare normalized records in your pipeline. Do not overwrite your only prior snapshot if auditability matters.

Supported pages and limitations

The Actor recognizes common server-rendered course-block markup, especially CourseLeaf-style classes such as courseblocktitle, courseblockdesc, courseblockprereq, courseblockterms, and courseblockhours.

Limitations:

• JavaScript-only catalogs may not expose records to the HTTP extractor.
• Login-only, CAPTCHA-protected, or private student systems are unsupported.
• A program overview may link to courses without embedding their details.
• Custom layouts may require a future template adapter.
• Pagination is followed only through explicit same-origin next-page links.
• The Actor does not crawl an entire university domain.
• PDF catalogs are not parsed.
• It does not infer prerequisites or schedules from unrelated prose.

Legality and responsible use

Only collect public data that you are authorized to process. Review the university’s terms, robots guidance, licensing notices, and applicable law. Use conservative limits and request delays. Do not use the Actor to access private student records, authenticated portals, personal data, or systems that prohibit automated access.

Catalog facts can be copyrighted or licensed. Source attribution does not by itself grant redistribution rights.

Troubleshooting

The run says no course records were found

Open the supplied URL in a fresh browser session and confirm that complete course details appear on that exact page. A university home page, search form, PDF, or JavaScript shell is not a supported catalog page.

One field is null

The source may omit that field or express it outside the recognized course block. Check sourceUrl before treating null as a factual “none.”

A page returns HTTP 403 or 429

Increase requestDelayMillis, reduce page scope, and check the source’s access policy. If permitted, configure an Apify proxy. Repeated requests do not solve an authentication or CAPTCHA requirement.

The result limit is reached before all pages

Increase maxItems and ensure maxPages covers the intended pagination. Test a small bounded run before a large refresh.

Duplicate courses are missing

Deduplication uses source hostname plus course code, or title when no code is found. Cross-listed codes published as different codes remain separate rows.

Frequently asked questions

Can I scrape any university website?

No universal parser can guarantee every custom catalog. The Actor supports public HTML pages with recognizable course-detail blocks and fails clearly on unsupported inputs.

Does it include faculty or admission requirements?

No. The product focuses on course details: code, title, description, prerequisites, schedule, units, department, and source context.

Does it compare two intakes automatically?

Not yet. Schedule repeated snapshots and compare them in a spreadsheet, database, or workflow tool.

Can I filter a catalog?

Yes. keyword checks code, title, description, prerequisites, and department.

Are CSV and Excel supported?

Yes. Apify datasets provide CSV, Excel, JSON, XML, and other export formats.

Will a failed page be charged as a course?

No item event is emitted for a failed, duplicate, rejected, or empty record.

Related Automation Lab Actors

For website-level extraction tasks that are not course catalogs, see the Flexible HTTP Request Runner. It returns HTTP response data rather than normalized university course records.

For recurring website quality checks alongside catalog operations, see Lighthouse Website Audit.

Data quality checklist

Before operational use:

• run a small sample;
• compare several records with the source;
• confirm null-field meaning;
• confirm expected departments are represented;
• retain source URLs and timestamps;
• test the exact scheduled input;
• keep historical snapshots when changes matter;
• review source permissions and crawl guidance.

Changelog

See the Actor’s Changelog tab for customer-visible release notes.