🎓 University Course Catalog Scraper

An Apify Actor that extracts structured University Course data from any university or college catalog website. Provide a catalog URL and the actor returns clean, structured University Course records — including course code, title, credits, department, description, and prerequisites — across paginated catalog pages.

With browser automation, multi-strategy extraction, and residential proxy support, this actor reliably scrapes University Course listings from virtually any academic institution's website.

---

📋 Table of Contents

• Features
• How It Works
• Input
• Output
• Use Cases
• Quick Start
• Technical Stack
• Changelog
• Support

---

🔥 Features

• ✅ Multi-Strategy Extraction — 4 fallback strategies to extract University Course data from any page layout
• ✅ ** Browser Automation** — Real Chromium browser renders JavaScript-heavy catalog pages accurately
• ✅ Automatic Pagination — Follows "Next" links to collect University Course listings across multiple pages
• ✅ Deduplication — Skips duplicate University Course entries automatically
• ✅ Anti-Detection — Rotates user agents and disables automation fingerprints
• ✅ Proxy Support — Uses Apify residential proxies to bypass IP restrictions
• ✅ Anti-Blocking Delays — Random delays between page requests to mimic human browsing
• ✅ Configurable Limit — Set a maximum number of University Course records to collect
• ✅ Error Handling — Graceful error recovery with detailed logging
• ✅ Dataset Integration — Pushes all University Course data to Apify dataset in real time

---

⚙️ How It Works

The actor uses 4 progressive extraction strategies to handle different types of University Course catalog pages:

Strategy	Method	Best For
1	Course block <div> / <li> / <article> elements	Standard catalog pages with course cards
2	Single course page with <h1> title	Individual University Course detail pages
3	HTML <table> with course headers	Table-based University Course listings
4	Heading fallback (h1–h4 with course code pattern)	Simple or legacy catalog pages

Step-by-step flow:

1. Input Parsing — Read the catalog URL, limit, and proxy settings
2. Browser Launch — Start headless Chromium with anti-detection configuration
3. Page Fetch — Navigate using fallback strategies (domcontentloaded → load → commit)
4. Course Extraction — Apply the 4 strategies in order until courses are found
5. Deduplication — Skip any already-seen course code + title combinations
6. Dataset Push — Push each unique University Course record to Apify dataset
7. Pagination — Follow "Next" page links and repeat until limit is reached
8. Completion — Log total University Course records saved

---

📥 Input

Field	Type	Default	Description
url	string	Required	URL of the University Course catalog page to scrape
maxCourses	integer	100	Maximum number of University Course records to collect
waitSeconds	integer	5	Seconds to wait after page load before extracting
useApifyProxy	boolean	true	Whether to use Apify proxy
apifyProxyGroups	array	["RESIDENTIAL"]	Proxy groups to use

Example Input

{
  "url": "https://catalog.university.edu/courses",
  "maxCourses": 200,
  "waitSeconds": 5,
  "useApifyProxy": true,
  "apifyProxyGroups": ["RESIDENTIAL"]
}

---

📤 Output

Each University Course record is pushed as a separate dataset item.

Field	Type	Description
course_code	string	Course identifier (e.g., CS 101, ENG-202)
title	string	University Course title/name
credits	string	Credit hours or units
department	string	Department or school offering the course
description	string	Course description (up to 400–500 characters)
prerequisites	string	Prerequisite courses or requirements
source_url	string	Page URL where the course was found
scraped_at	string	ISO 8601 UTC timestamp

Example Output

{
  "course_code": "CS 301",
  "title": "Data Structures and Algorithms",
  "credits": "3",
  "department": "Computer Science",
  "description": "Study of fundamental data structures including arrays, linked lists, trees, and graphs. Analysis of sorting and searching algorithms.",
  "prerequisites": "CS 101, MATH 201",
  "source_url": "https://catalog.university.edu/courses/cs",
  "scraped_at": "2025-03-22T12:34:56Z"
}

---

🎯 Use Cases

• 🎓 Course Catalog Aggregation — Build a searchable database of University Course offerings
• 📊 Curriculum Research — Compare University Course structures across institutions
• 🤖 Academic Recommendation Systems — Power course recommendation engines with structured data
• 📚 EdTech Platforms — Enrich platforms with real University Course metadata
• 🔬 Higher Education Research — Analyze trends in University Course offerings by department
• 🏫 Institutional Benchmarking — Compare credit hours, prerequisites, and departments
• 📝 Accreditation Support — Collect structured University Course data for reporting

---

🚀 Quick Start

1. Open on Apify — Visit the actor page and click Try for free
2. Set Input — Paste your university catalog URL into the url field
3. Configure Limit — Set maxCourses to how many courses you need
4. Enable Proxy — Keep useApifyProxy enabled for reliable scraping
5. Run the Actor — Click Start and monitor progress in the logs
6. Download Results — Export the University Course dataset as JSON, CSV, or Excel

Sample Log Output

Starting scrape: https://catalog.university.edu/courses | limit=200
[Page 1]
  Strategy 1: 48 course block(s) found
  Total so far: 48 courses
[Page 2]
  Strategy 1: 45 course block(s) found
  Total so far: 93 courses
Done! Total courses saved: 200

---

🧰 Technical Stack

Component	Technology
Browser Automation	(Chromium)
Anti-Detection	Random user agents, disabled webdriver fingerprint
Navigation	Multi-strategy (domcontentloaded, load, commit)
Async	asyncio
Proxy	Apify Proxy (Residential)
Platform	Apify Actor (serverless, scalable)

---

📦 Changelog

v1.0.0 — Initial Release

• -based University Course catalog scraping
• 4-strategy extraction (blocks, single page, table, heading fallback)
• Automatic pagination with "Next" link detection
• Course code, title, credits, department, description, prerequisites extraction
• Deduplication by course code + title
• Configurable course limit (maxCourses)
• Configurable page wait time (waitSeconds)
• Residential proxy support
• Anti-detection user agent rotation
• Random anti-blocking delays (2–4 seconds)
• Real-time dataset push with ISO 8601 timestamp
• Graceful error handling and browser cleanup

---

🧑‍💻 Support & Feedback

• Issues & Ideas — Open a ticket on the Apify Actor issue tracker
• Documentation — Visit Apify Docs for platform guides
• Scraping Notes — Increase waitSeconds for slower university websites
• Proxy Tips — Always use residential proxies for university catalog scraping

---

⚠️ Disclaimer: This actor scrapes publicly visible data from university course catalog pages. Please ensure your usage complies with the terms of service of the target institution. Intended for research and informational purposes only.