Edx Allcourse Details Spider

Edx Allcourse Details Spider

Introduction

The Edx Allcourse Details Spider is an Apify Actor that scrapes comprehensive course details from edX.org. It extracts information like course title, partner, description, skills, and more, delivering structured JSON output. Ideal for educational research, course cataloging, or market analysis, this actor processes edX course data efficiently.

Input Parameters

The actor accepts the following input parameters:

Parameter	Type	Description	Required
Urls	Array	List of edX course page URLs to scrape data from (e.g., https://www.edx.org/learn/sql/harvard-university-cs50-s-introduction-to-databases-with-sql). If empty, it scrapes all available courses.	No
proxyConfiguration	Object	Proxy settings for the actor. Optional, but recommended to use {"useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"]} for reliable scraping.	No

Example Input

{
    "proxyConfiguration": {
        "useApifyProxy": true,
        "apifyProxyGroups": ["RESIDENTIAL"]
    }
}

Features

• Detailed Course Data: Extracts course title, partner, descriptions, subjects, skills, staff, and more.
• Flexible Input: Scrapes all courses if no URLs provided or processes specific course URLs.
• Structured JSON Output: Delivers clean, organized data for easy integration into analysis tools.
• Proxy Support: Integrates with Apify’s proxy system for reliable and anonymous scraping.
• Robust Error Handling: Manages page errors and timeouts for consistent performance.

Output

The actor outputs a JSON array containing course details. Below is an example output:

[
    {
        "title": "CS50's Introduction to Databases with SQL",
        "partner": ["Harvard University"],
        "product_source": "edX",
        "primary_description": "An introduction to databases using a language called SQL.",
        "secondary_description": "CREATE TABLE\nSELECT\nINSERT\nUPDATE\nDELETE\nDROP\nCREATE INDEX\nCREATE VIEW\nSQLite, PostgreSQL, MySQL\nconnectors (Python, Java)\nreplication",
        "tertiary_description": "This is CS50's introduction to databases using a language called SQL. Learn how to create, read, update, and delete data with relational databases, which store data in rows and columns. Learn how to model real-world entities and relationships among them using tables with appropriate types, triggers, and constraints. Learn how to normalize data to eliminate redundancies and reduce potential for errors. Learn how to join tables together using primary and foreign keys. Learn how to automate searches with views and expedite searches with indexes. Learn how to connect SQL with other languages like Python and Java. Course begins with SQLite for portability's sake and ends with introductions to PostgreSQL and MySQL for scalability's sake as well. Assignments inspired by real-world datasets.",
        "tags": [],
        "availability": ["Available now"],
        "subject": ["Computer Science", "Data Analysis & Statistics", "Business & Management"],
        "level": ["Introductory"],
        "language": ["English"],
        "product": "Course",
        "program_type": ["Professional Certificate"],
        "staff": ["david-j-malan", "carter-zenke"],
        "subscription_eligible": null,
        "subscription_prices": [],
        "learning_type": ["Course", "Professional Certificate"],
        "marketing_url": "https://www.edx.org/learn/sql/harvard-university-cs50-s-introduction-to-databases-with-sql",
        "card_image_url": "https://prod-discovery.edx-cdn.org/media/course/image/3e45c431-10df-423e-9f03-fb98b713cd4a-2397c9098976.png",
        "uuid": "3e45c431-10df-423e-9f03-fb98b713cd4a",
        "weeks_to_complete": 7,
        "max_effort": 12,
        "min_effort": 6,
        "skills": [
            {"skill": "Relational Databases", "category": "Information Technology", "subcategory": "Databases"},
            {"skill": "MySQL", "category": "Information Technology", "subcategory": "Databases"},
            {"skill": "PostgreSQL", "category": "Information Technology", "subcategory": "Databases"},
            {"skill": "Python (Programming Language)", "category": "Information Technology", "subcategory": "Scripting Languages"},
            {"skill": "SQLite", "category": "Information Technology", "subcategory": "Databases"},
            {"skill": "Java (Programming Language)", "category": "Information Technology", "subcategory": "Java"},
            {"skill": "Foreign Keys", "category": "Information Technology", "subcategory": "Databases"},
            {"skill": "SQL (Programming Language)", "category": "Information Technology", "subcategory": "Query Languages"},
            {"skill": "Data Store", "category": "Information Technology", "subcategory": "Data Storage"},
            {"skill": "Scalability", "category": "Information Technology", "subcategory": "Software Development"}
        ],
        "objectID": "course-3e45c431-10df-423e-9f03-fb98b713cd4a"
    }
]

Use Cases

• Educational Research: Analyze course offerings to study trends in online education.
• Course Cataloging: Build comprehensive databases of edX courses for educational platforms.
• Market Analysis: Compare course subjects, skills, and partners to identify market demands.
• Program Development: Use course details to inform the creation of new educational programs.

Support

For bug reports or custom needs, contact support@getdataforme.com or use our contact form: https://getdataforme.com/contact/.
Please include a clear subject line in your message, such as "Edx Allcourse Details Spider Support."