# Edx Allcourse Details Spider (`getdataforme/edx-allcourse-details-spider`) Actor The Edx Allcourse Details Spider is an Apify Actor that scrapes comprehensive details on all edX courses, including titles, descriptions, partners, subjects, levels, and skills.... - **URL**: https://apify.com/getdataforme/edx-allcourse-details-spider.md - **Developed by:** [GetDataForMe](https://apify.com/getdataforme) (community) - **Categories:** Automation, AI, Other - **Stats:** 9 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $9.00 / 1,000 results This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## Edx Allcourse Details Spider ### Introduction The Edx Allcourse Details Spider is a powerful Apify Actor designed to scrape comprehensive details of all courses available on the edX platform. It extracts rich metadata including titles, descriptions, partners, subjects, levels, skills, and more, enabling users to gather extensive educational content data efficiently. This Actor is ideal for researchers, educators, and businesses seeking to analyze or aggregate course information from one of the world's leading online learning platforms. ### Features - **Comprehensive Data Extraction**: Scrapes detailed information for every course on edX, including titles, descriptions, partners, subjects, levels, languages, and skills. - **High Reliability**: Built with robust error handling to ensure consistent data retrieval even from dynamic web pages. - **Fast and Scalable**: Utilizes efficient scraping techniques to process large volumes of data quickly, suitable for batch operations. - **Structured Output**: Delivers clean, JSON-formatted data that's easy to integrate into databases, analytics tools, or downstream applications. - **No Input Required**: Simple to run with no configuration needed, making it accessible for users of all technical levels. - **Real-Time Updates**: Captures the latest course offerings and metadata directly from edX's live site. - **Apify Integration**: Seamlessly runs on the Apify platform with built-in monitoring, scheduling, and export options. ### Input Parameters This Actor does not require any input parameters. It automatically scrapes all available course details from edX without user configuration. | Parameter | Type | Required | Description | Example | |-----------|------|----------|-------------|---------| | (None) | N/A | N/A | No input parameters are needed. | N/A | ### Example Usage To run the Actor, provide an empty JSON object as input (since no parameters are required): ```json {} ``` Example output (truncated for brevity): ```json [ { "title": "Google Cloud Computing Foundations: Cloud Computing Fundamentals", "partner": [ "Google Cloud" ], "product_source": "edX", "primary_description": "The first course in Google Cloud Computing Foundations Professional Certificate provides an overview of cloud computing, ways to use Google Cloud, and different compute options.", "secondary_description": "Discuss what the cloud is and why it\u2019s a technological and business game changer.\n\n\nDescribe the different ways a user can interact with Google Cloud.\n\n\nDiscover the different compute options in Google Cloud.", "tertiary_description": "Cloud Computing is revolutionizing today\u2019s world. It affects the way we communicate, do business, interact with everyday things and one another. This course discusses the fundamental theoretical and practical applications of Cloud Computing. It provides the basic concepts to understand why cloud computing is and will be such an important part of future jobs and businesses. It focuses on Google Cloud\u2019s extended solutions and compute options.", "tags": [], "availability": [ "Available now" ], "subject": [ "Computer Science" ], "level": [ "Introductory" ], "language": [ "English" ], "product": "Course", "program_type": [ "Professional Certificate", "Professional Certificate" ], "staff": [ "google-cloud" ], "subscription_eligible": null, "subscription_prices": [], "learning_type": [ "Course", "Professional Certificate", "Professional Certificate" ], "marketing_url": "https://www.edx.org/learn/cloud-computing/google-cloud-google-cloud-computing-foundations-cloud-computing-fundamentals", "card_image_url": "https://prod-discovery.edx-cdn.org/media/course/image/fcd082c5-7eb6-4b35-909f-20b5ac723d06-287bb6bb519f.png", "uuid": "fcd082c5-7eb6-4b35-909f-20b5ac723d06", "weeks_to_complete": 1, "max_effort": 6, "min_effort": 5, "skills": [ { "skill": "Cloud Computing", "category": "Information Technology", "subcategory": "Cloud Computing" }, { "skill": "Google Cloud", "category": "Information Technology", "subcategory": "Cloud Solutions" } ], "objectID": "course-fcd082c5-7eb6-4b35-909f-20b5ac723d06" } ] ``` ### Use Cases - **Market Research**: Analyze trends in online education by collecting data on course offerings, partners, and subjects. - **Competitive Intelligence**: Monitor competitors' course catalogs and pricing strategies on edX. - **Content Aggregation**: Build databases of educational resources for platforms or apps. - **Academic Research**: Study patterns in course availability, levels, and skills across disciplines. - **Business Automation**: Automate data feeds for dashboards tracking educational market insights. - **Curriculum Planning**: Identify relevant courses for institutions or individuals based on subjects and levels. ### Installation and Usage 1. Search for "Edx Allcourse Details Spider" in the Apify Store 2. Click "Try for free" or "Run" 3. Configure input parameters (none required) 4. Click "Start" to begin extraction 5. Monitor progress in the log 6. Export results in your preferred format (JSON, CSV, Excel) ### Output Format The output is a JSON array of objects, each representing a course. Key fields include: - `title`: The course name. - `partner`: Array of affiliated organizations. - `primary_description`, `secondary_description`, `tertiary_description`: Detailed course overviews. - `subject`, `level`, `language`: Categorization details. - `skills`: Array of skill objects with names and categories. - `marketing_url`, `card_image_url`: Links to course page and image. - `weeks_to_complete`, `max_effort`, `min_effort`: Time and effort estimates. - `uuid`, `objectID`: Unique identifiers. This structured format ensures easy parsing and integration. ### Support For custom/simplified outputs or bug reports, please contact: - Email: support@getdataforme.com - Subject line: "custom support" - Contact form: https://getdataforme.com/contact/ We're here to help you get the most out of this Actor! # Actor input Schema ## `itemLimit` (type: `integer`): Specifies the maximum number of items to process. ## Actor input object example ```json { "itemLimit": 50 } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "itemLimit": 50 }; // Run the Actor and wait for it to finish const run = await client.actor("getdataforme/edx-allcourse-details-spider").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "itemLimit": 50 } # Run the Actor and wait for it to finish run = client.actor("getdataforme/edx-allcourse-details-spider").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "itemLimit": 50 }' | apify call getdataforme/edx-allcourse-details-spider --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,getdataforme/edx-allcourse-details-spider" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/dg8km1OieXBJjVWo9/builds/Nuxt1hIGL5JZafLaz/openapi.json