# Skill Curator Scraper (`datapilot/skill-curator-scraper`) Actor MCP Skill Scraper collects AI skills from SkillsMP and GitHub. It extracts name, description, stars, license, and URLs, then calculates a quality score. Outputs structured JSON for discovering MCP tools, AI skills, and developer resources. - **URL**: https://apify.com/datapilot/skill-curator-scraper.md - **Developed by:** [Data Pilot](https://apify.com/datapilot) (community) - **Categories:** SEO tools, Jobs, Videos - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, NaN bookmarks - **User rating**: No ratings yet ## Pricing from $0.02 / 1,000 results This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Skill Curator Scraper 🎓 **Skill Curator Scraper** is a powerful Apify Actor designed to discover and curate comprehensive **Skill Curator** resources from multiple sources including SkillsMP and GitHub. This tool aggregates **Skill Curator** content, learning resources, and skill repositories for any topic or domain. Whether you're building skill inventories, curating learning resources, or discovering professional development materials, the Skill Curator Scraper delivers quality **Skill Curator** content efficiently. With multi-source aggregation, intelligent quality scoring, duplicate detection, and GitHub integration, the Skill Curator Scraper ensures comprehensive discovery of **Skill Curator** resources with reliability metrics. It focuses on key **Skill Curator** metrics including quality scores, repository stars, licensing information, and skill descriptions, making it an essential tool for **Skill Curator** research and professional development intelligence. --- ### 📋 Table of Contents - [Features](#-features) - [Sources](#-sources) - [How It Works](#-how-it-works) - [Input](#-input) - [Output](#-output) - [Quality Scoring](#-quality-scoring) - [Technical Stack](#-technical-stack) - [Data Fields](#-data-fields) - [Use Cases](#-use-cases) - [Quick Start](#-quick-start) - [Configuration](#-configuration) - [Performance](#-performance) - [Important Notes](#-important-notes) - [Keywords](#-keywords) - [Changelog](#-changelog) - [Support](#-support) --- ### 🔥 Features - **Multi-Source Skill Discovery** – Aggregates **Skill Curator** resources from SkillsMP and GitHub simultaneously using parallel requests. - **SkillsMP Scraping** – HTML scraping of SkillsMP platform for **Skill Curator** content discovery. - **GitHub Repository Search** – GitHub API integration for discovering skill-related repositories and resources. - **Quality Scoring** – Intelligent algorithm scoring skills based on stars, licensing, description quality, and availability. - **Duplicate Detection** – Automatic deduplication of skills from multiple sources. - **Star-Based Ranking** – Prioritizes popular repositories and well-maintained projects. - **License Information** – Extracts and includes SPDX license identifiers for legal compliance. - **Author Attribution** – Captures author/owner information from repositories. - **Keyword-Based Search** – Supports multiple keywords for comprehensive skill discovery. - **Bulk Keyword Processing** – Analyzes multiple skill keywords simultaneously. - **Rate Limiting** – Includes automatic delays to respect API rate limits. - **Proxy Support** – Apify residential proxy support for reliable access. - **Real-Time Dataset Push** – Pushes results to Apify Dataset with metadata. - **Timestamp Recording** – Records discovery timestamp for audit trails. - **Error Handling** – Graceful error handling with detailed logging. - **Asyncio-Friendly** – Non-blocking async/await architecture. --- ### 🌍 Sources #### **1. SkillsMP** - **Platform**: Skill marketplace and curator platform - **Search Type**: HTML scraping - **Content**: Skill cards, descriptions, skill URLs - **Data Extracted**: Name, description, skill URL - **URL Format**: `https://skillsmp.com/skills/{name}` - **Coverage**: Broad skill marketplace #### **2. GitHub Repositories** - **Platform**: GitHub version control and open source - **Search Type**: REST API (JSON) - **Content**: Repositories, code projects, implementations - **Data Extracted**: Name, description, stars, license, author, GitHub URL - **API Endpoint**: `https://api.github.com/search/repositories` - **Search Query**: Keywords + "mcp" + "skill" - **Sorting**: By stars (most popular first) --- ### ⚙️ How It Works The Skill Curator Scraper takes skill keywords as input and searches multiple sources simultaneously. It scrapes SkillsMP for skill cards and queries GitHub API for repositories. Each skill is assigned a quality score based on stars, licensing, description quality, and availability. Results are deduplicated and pushed to the Apify Dataset. **Key Processing Steps:** 1. **Input Parsing** – Accept skill keywords from Actor input 2. **Proxy Setup** – Configure Apify residential proxy if available 3. **Parallel Source Queries** – Launch SkillsMP scraping and GitHub API search 4. **SkillsMP Scraping** – HTML parse skill cards from SkillsMP 5. **GitHub API Search** – Query GitHub with keyword filters 6. **Data Extraction** – Extract name, description, stars, license, author 7. **Quality Scoring** – Calculate quality score for each skill 8. **Deduplication** – Remove duplicate entries from multiple sources 9. **Result Compilation** – Aggregate findings from all sources 10. **Dataset Push** – Push to Apify Dataset with metadata **Key Benefits:** - Discover **Skill Curator** resources from multiple trusted sources - Find popular and well-maintained skill implementations - Compare skills across SkillsMP and GitHub - Identify high-quality learning resources - Build comprehensive skill inventories - Research skill implementations and examples --- ### 📥 Input The Actor accepts the following input parameters: | Field | Type | Default | Description | |-------|------|---------|-------------| | `keywords` | array | required | Skill keywords to search (e.g., ["React", "Python", "DevOps"]) | | `limit_per_keyword` | integer | `20` | Maximum skills per keyword (1-100) | | `proxyConfiguration` | object | `{"useApifyProxy": true}` | Proxy configuration settings | **Example Input:** ```json { "keywords": ["React", "Python", "DevOps", "GraphQL", "Kubernetes"], "limit_per_keyword": 25, "proxyConfiguration": { "useApifyProxy": true } } ```` **Single Keyword Example:** ```json { "keywords": ["Machine Learning"], "limit_per_keyword": 30 } ``` *** ### 📤 Output The Actor pushes **Skill Curator** records with the following structure: | Field | Type | Description | |-------|------|-------------| | `name` | string | Skill or repository name | | `description` | string | Skill/repo description (max 300 chars) | | `author` | string | Repository owner/author name | | `repoMetadata.stars` | integer | GitHub stars count | | `repoMetadata.license` | string | SPDX license identifier | | `githubUrl` | string | Direct GitHub repository URL | | `skillUrl` | string | SkillsMP or project skill URL | | `qualityScore.overall` | integer | Quality score (0-100) | | `keyword` | string | Search keyword used | | `detected_at` | string | ISO 8601 discovery timestamp | **Example Output Record (GitHub):** ```json { "name": "react-query", "author": "tannerlinsley", "description": "Powerful asynchronous state management, server-state utilities and data fetching with TS/JS, React Query, Solid Query, Svelte Query and Vue Query.", "repoMetadata.stars": 42000, "repoMetadata.license": "MIT", "githubUrl": "https://github.com/tannerlinsley/react-query", "skillUrl": "https://skillsmp.com/skills/react-query", "qualityScore.overall": 95, "keyword": "React", "detected_at": "2025-02-14T12:00:00Z" } ``` **Example Output Record (SkillsMP):** ```json { "name": "Advanced React Patterns", "description": "Learn advanced React patterns including render props, custom hooks, and compound components for building scalable applications.", "repoMetadata.stars": 0, "githubUrl": "", "skillUrl": "https://skillsmp.com/skills/advanced-react-patterns", "qualityScore.overall": 65, "keyword": "React", "detected_at": "2025-02-14T12:00:00Z" } ``` *** ### 🎯 Quality Scoring The Skill Curator Scraper uses an intelligent quality scoring algorithm to rank skills: #### **Scoring Criteria** | Factor | Points | Threshold | |--------|--------|-----------| | GitHub Stars | 40 | ≥100 stars = 40, ≥50 = 30, ≥10 = 15 | | License | 10 | Has SPDX license | | Description | 15 | Description > 100 characters | | GitHub URL | 10 | Repository URL available | | **Total** | **100** | Maximum score | #### **Scoring Examples** ``` High-Quality Repository (100+ stars + license + good description): - Stars (100+): 40 points - License: 10 points - Description (>100 chars): 15 points - GitHub URL: 10 points - Total: 75/100 (Good) Popular Repository (1000+ stars + license + excellent description): - Stars (100+): 40 points (capped at 40) - License: 10 points - Description: 15 points - GitHub URL: 10 points - Total: 75/100 SkillsMP Card (No GitHub, good description): - Description: 15 points - Total: 15/100 (Low) ``` #### **Score Interpretation** - **90-100**: Excellent (Popular, well-maintained, licensed) - **70-89**: Good (Solid project with community support) - **50-69**: Fair (Emerging projects, niche tools) - **30-49**: Basic (Limited info or new projects) - **0-29**: Limited (Minimal metadata, research phase) *** ### 🧰 Technical Stack - **HTTP Requests:** requests library with asyncio executor - **HTML Parsing:** BeautifulSoup4 for SkillsMP scraping - **APIs:** GitHub REST API v3 (JSON) - **Async:** asyncio for concurrent requests - **Pattern Matching:** Python regex for text cleaning - **Proxy:** Apify Proxy with residential support - **Logging:** Apify Actor logging system - **Platform:** Apify Actor serverless environment - **Timeout:** 20 seconds per request ### 🎯 Use Cases - **Skill Inventory Building** – Create comprehensive **Skill Curator** inventories - **Learning Resource Curation** – Discover quality learning materials - **Technology Research** – Research popular skill implementations - **Competitive Analysis** – Compare skills across platforms - **Professional Development** – Find resources for skill enhancement - **Project Reference** – Discover implementation examples - **Technology Stack Planning** – Evaluate skill options - **Team Skill Assessment** – Identify skill gaps and opportunities - **Startup Research** – Discover emerging skills and tools - **Education Planning** – Build curriculum with curated resources - **Vendor Evaluation** – Assess skill availability and quality - **Open Source Discovery** – Find high-quality open source projects - **Technology Benchmarking** – Compare skills across metrics - **Knowledge Management** – Build skill knowledge bases - **Job Market Analysis** – Research in-demand skills *** #### **Limit Configuration** Balanced (20 per keyword): ```json { "limit_per_keyword": 20 } ``` Comprehensive (50 per keyword): ```json { "limit_per_keyword": 50 } ``` ### 📦 Changelog **Initial Release:** - Multi-source skill discovery (SkillsMP + GitHub) - SkillsMP HTML scraping for skill cards - GitHub API integration for repositories - Quality scoring algorithm (0-100 scale) - Star-based popularity ranking - License information extraction - Duplicate detection across sources - Author/owner attribution - Bulk keyword processing - Keyword-based search capability - Rate limiting (1 second between keywords) - Apify proxy support - Asyncio executor for non-blocking requests - Real-time Dataset push - ISO 8601 timestamp recording - Error handling and logging **Disclaimer:** Skill Curator Scraper is provided as-is for skill discovery purposes. Users are responsible for ensuring compliance with platform ToS and laws. Always respect original authors and licenses. *** ### 🎉 Get Started Today **Deploy now for skill discovery!** Use for: - 📚 Learning Resource Curation - 🔍 Skill Research - 💡 Technology Intelligence - 📋 Skill Inventory - 🎯 Professional Development **Perfect for:** - Learning Platforms - Career Coaches - Educators - Researchers - Product Managers *** ### 📚 Related Tools - Smart Article Extractor - Business Social Media Finder - Fast News Content Scraper - Startup Company Data Collector **Your complete Apify-powered skill discovery solution!** 🚀✨ *** ### 🎓 Skill Discovery Excellence This Actor is optimized for **Skill Curator** discovery with: - ✅ Multi-source aggregation - ✅ Intelligent quality scoring - ✅ GitHub API integration - ✅ SkillsMP scraping - ✅ Duplicate detection - ✅ Real-time Dataset integration - ✅ Error recovery - ✅ Production-ready code **Discover and curate skills effortlessly!** 💎🚀 # Actor input Schema ## `keywords` (type: `array`): List of keywords to search for skills. ## `limit_per_keyword` (type: `integer`): Number of results to fetch for each keyword. ## `proxyConfiguration` (type: `object`): Select the proxy to avoid rate limits. ## Actor input object example ```json { "keywords": [ "python", "automation" ], "limit_per_keyword": 20, "proxyConfiguration": { "useApifyProxy": true } } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "keywords": [ "python", "automation" ] }; // Run the Actor and wait for it to finish const run = await client.actor("datapilot/skill-curator-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "keywords": [ "python", "automation", ] } # Run the Actor and wait for it to finish run = client.actor("datapilot/skill-curator-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "keywords": [ "python", "automation" ] }' | apify call datapilot/skill-curator-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=datapilot/skill-curator-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Skill Curator Scraper", "description": "MCP Skill Scraper collects AI skills from SkillsMP and GitHub. It extracts name, description, stars, license, and URLs, then calculates a quality score. Outputs structured JSON for discovering MCP tools, AI skills, and developer resources.", "version": "0.0", "x-build-id": "v6iSLzYI3b8OImfxJ" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/datapilot~skill-curator-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-datapilot-skill-curator-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/datapilot~skill-curator-scraper/runs": { "post": { "operationId": "runs-sync-datapilot-skill-curator-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/datapilot~skill-curator-scraper/run-sync": { "post": { "operationId": "run-sync-datapilot-skill-curator-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "keywords" ], "properties": { "keywords": { "title": "Keywords", "type": "array", "description": "List of keywords to search for skills.", "items": { "type": "string" } }, "limit_per_keyword": { "title": "Max Results Per Keyword", "type": "integer", "description": "Number of results to fetch for each keyword.", "default": 20 }, "proxyConfiguration": { "title": "Proxy Configuration", "type": "object", "description": "Select the proxy to avoid rate limits.", "default": { "useApifyProxy": true } } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```