# Drupal.org Modules Scraper (`crawlerbros/drupal-org-modules-scraper`) Actor Scrape Drupal.org modules with search by keyword, browse by category, fetch trending by install count, or look up specific module IDs. Returns title, machine name, install count, compatible Drupal versions, author, categories, and more. - **URL**: https://apify.com/crawlerbros/drupal-org-modules-scraper.md - **Developed by:** [Crawler Bros](https://apify.com/crawlerbros) (community) - **Categories:** Developer tools, Automation, Other - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, NaN bookmarks - **User rating**: No ratings yet ## Pricing from $3.00 / 1,000 results This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Drupal.org Modules Scraper Scrape module listings from [Drupal.org](https://www.drupal.org/project/project_module) — the official marketplace for Drupal CMS extensions. Extract module titles, machine names, install counts, compatible Drupal versions, categories, author information, and more — no API key, no proxy, no login required. ### Data Source Data is fetched from the public Drupal.org REST JSON API (`https://www.drupal.org/api-d7/node.json`) which is openly accessible without authentication. Drupal.org is polite to scrapers when using appropriate headers. ### Features - **Search** modules by keyword (partial title match) - **Trending** modules sorted by total install count - **Browse by category** (Developer Tools, E-commerce, Media, Security, SEO, and more) - **Fetch by IDs** — retrieve specific modules by their node IDs - Filter by **compatible Drupal version** (7.x through 11.x) - Filter by **minimum install count** - Sort by installs, creation date, or last updated ### Input | Field | Type | Description | |-------|------|-------------| | `mode` | select | `search`, `byIds`, `trending`, or `byCategory` | | `searchQuery` | string | Keyword to search module titles (mode=search) | | `moduleIds` | array | List of node IDs to fetch (mode=byIds) | | `category` | select | Category TID — Developer Tools, E-commerce, Media, etc. | | `compatibleWith` | select | Drupal version filter: 7.x, 8.x, 9.x, 10.x, 11.x | | `sortBy` | select | `installs`, `created`, or `updated` | | `minInstalls` | integer | Minimum install count threshold | | `maxItems` | integer | Maximum number of results (1–1000, default 50) | #### Example Inputs **Search for SEO modules:** ```json { "mode": "search", "searchQuery": "seo", "maxItems": 20 } ```` **Top trending modules:** ```json { "mode": "trending", "maxItems": 50 } ``` **E-commerce modules compatible with Drupal 10:** ```json { "mode": "byCategory", "category": "48", "compatibleWith": "10.x", "maxItems": 30 } ``` **Fetch specific modules by ID:** ```json { "mode": "byIds", "moduleIds": ["1005090", "2660066", "3060061"] } ``` ### Output Each record contains: | Field | Type | Description | |-------|------|-------------| | `moduleId` | string | Drupal node ID | | `title` | string | Module project title | | `machineName` | string | URL-safe machine name | | `url` | string | Full project URL on Drupal.org | | `summary` | string | Module description (HTML stripped) | | `status` | string | Maintenance status (e.g., "Actively maintained") | | `installCount` | integer | Reported active installs | | `issueCount` | integer | Open issue count | | `createdAt` | string | ISO 8601 creation date | | `updatedAt` | string | ISO 8601 last updated date | | `categories` | array | Category names | | `compatibleVersions` | array | Compatible Drupal versions (e.g., \["10.x", "11.x"]) | | `authorName` | string | Module author username | | `authorUrl` | string | Author profile URL | | `screenshotUrl` | string | Screenshot image URL (when available) | | `recordType` | string | Always `"module"` | | `scrapedAt` | string | ISO 8601 scrape timestamp | Fields with no data are omitted (never null). #### Example Output Record ```json { "moduleId": "2627832", "title": "Metatag", "machineName": "metatag", "url": "https://www.drupal.org/project/metatag", "summary": "Manage meta tags for all entities.", "status": "Actively maintained", "installCount": 500000, "issueCount": 42, "createdAt": "2015-04-01T00:00:00+00:00", "updatedAt": "2024-11-01T12:00:00+00:00", "categories": ["SEO/Optimization"], "compatibleVersions": ["10.x", "11.x"], "authorName": "DamienMcKenna", "authorUrl": "https://www.drupal.org/u/DamienMcKenna", "recordType": "module", "scrapedAt": "2026-05-22T10:00:00+00:00" } ``` ### Categories | TID | Name | |-----|------| | 46 | Developer Tools | | 48 | E-commerce | | 50 | Media | | 52 | Performance and Scalability | | 54 | Security | | 56 | SEO/Optimization | | 58 | Site Structure | | 60 | Spam Prevention | | 62 | User Management | | 64 | Other | ### FAQs **Does this require a Drupal.org account or API key?** No. The Drupal.org REST API is fully public. No authentication needed. **Does this require a proxy?** No. Drupal.org serves API requests from datacenter IPs without blocking. **How many modules can I scrape?** Up to 1,000 per run (configurable via `maxItems`). Drupal.org has 50,000+ modules. **How accurate are the install counts?** Install counts are as reported by Drupal.org's update statistics system, refreshed daily. **Can I filter by multiple categories?** Currently one category per run. Run multiple times for multiple categories. **What Drupal versions are supported?** Drupal 7.x through 11.x. Filter using the `compatibleWith` input. # Actor input Schema ## `mode` (type: `string`): What to fetch. ## `searchQuery` (type: `string`): Keyword to search module titles (partial match). Used in search mode. ## `moduleIds` (type: `array`): List of Drupal node IDs (nids) to fetch. E.g. \["17345", "640498"]. ## `machineNames` (type: `array`): List of Drupal module machine names to fetch. E.g. \["views", "token", "pathauto"]. ## `category` (type: `string`): Filter by module category. Used in byCategory mode. Accepts category name (e.g. 'Security', 'Media', 'Developer Tools', 'E-commerce', 'Commerce', 'Administration', 'SEO', 'Performance', 'Integrations', 'Site Structure', 'User Engagement') or a raw taxonomy TID. ## `projectType` (type: `string`): Filter by project type. 'full' for production-ready modules, 'sandbox' for experimental/work-in-progress. ## `sortBy` (type: `string`): Sort results. 'updated' sorts by most recently changed, 'created' sorts by newest first. ## `minDownloads` (type: `integer`): Only include modules with at least this many reported installs (when available). ## `maxItems` (type: `integer`): Maximum number of modules to return. ## Actor input object example ```json { "mode": "search", "searchQuery": "seo", "moduleIds": [], "machineNames": [], "projectType": "full", "sortBy": "updated", "maxItems": 50 } ``` # Actor output Schema ## `modules` (type: `string`): Dataset containing all scraped Drupal.org modules. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "mode": "search", "searchQuery": "seo", "moduleIds": [], "machineNames": [], "projectType": "full", "sortBy": "updated", "maxItems": 50 }; // Run the Actor and wait for it to finish const run = await client.actor("crawlerbros/drupal-org-modules-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "mode": "search", "searchQuery": "seo", "moduleIds": [], "machineNames": [], "projectType": "full", "sortBy": "updated", "maxItems": 50, } # Run the Actor and wait for it to finish run = client.actor("crawlerbros/drupal-org-modules-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "mode": "search", "searchQuery": "seo", "moduleIds": [], "machineNames": [], "projectType": "full", "sortBy": "updated", "maxItems": 50 }' | apify call crawlerbros/drupal-org-modules-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=crawlerbros/drupal-org-modules-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Drupal.org Modules Scraper", "description": "Scrape Drupal.org modules with search by keyword, browse by category, fetch trending by install count, or look up specific module IDs. Returns title, machine name, install count, compatible Drupal versions, author, categories, and more.", "version": "1.0", "x-build-id": "mkVWBlRkFhCtmd6m6" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/crawlerbros~drupal-org-modules-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-crawlerbros-drupal-org-modules-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/crawlerbros~drupal-org-modules-scraper/runs": { "post": { "operationId": "runs-sync-crawlerbros-drupal-org-modules-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/crawlerbros~drupal-org-modules-scraper/run-sync": { "post": { "operationId": "run-sync-crawlerbros-drupal-org-modules-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "mode" ], "properties": { "mode": { "title": "Mode", "enum": [ "search", "byIds", "byMachineNames", "byCategory", "newest", "mostInstalled" ], "type": "string", "description": "What to fetch.", "default": "search" }, "searchQuery": { "title": "Search query (mode=search)", "type": "string", "description": "Keyword to search module titles (partial match). Used in search mode.", "default": "seo" }, "moduleIds": { "title": "Module IDs (mode=byIds)", "type": "array", "description": "List of Drupal node IDs (nids) to fetch. E.g. [\"17345\", \"640498\"].", "default": [], "items": { "type": "string" } }, "machineNames": { "title": "Machine names (mode=byMachineNames)", "type": "array", "description": "List of Drupal module machine names to fetch. E.g. [\"views\", \"token\", \"pathauto\"].", "default": [], "items": { "type": "string" } }, "category": { "title": "Category", "type": "string", "description": "Filter by module category. Used in byCategory mode. Accepts category name (e.g. 'Security', 'Media', 'Developer Tools', 'E-commerce', 'Commerce', 'Administration', 'SEO', 'Performance', 'Integrations', 'Site Structure', 'User Engagement') or a raw taxonomy TID." }, "projectType": { "title": "Project type", "enum": [ "full", "sandbox" ], "type": "string", "description": "Filter by project type. 'full' for production-ready modules, 'sandbox' for experimental/work-in-progress.", "default": "full" }, "sortBy": { "title": "Sort by", "enum": [ "updated", "created" ], "type": "string", "description": "Sort results. 'updated' sorts by most recently changed, 'created' sorts by newest first.", "default": "updated" }, "minDownloads": { "title": "Minimum install count", "minimum": 0, "maximum": 100000000, "type": "integer", "description": "Only include modules with at least this many reported installs (when available)." }, "maxItems": { "title": "Max items", "minimum": 1, "maximum": 1000, "type": "integer", "description": "Maximum number of modules to return.", "default": 50 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```