# Terraform Registry Scraper (`crawlerbros/terraform-registry-scraper`) Actor Scrape the Terraform Registry - search modules and providers, or fetch details for specific modules/providers. Returns downloads, versions, descriptions, source URLs, and verification status. - **URL**: https://apify.com/crawlerbros/terraform-registry-scraper.md - **Developed by:** [Crawler Bros](https://apify.com/crawlerbros) (community) - **Categories:** Automation, Developer tools, Integrations - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, NaN bookmarks - **User rating**: No ratings yet ## Pricing from $3.00 / 1,000 results This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Terraform Registry Scraper Search and extract data from the [Terraform Registry](https://registry.terraform.io/) — HashiCorp's official repository for Terraform modules and providers. Retrieve module details, provider information, download counts, version histories, and verification status without any authentication. ### What Does This Actor Do? This actor scrapes the public Terraform Registry API (`registry.terraform.io`) to help you: - **Discover Terraform modules** by searching across thousands of community and verified modules - **Find Terraform providers** covering AWS, Azure, Google Cloud, Kubernetes, and hundreds more - **Get detailed metadata** for specific modules or providers by namespace/name - **Monitor download trends** and identify the most popular infrastructure components - **Track module versions** and see the full version history of any module or provider ### Output Data Each record contains fields such as: | Field | Description | |---|---| | `id` | Full registry ID (e.g. `terraform-aws-modules/vpc/aws/6.6.1`) | | `namespace` | Publisher namespace (e.g. `terraform-aws-modules`) | | `name` | Module or provider name | | `provider` | Cloud provider (modules only, e.g. `aws`, `azurerm`, `google`) | | `version` | Latest published version | | `description` | Short description of the module/provider | | `owner` | Owner/author username | | `source` | Source code repository URL | | `sourceUrl` | Canonical Terraform Registry URL | | `published_at` | ISO 8601 publication timestamp | | `downloads` | Total download count | | `verified` | Whether the module is HashiCorp-verified | | `versions` | List of all published versions | | `tier` | Provider tier: `official`, `partner`, or `community` | | `logo_url` | Provider logo image URL | | `recordType` | Record type: `module` or `provider` | | `scrapedAt` | ISO 8601 timestamp of when the record was scraped | ### Input Configuration #### Mode Select the scraping mode: | Mode | Description | |---|---| | `searchModules` | Full-text search across all Terraform modules | | `searchProviders` | Browse and search all Terraform providers | | `getModule` | Fetch a specific module by namespace/name/provider | | `getProvider` | Fetch a specific provider by namespace/type | #### Input Fields | Field | Type | Description | |---|---|---| | `mode` | string (required) | Scraping mode (see above) | | `query` | string | Free-text search query (searchModules / searchProviders) | | `provider` | string | Filter modules by provider, e.g. `aws`, `azurerm`, `google` (searchModules) | | `namespace` | string | Publisher namespace for direct lookup (getModule / getProvider) | | `name` | string | Module name for direct lookup (getModule) | | `providerType` | string | Provider type for direct lookup (getProvider) | | `verified` | boolean | Only return verified modules (default: `false`) | | `maxItems` | integer | Maximum records to return (1–10000, default: 100) | ### Example Inputs #### Search VPC modules for AWS ```json { "mode": "searchModules", "query": "vpc", "provider": "aws", "maxItems": 50 } ```` #### Search all AWS providers ```json { "mode": "searchProviders", "query": "aws", "maxItems": 20 } ``` #### Get a specific module ```json { "mode": "getModule", "namespace": "terraform-aws-modules", "name": "vpc", "provider": "aws" } ``` #### Get the HashiCorp AWS provider ```json { "mode": "getProvider", "namespace": "hashicorp", "providerType": "aws" } ``` #### Only verified modules ```json { "mode": "searchModules", "query": "kubernetes", "verified": true, "maxItems": 100 } ``` ### Use Cases - **Infrastructure discovery** — Find reusable Terraform modules for your cloud infrastructure - **DevOps tooling** — Build internal catalogs of approved/verified Terraform modules - **Market research** — Analyze download trends and popular providers in the IaC ecosystem - **Security auditing** — Track module versions and identify outdated infrastructure components - **CI/CD automation** — Monitor new module releases and automate dependency updates - **Developer portals** — Populate internal developer portals with curated Terraform resources ### FAQ **Do I need authentication?** No. The Terraform Registry public API requires no API keys or credentials. **How many records can I get?** Up to 10,000 records per run. The registry contains thousands of modules and hundreds of providers. **What providers are available?** The registry includes providers for all major clouds (AWS, Azure, Google Cloud, Oracle, IBM), databases (PostgreSQL, MySQL, MongoDB), networking (Cloudflare, Datadog), and hundreds of SaaS platforms. **Are versions included?** Yes, the `versions` field contains the full list of published versions for each module or provider. **Can I filter by provider in searchModules?** Yes, use the `provider` field to filter by provider slug (e.g. `aws`, `azurerm`, `google`, `kubernetes`). **What is a verified module?** HashiCorp-verified modules are maintained by trusted partners and follow HashiCorp's module standards. Set `verified: true` to only return these. # Actor input Schema ## `mode` (type: `string`): Select what to fetch from the Terraform Registry. ## `query` (type: `string`): Free-text search query (used in searchModules and searchProviders modes). ## `provider` (type: `string`): Filter modules by provider name, e.g. aws, azurerm, google, kubernetes (mode=searchModules only). ## `namespace` (type: `string`): Module or provider namespace (required for getModule and getProvider modes, e.g. hashicorp, terraform-aws-modules). ## `name` (type: `string`): Module name within the namespace (required for getModule mode, e.g. vpc, s3-bucket). ## `providerType` (type: `string`): Provider type/name within the namespace (required for getProvider mode, e.g. aws, google, azurerm). ## `verified` (type: `boolean`): Only return verified modules/providers (searchModules only). ## `maxItems` (type: `integer`): Maximum number of records to return. ## Actor input object example ```json { "mode": "searchModules", "query": "vpc", "provider": "aws", "verified": false, "maxItems": 100 } ``` # Actor output Schema ## `results` (type: `string`): Dataset containing all scraped Terraform modules and providers. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "mode": "searchModules", "query": "vpc", "provider": "aws", "verified": false, "maxItems": 100 }; // Run the Actor and wait for it to finish const run = await client.actor("crawlerbros/terraform-registry-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "mode": "searchModules", "query": "vpc", "provider": "aws", "verified": False, "maxItems": 100, } # Run the Actor and wait for it to finish run = client.actor("crawlerbros/terraform-registry-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "mode": "searchModules", "query": "vpc", "provider": "aws", "verified": false, "maxItems": 100 }' | apify call crawlerbros/terraform-registry-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=crawlerbros/terraform-registry-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Terraform Registry Scraper", "description": "Scrape the Terraform Registry - search modules and providers, or fetch details for specific modules/providers. Returns downloads, versions, descriptions, source URLs, and verification status.", "version": "1.0", "x-build-id": "UVYnKFFGQFVMlecjc" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/crawlerbros~terraform-registry-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-crawlerbros-terraform-registry-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/crawlerbros~terraform-registry-scraper/runs": { "post": { "operationId": "runs-sync-crawlerbros-terraform-registry-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/crawlerbros~terraform-registry-scraper/run-sync": { "post": { "operationId": "run-sync-crawlerbros-terraform-registry-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "mode" ], "properties": { "mode": { "title": "Mode", "enum": [ "searchModules", "searchProviders", "getModule", "getProvider" ], "type": "string", "description": "Select what to fetch from the Terraform Registry.", "default": "searchModules" }, "query": { "title": "Search query", "type": "string", "description": "Free-text search query (used in searchModules and searchProviders modes)." }, "provider": { "title": "Provider filter (searchModules)", "type": "string", "description": "Filter modules by provider name, e.g. aws, azurerm, google, kubernetes (mode=searchModules only)." }, "namespace": { "title": "Namespace", "type": "string", "description": "Module or provider namespace (required for getModule and getProvider modes, e.g. hashicorp, terraform-aws-modules)." }, "name": { "title": "Module name", "type": "string", "description": "Module name within the namespace (required for getModule mode, e.g. vpc, s3-bucket)." }, "providerType": { "title": "Provider type", "type": "string", "description": "Provider type/name within the namespace (required for getProvider mode, e.g. aws, google, azurerm)." }, "verified": { "title": "Verified only", "type": "boolean", "description": "Only return verified modules/providers (searchModules only).", "default": false }, "maxItems": { "title": "Max items", "minimum": 1, "maximum": 10000, "type": "integer", "description": "Maximum number of records to return.", "default": 100 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```