# Go Modules Scraper (`bigibyte/go-modules-scraper`) Actor

Go Modules Scraper collects Go module versions, release details, index feeds, and package metadata. Pick a operational mode, pass module paths or a start timestamp, and extract dataset rows ready to export as JSON, CSV, or Excel spreadsheets.

- **URL**: https://apify.com/bigibyte/go-modules-scraper.md
- **Developed by:** [BigiByte](https://apify.com/bigibyte) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Go Modules Scraper

Go Modules Scraper helps you easily gather Go module versions and package details. Whether you need to inspect existing projects, track fresh releases, or audit dependencies, you can pick a mode, provide your module paths or a start timestamp, and collect structured data ready for analysis.

Outputs can be exported seamlessly into JSON, CSV, or Excel formats.

### Use Cases

- **Dependency Inventory**: Keep track of published versions for modules used in your Go services.
- **Release Tracking**: Monitor the latest versions, publish timestamps, and commit details for specific module paths.
- **Index Monitoring**: Catch newly published module versions over time based on specific timestamps.
- **Deprecation Checks**: Quickly check for deprecated packages in your software stack.
- **Spreadsheet Exports**: Export your dataset as a spreadsheet for quick reviews or scheduled audits.

### Operation Modes

- **Versions (`versions`)**: Collects published versions for your selected modules line by line.
- **Latest (`latest`)**: Pulls the newest release version, publish timestamp, source URL, and commit hash.
- **New Versions Feed (`index_feed`)**: Scans recently published module versions using page limits and optional timestamp filters.
- **Metadata (`metadata`)**: Retrieves detailed package information per version, including deprecation flags.

Uppercase letters in module paths are automatically formatted according to standard Go path rules before processing.

### Input Parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `mode` | string | `versions` | Select from: `versions`, `latest`, `index_feed`, or `metadata`. |
| `modules` | array | `github.com/gin-gonic/gin` | List of module paths to process. |
| `since` | string | empty | Timestamp filter for feed tracking. |
| `maxItems` | integer | `100` | Maximum rows to collect per run (capped at 1000). |
| `requestDelaySecs` | number | `1` | Pause duration between consecutive requests. |
| `userAgent` | string | `BigiByteActorResearch/1.0 (contact: research@bigibyte.com)` | Custom identifier sent with requests. |
| `timeoutSecs` | integer | `300` | Total run timeout in seconds. |
| `requestTimeoutSecs` | integer | `30` | Timeout per individual request. |
| `proxyConfiguration` | object | Default settings | Optional proxy setup if required. |

#### Example Input

```json
{
  "mode": "versions",
  "modules": ["[github.com/gin-gonic/gin](https://github.com/gin-gonic/gin)"],
  "since": "",
  "maxItems": 100,
  "requestDelaySecs": 1,
  "userAgent": "BigiByteActorResearch/1.0 (contact: research@bigibyte.com)",
  "timeoutSecs": 300,
  "requestTimeoutSecs": 30,
  "proxyConfiguration": { "useApifyProxy": false }
}

```

### Extracted Data Format

Here is an example of the output structure generated in `latest` mode:

```json
{
  "module": "[github.com/gin-gonic/gin](https://github.com/gin-gonic/gin)",
  "Version": "v1.12.0",
  "Time": "2026-02-28T10:10:09Z",
  "Origin_URL": "[https://github.com/gin-gonic/gin](https://github.com/gin-gonic/gin)",
  "Origin_Hash": "73726dc606796a025971fe451f0aa6f1b9b847f6",
  "scraped_at": "2026-10-12T00:00:00+00:00"
}

```

| Field | Type | Description |
| --- | --- | --- |
| `module` | string | Module path or target identifier. |
| `Version` | string | Extracted version string. |
| `Time` | string | Publication timestamp. |
| `Origin_URL` | string | Source repository URL. |
| `Origin_Hash` | string | Associated commit hash. |
| `versions` | array | List of version strings collected in versions mode. |
| `isDeprecated` | boolean | Indicates whether the package version is deprecated. |
| `scraped_at` | string | UTC timestamp when the record was generated. |
| `error` | boolean | Included if an issue occurred during lookup. |
| `url` | string | Target URL involved in an error. |
| `status` | integer | HTTP response status code. |
| `message` | string | Explanation of why a lookup failed. |

### Pricing & Usage Limits

This tool is free to use. You only pay for standard Apify platform runtime and storage fees.

To prevent unnecessary rate limits, requests are automatically spaced out with default delay settings and automatic retries. Runs are capped at a maximum of 1,000 items (`maxItems`) or governed by your specified runtime timeout (`timeoutSecs`).

### How It Works

1. **Configuration**: The tool reads your requested mode, module list, item limits, and timing preferences.
2. **Data Extraction**: Performs structured requests tailored to your selected operation mode.
3. **Error Handling**: Handles rate limits gracefully with built-in retries while generating clear error records for missing modules.
4. **Output Storage**: Formats the extracted records and saves them cleanly into your dataset.

### FAQ

**Can I export module data to CSV?**

Yes, datasets can be downloaded in JSON, CSV, or Excel formats anytime after a run completes.

**What happens if a module path is incorrect?**

If a path cannot be found, an error row is recorded with the status code and explanation. The rest of your requested modules will continue processing normally.

**How many versions can I collect per run?**

`maxItems` defaults to 100 items per run and can be set up to 1,000. Total execution is also subject to your configured `timeoutSecs`.

### Related Tools

If you plan to organize, aggregate, or process these module dependencies, a CSV cleanup or custom data pipeline is often the next step. BigiByte also builds custom scrapers and automation tools for other package registries and data portals when you need tailored data workflows.

Need a scheduled dependency monitor, custom data feed, or specialized integration? BigiByte builds custom scrapers and APIs: https://bigibyte.com

### Trademark Notice

Go Modules and related names, logos, and trademarks belong to their respective owners. Their mention in this documentation is strictly for identification purposes and does not imply affiliation, sponsorship, or endorsement.

### Disclaimer

This scraper is an independent automation tool. It is not affiliated with, endorsed by, or officially associated with Go Modules or its parent entities. All data collected originates from publicly available resources. Users are responsible for adhering to relevant terms of use and data handling policies.

### Is web scraping legal?

This tool only collects publicly available data without bypassing any security or access controls. Collecting public data is generally legal, but how you manage the resulting data is important:

- **Personal Data**: If your datasets contain personal information, regulations such as GDPR or CCPA may apply. Ensure you handle personal data under valid legal bases.
- **Copyright**: Certain text descriptions or media may be protected by copyright. Avoid republishing copyrighted content without authorization.
- **Terms of Use**: Respect standard rate limits and source guidelines.
- **Legal Advice**: This overview provides general information, not formal legal counsel. Consult a qualified attorney for specific legal guidance.

For further reading, check out Apify's full guide: [Is web scraping legal?](https://blog.apify.com/is-web-scraping-legal/)

# Actor input Schema

## `mode` (type: `string`):

What to scrape. Versions and latest read proxy.golang.org. The new versions feed reads index.golang.org. deps.dev reads api.deps.dev package metadata.

## `modules` (type: `array`):

Lowercase module paths (uppercase letters are escaped to !lower per the Go module proxy spec). Used by versions, latest, and deps.dev. Ignored by the index feed.

## `since` (type: `string`):

index_feed RFC3339 start. Leave empty to read from the beginning of the public module index. Later pages continue from the last timestamp returned.

## `maxItems` (type: `integer`):

Free cap (hard max 1000). Stops the run once this many dataset rows have been collected, including error rows.

## `requestDelaySecs` (type: `number`):

Polite pacing. Seconds to wait between requests. Default is 1 because the public module hosts do not publish a rate limit.

## `userAgent` (type: `string`):

Identify your app with a contact. Sent on every request. Some hosts reject an empty User-Agent.

## `timeoutSecs` (type: `integer`):

Stop the run after this many seconds, even if more modules or index pages are still queued.

## `requestTimeoutSecs` (type: `integer`):

How long to wait for a single response before recording that request as an error.

## `proxyConfiguration` (type: `object`):

Optional proxy for module requests. Leave Apify Proxy off unless a host blocks your IP.

## Actor input object example

```json
{
  "mode": "versions",
  "modules": [
    "github.com/gin-gonic/gin"
  ],
  "since": "",
  "maxItems": 100,
  "requestDelaySecs": 1,
  "userAgent": "BigiByteActorResearch/1.0 (contact: research@bigibyte.com)",
  "timeoutSecs": 300,
  "requestTimeoutSecs": 30,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `datasetUrl` (type: `string`):

Default dataset for this run.

## `overviewUrl` (type: `string`):

Dataset rows in the overview table.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "modules": [
        "github.com/gin-gonic/gin"
    ],
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("bigibyte/go-modules-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "modules": ["github.com/gin-gonic/gin"],
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("bigibyte/go-modules-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "modules": [
    "github.com/gin-gonic/gin"
  ],
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call bigibyte/go-modules-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,bigibyte/go-modules-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/3CxfYQfi8RfNaMhyO/builds/12FvIyTavGakkjR4H/openapi.json