# Ansible Galaxy Scraper (`kawsar/ansible-galaxy-scraper`) Actor

Easily extract Ansible collection details, download counts, and legacy roles without extra configuration. Collect rich data across namespaces and download structured datasets directly as JSON, CSV, or Excel files.

- **URL**: https://apify.com/kawsar/ansible-galaxy-scraper.md
- **Developed by:** [Kawsar](https://apify.com/kawsar) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.99 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### Ansible Galaxy Scraper: Search Ansible Collections Easily

Ansible Galaxy Scraper lets you search, discover, and export collection versions, download stats, and legacy roles from Ansible Galaxy without hassle. Everything you extract gets stored neatly in an Apify dataset ready to download as CSV, JSON, or Excel.

The tool processes queries step by step, pausing briefly between requests to ensure reliable execution.

### Common Use Cases

- **Collection Discovery**: Search for topics like `aws` or `kubernetes` and easily pull the newest release of each collection.
- **Track Downloads**: Gather `download_count` figures for specific collections like `amazon.aws` to analyze adoption rates or usage trends over time.
- **Role Inventories**: Search for legacy roles by keyword (e.g., `nginx`) and compile details like GitHub repository links, branches, and commits.
- **Data Export**: Save your search results or stats directly into spreadsheets or data pipelines.
- **Scheduled Monitoring**: Set up recurring runs to check for updated collection versions or changed IDs automatically.

### Data Fields Extracted

Depending on the mode you run, here is a breakdown of the data fields provided:

| Field | Description |
|-------|-------------|
| `item_id` | Deduplication key for the row. |
| `namespace` | Collection namespace, or legacy role creator username. |
| `name` | Collection name, or legacy role repo name. |
| `version` | Collection version tag. |
| `requires_ansible` | Minimum required Ansible version. |
| `collectionDescription` | Short summary text. |
| `tags` | Tags assigned to the collection version. |
| `repository` | Repository classification (e.g., `published`). |
| `is_deprecated` | Indicates whether the collection is deprecated. |
| `is_signed` | Indicates signature status when available. |
| `download_count` | Total download stats. |
| `role_id` | Legacy role identifier. |
| `github_user` | GitHub account hosting the legacy role. |
| `github_repo` | GitHub repository name for the legacy role. |
| `github_branch` | Branch name imported for the role. |
| `commit_sha` | Commit SHA recorded during import. |
| `upstream_id` | Associated upstream role identifier. |
| `scraped_at` | UTC timestamp when the record was retrieved. |
| `error` | Set to true if a specific record or page encountered an issue. |
| `url` | Web address associated with a failed request. |
| `status` | Response status code. |
| `message` | Informational message explaining any issue. |

Each extracted row uses `item_id` to prevent duplicate entries. If a page or collection returns a missing result, it registers as an error row rather than crashing the execution.

### Input Parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `mode` | string | `collections` | Mode options: `collections` (version search), `collection_detail` (detailed info), or `roles` (legacy roles search). |
| `keywords` | string | `aws` | Search keyword for collections or roles. |
| `collections` | array | `["amazon.aws"]` | List of `namespace.name` collections to fetch details for. |
| `highestOnly` | boolean | `true` | Restricts collection search results to the highest release version. |
| `maxItems` | integer | `100` | Maximum rows to collect (up to 1000). |
| `requestDelaySecs` | number | `1` | Delay in seconds between sequential requests. |
| `timeoutSecs` | integer | `300` | Total run timeout in seconds. |
| `requestTimeoutSecs` | integer | `30` | Per-request timeout limit. |
| `userAgent` | string | `BigiByteActorResearch/1.0` | Custom HTTP User-Agent string. |
| `proxyConfiguration` | object | `{"useApifyProxy": false}` | Network connection proxy configuration. |

#### Sample Inputs

**Searching Collections:**

```json
{
    "mode": "collections",
    "keywords": "aws",
    "highestOnly": true,
    "maxItems": 100,
    "requestDelaySecs": 1,
    "timeoutSecs": 300,
    "requestTimeoutSecs": 30,
    "userAgent": "BigiByteActorResearch/1.0 (contact: research@bigibyte.com)",
    "proxyConfiguration": { "useApifyProxy": false }
}

```

**Fetching Specific Collection Details:**

```json
{
    "mode": "collection_detail",
    "collections": ["amazon.aws"],
    "maxItems": 100,
    "proxyConfiguration": { "useApifyProxy": false }
}

```

### How It Works

1. **Configuration**: The tool verifies your inputs, validates limits, and configures delay and network settings.
2. **Data Processing**: Requests are sent sequentially using your custom User-Agent header.
3. **Resilience & Retry**: Temporary network issues automatically trigger standard backoff retries.
4. **Pagination**: Extraction stops when reaching `maxItems`, hitting user timeouts, or running out of additional pages.
5. **Deduplication & Storage**: Duplicate entries are automatically filtered before writing batches directly into the Apify dataset.

### Frequently Asked Questions

**How many items can I collect in one run?**
By default, `maxItems` is set to 100 and can be adjusted up to 1000 items per run.

**What happens if a collection name doesn't exist?**
If a target item is not found, the run logs a single error record containing the status code and message, then moves seamlessly to the next item.

**Can I export the data directly to Excel or CSV?**
Yes. You can export your dataset directly from the Apify console in CSV, Excel, or JSON formats.

### Trademark Notice

Ansible Galaxy and related names, logos, and trademarks belong to their respective owners. Their use in this documentation is solely to identify the subject matter and does not imply affiliation, sponsorship, endorsement, or partnership.

### Disclaimer

This tool is an independent scraper and is not officially associated with, endorsed by, or sponsored by Ansible Galaxy or its parent entities. Users are responsible for complying with all applicable terms of use, guidelines, and licenses.

### Is web scraping legal?

This Actor only collects publicly available data without bypassing access controls. Scraping public data is generally legal, but how you use it matters:

- **Personal data:** If results include personal details (such as developer names or contact info), privacy laws like the GDPR or CCPA may apply depending on your region.
- **Copyright:** Content like descriptions or README files may be subject to copyright. Avoid republishing full copyrighted text verbatim.
- **Terms of use:** Always respect fair usage practices, rate limitations, and source attribution rules.
- **Legal advice:** This overview provides general information and does not constitute formal legal counsel.

To learn more, read Apify's guide: [Is web scraping legal?](https://blog.apify.com/is-web-scraping-legal/)

# Actor input Schema

## `mode` (type: `string`):

Choose collection search, one collection's download count, or legacy role search.

## `keywords` (type: `string`):

Search text sent as keywords for collection versions, or as keyword for legacy roles.

## `collections` (type: `array`):

Collection ids as namespace.name, one per line. Used when mode is collection_detail.

## `highestOnly` (type: `boolean`):

When searching collections, send is_highest=true so each collection appears once at its newest version.

## `maxItems` (type: `integer`):

Stop after this many records. Hard cap is 1000.

## `requestDelaySecs` (type: `number`):

Seconds to wait between Galaxy requests. Values under 1 are raised to 1.

## `timeoutSecs` (type: `integer`):

Stop the run after this many seconds, even if maxItems is not reached.

## `requestTimeoutSecs` (type: `integer`):

How long to wait for a single Galaxy HTTP response.

## `userAgent` (type: `string`):

Identify the client in the User-Agent header, including a contact address.

## `proxyConfiguration` (type: `object`):

Optional proxy for Galaxy requests. Leave useApifyProxy off unless the source blocks your IP.

## Actor input object example

```json
{
  "mode": "collections",
  "keywords": "aws",
  "collections": [
    "amazon.aws"
  ],
  "highestOnly": true,
  "maxItems": 100,
  "requestDelaySecs": 1,
  "timeoutSecs": 300,
  "requestTimeoutSecs": 30,
  "userAgent": "",
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `datasetUrl` (type: `string`):

No description

## `datasetItemsUrl` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "mode": "collections",
    "keywords": "aws",
    "collections": [
        "amazon.aws"
    ],
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("kawsar/ansible-galaxy-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "mode": "collections",
    "keywords": "aws",
    "collections": ["amazon.aws"],
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("kawsar/ansible-galaxy-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "mode": "collections",
  "keywords": "aws",
  "collections": [
    "amazon.aws"
  ],
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call kawsar/ansible-galaxy-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,kawsar/ansible-galaxy-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/TqEbNMeWmXL2LNhoR/builds/NW5h4QG6Pzn3YJo9h/openapi.json