Ansible Galaxy Scraper

Ansible Galaxy Scraper: Search Ansible Collections Easily

Ansible Galaxy Scraper lets you search, discover, and export collection versions, download stats, and legacy roles from Ansible Galaxy without hassle. Everything you extract gets stored neatly in an Apify dataset ready to download as CSV, JSON, or Excel.

The tool processes queries step by step, pausing briefly between requests to ensure reliable execution.

Common Use Cases

• Collection Discovery: Search for topics like aws or kubernetes and easily pull the newest release of each collection.
• Track Downloads: Gather download_count figures for specific collections like amazon.aws to analyze adoption rates or usage trends over time.
• Role Inventories: Search for legacy roles by keyword (e.g., nginx) and compile details like GitHub repository links, branches, and commits.
• Data Export: Save your search results or stats directly into spreadsheets or data pipelines.
• Scheduled Monitoring: Set up recurring runs to check for updated collection versions or changed IDs automatically.

Data Fields Extracted

Depending on the mode you run, here is a breakdown of the data fields provided:

Field	Description
item_id	Deduplication key for the row.
namespace	Collection namespace, or legacy role creator username.
name	Collection name, or legacy role repo name.
version	Collection version tag.
requires_ansible	Minimum required Ansible version.
collectionDescription	Short summary text.
tags	Tags assigned to the collection version.
repository	Repository classification (e.g., published).
is_deprecated	Indicates whether the collection is deprecated.
is_signed	Indicates signature status when available.
download_count	Total download stats.
role_id	Legacy role identifier.
github_user	GitHub account hosting the legacy role.
github_repo	GitHub repository name for the legacy role.
github_branch	Branch name imported for the role.
commit_sha	Commit SHA recorded during import.
upstream_id	Associated upstream role identifier.
scraped_at	UTC timestamp when the record was retrieved.
error	Set to true if a specific record or page encountered an issue.
url	Web address associated with a failed request.
status	Response status code.
message	Informational message explaining any issue.

Each extracted row uses item_id to prevent duplicate entries. If a page or collection returns a missing result, it registers as an error row rather than crashing the execution.

Input Parameters

Parameter	Type	Default	Description
mode	string	collections	Mode options: collections (version search), collection_detail (detailed info), or roles (legacy roles search).
keywords	string	aws	Search keyword for collections or roles.
collections	array	["amazon.aws"]	List of namespace.name collections to fetch details for.
highestOnly	boolean	true	Restricts collection search results to the highest release version.
maxItems	integer	100	Maximum rows to collect (up to 1000).
requestDelaySecs	number	1	Delay in seconds between sequential requests.
timeoutSecs	integer	300	Total run timeout in seconds.
requestTimeoutSecs	integer	30	Per-request timeout limit.
userAgent	string	BigiByteActorResearch/1.0	Custom HTTP User-Agent string.
proxyConfiguration	object	{"useApifyProxy": false}	Network connection proxy configuration.

Sample Inputs

Searching Collections:

{
    "mode": "collections",
    "keywords": "aws",
    "highestOnly": true,
    "maxItems": 100,
    "requestDelaySecs": 1,
    "timeoutSecs": 300,
    "requestTimeoutSecs": 30,
    "userAgent": "BigiByteActorResearch/1.0 (contact: research@bigibyte.com)",
    "proxyConfiguration": { "useApifyProxy": false }
}

Fetching Specific Collection Details:

{
    "mode": "collection_detail",
    "collections": ["amazon.aws"],
    "maxItems": 100,
    "proxyConfiguration": { "useApifyProxy": false }
}

How It Works

1. Configuration: The tool verifies your inputs, validates limits, and configures delay and network settings.
2. Data Processing: Requests are sent sequentially using your custom User-Agent header.
3. Resilience & Retry: Temporary network issues automatically trigger standard backoff retries.
4. Pagination: Extraction stops when reaching maxItems, hitting user timeouts, or running out of additional pages.
5. Deduplication & Storage: Duplicate entries are automatically filtered before writing batches directly into the Apify dataset.

Frequently Asked Questions

How many items can I collect in one run? By default, maxItems is set to 100 and can be adjusted up to 1000 items per run.

What happens if a collection name doesn't exist? If a target item is not found, the run logs a single error record containing the status code and message, then moves seamlessly to the next item.

Can I export the data directly to Excel or CSV? Yes. You can export your dataset directly from the Apify console in CSV, Excel, or JSON formats.

Trademark Notice

Ansible Galaxy and related names, logos, and trademarks belong to their respective owners. Their use in this documentation is solely to identify the subject matter and does not imply affiliation, sponsorship, endorsement, or partnership.

Disclaimer

This tool is an independent scraper and is not officially associated with, endorsed by, or sponsored by Ansible Galaxy or its parent entities. Users are responsible for complying with all applicable terms of use, guidelines, and licenses.

Is web scraping legal?

This Actor only collects publicly available data without bypassing access controls. Scraping public data is generally legal, but how you use it matters:

• Personal data: If results include personal details (such as developer names or contact info), privacy laws like the GDPR or CCPA may apply depending on your region.
• Copyright: Content like descriptions or README files may be subject to copyright. Avoid republishing full copyrighted text verbatim.
• Terms of use: Always respect fair usage practices, rate limitations, and source attribution rules.
• Legal advice: This overview provides general information and does not constitute formal legal counsel.

To learn more, read Apify's guide: Is web scraping legal?