Agent.ai Scraper

What does it do?

Agent.ai Scraper extracts structured data from the Agent.ai marketplace — the largest public directory of AI agents. It crawls the platform's 2,380+ public agents and collects every key attribute: name, description, type, review scores, execution counts, pricing, author info, tags, and timestamps.

By default the scraper discovers all agents automatically via the sitemap. You can also target specific agents by slug or URL. No authentication, no browser — data is parsed directly from the server-side rendered HTML, making runs fast and cheap.

Who is it for?

• AI researchers and analysts tracking the growth and diversity of the AI agent ecosystem
• Product managers and founders benchmarking competitor agents or tracking launch activity
• Marketers and growth teams monitoring popular agents and engagement trends (execution counts, review scores)
• Developers building agent discovery tools, recommendation engines, or aggregator platforms
• Investors and VCs tracking which AI agent categories and builders are gaining traction
• Recruiters and talent scouts finding active builders with large agent portfolios on Agent.ai

Why use it?

• ✅ Extracts all 2,380+ public agents from Agent.ai in a single run
• ✅ No login or cookies required — public data only
• ✅ HTTP-only (no browser), so runs are fast and affordable
• ✅ Full data including review scores, execution counts, pricing, and author metadata
• ✅ Supports targeted scraping by providing a list of specific agent slugs or URLs
• ✅ Handles errors gracefully — invalid or private agents are skipped automatically
• ✅ All data flattened and ready for spreadsheets, databases, or BI tools

What data does it extract?

Field	Description
url	Full URL to the agent page
agentId	Internal agent ID (e.g. earningschat)
name	Display name of the agent
description	Short description of what the agent does
type	Agent type (e.g. studio, workflow, custom)
iconUrl	Agent icon image URL
reviewScore	Average review score (0–5)
reviewCount	Total number of reviews
executionCount	Number of times the agent has been run
tags	Category tags (e.g. Sales, Marketing, Operations)
price	Price value (credits per run)
isPremium	Whether the agent is a premium/paid agent
isIncludedInPro	Whether included in Agent.ai Pro plan
isMonetized	Whether the agent has monetization enabled
authorUsername	Author's Agent.ai username
authorName	Author's display name or email
authorAvatarUrl	Author profile picture URL
authorAgentsBuiltCount	Number of agents the author has built
status	Agent visibility status (e.g. public)
llm	LLM model used (if specified)
approximateTimeMins	Approximate time to run in minutes
createdAt	When the agent was created
updatedAt	When the agent was last updated

How much does it cost to scrape agents from Agent.ai?

This actor uses pay-per-event (PPE) pricing. You pay a small fee per extracted agent record, with volume discounts at higher tiers.

Volume tier	Price per agent
FREE (first 1,000)	$0.000118
BRONZE	$0.0001026
SILVER	$0.000080
GOLD	$0.0000615
PLATINUM	$0.000041
DIAMOND	$0.0000287

A run scraping all 2,380 agents costs approximately $0.005 (start) + $0.244 (results) = ~$0.25 at BRONZE pricing. With DIAMOND volume pricing for large operations, that drops to under $0.10.

The free plan on Apify gives you $5 of platform credits per month — enough to run a full scrape every two months for free.

How to scrape Agent.ai agents

1. Go to Agent.ai Scraper on Apify
2. Leave Agent slugs or URLs empty to scrape ALL public agents via the sitemap
3. Set Max agents to limit results (e.g. 100 for a sample run)
4. Click Start and wait for results
5. Download your data as JSON, CSV, or Excel

For targeted scraping, enter specific agent slugs (e.g. earningschat) or full URLs (e.g. https://agent.ai/agent/earningschat) in the Agent slugs or URLs field.

Input parameters

Parameter	Type	Required	Default	Description
agentSlugs	Array of strings	No	[]	List of agent slugs or full URLs. Leave empty to scrape all agents from the sitemap.
maxResults	Integer	No	100	Maximum number of agents to scrape. Set to 0 for no limit (scrapes all 2,380+).
proxyConfiguration	Object	No	None	Optional proxy settings. Agent.ai does not require proxies for most use cases.
maxRequestRetries	Integer	No	3	Number of retry attempts per failed HTTP request.

Example inputs

Scrape 50 agents from the sitemap:

{
  "maxResults": 50
}

Scrape specific agents by slug:

{
  "agentSlugs": ["earningschat", "competitoranalyst", "mememaker"]
}

Scrape all agents (no limit):

{
  "maxResults": 0
}

Output format

Each result is a flat JSON object:

{
  "url": "https://agent.ai/agent/earningschat",
  "agentId": "earningschat",
  "name": "Earnings Call Analyzer",
  "description": "Ask questions about a public company's earnings call discussion.",
  "type": "studio",
  "iconUrl": "https://agent.ai/icons/earnings.svg",
  "reviewScore": 4.33,
  "reviewCount": 229,
  "executionCount": 7388,
  "tags": ["Operations"],
  "price": 1,
  "isPremium": false,
  "isIncludedInPro": false,
  "isMonetized": false,
  "authorUsername": "AgentAI",
  "authorName": "agents@mail.agent.ai",
  "authorAvatarUrl": "https://s3.amazonaws.com/...",
  "authorAgentsBuiltCount": 50,
  "status": "public",
  "llm": null,
  "approximateTimeMins": 5,
  "createdAt": "Fri, 24 May 2024 00:00:00 GMT",
  "updatedAt": "Fri, 17 Apr 2026 20:58:50 GMT"
}

Tips and tricks

• 🔍 Monitor new agents: Run on a schedule (e.g. weekly) and compare results to detect newly published agents. Sort by updatedAt to find recent changes.
• 📊 Track popular agents: Sort by executionCount or reviewScore to identify top-performing agents in each category.
• 🏷️ Filter by tag: Scrape all agents and filter locally by tags (e.g. Sales, Marketing, Writing) to build category-specific lists.
• 📈 Trend analysis: Download weekly snapshots and compare executionCount growth to spot rising agents.
• 💰 Monetization analysis: Filter by isMonetized: true to track which agent builders are monetizing their work.
• ⚡ Fast targeted runs: Use agentSlugs for quick lookups of specific agents without crawling the full sitemap.
• 🚀 Full scrape: Set maxResults: 0 to scrape all 2,380+ agents. A full run takes 10–20 minutes at standard pricing.

Integrations

Agent.ai Scraper outputs to Apify's standard dataset, making integration with downstream tools simple:

• Google Sheets: Use the Apify Google Sheets integration to automatically export results to a spreadsheet. Refresh weekly to track agent growth.
• Airtable / Notion: Export as CSV from the dataset, then import into your database. Build a competitor intelligence base tracking top agents by category.
• Zapier / Make: Trigger a zap/scenario when new agents are found. For example: "When a new agent with tag Sales appears, add it to a Salesforce list."
• BigQuery / Snowflake: Use the Apify API to stream new results into your data warehouse for historical trend analysis.
• Python / Pandas: Download the dataset as JSON and load it with pd.DataFrame(results) for custom analysis (sorting, filtering, visualization).
• Slack alerts: Connect to a webhook to post a daily digest of new agents with high execution counts.

API usage

Node.js (Apify SDK)

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });

const run = await client.actor('automation-lab/agent-ai-scraper').call({
    maxResults: 100,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Python

from apify_client import ApifyClient

client = ApifyClient(token="YOUR_APIFY_TOKEN")

run = client.actor("automation-lab/agent-ai-scraper").call(run_input={
    "maxResults": 100,
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["name"], item["executionCount"])

cURL

curl -X POST \
  "https://api.apify.com/v2/acts/automation-lab~agent-ai-scraper/runs?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"maxResults": 100}'

Use with Claude and AI assistants (MCP)

You can use Agent.ai Scraper directly from Claude Code, Claude Desktop, Cursor, or VS Code via the Apify MCP server:

Claude Code (terminal)

$claude mcp add --transport http apify "https://mcp.apify.com?tools=automation-lab/agent-ai-scraper"

Claude Desktop / Cursor / VS Code

Add to your MCP config:

{
  "mcpServers": {
    "apify": {
      "type": "http",
      "url": "https://mcp.apify.com?tools=automation-lab/agent-ai-scraper",
      "headers": {
        "Authorization": "Bearer YOUR_APIFY_TOKEN"
      }
    }
  }
}

Example prompts

• "Scrape the top 50 agents from Agent.ai and show me which categories have the most activity"
• "Fetch agents with the tag 'Sales' from Agent.ai and export to a table"
• "Get execution counts for agents by earningschat, competitoranalyst, and mememaker"
• "Scrape all Agent.ai agents and find which authors have built the most agents"

Legality and compliance

Agent.ai Scraper collects only publicly available data from agent.ai. No authentication, private accounts, or protected content is accessed.

The data extracted (agent names, descriptions, execution counts, review scores) is public information visible to any visitor without logging in. This scraper respects the site's rate limits with built-in request delays.

Always review Agent.ai's Terms of Service and their Privacy Policy before using scraped data commercially. Ensure your use case complies with applicable data protection laws (GDPR, CCPA, etc.).

FAQ

How many agents can I scrape?

Agent.ai currently lists 2,380+ public agents. Set maxResults: 0 to scrape all of them in one run.

Do I need an Agent.ai account?

No. The scraper only accesses public agent pages that are visible without logging in.

How often does Agent.ai update its agent catalog?

New agents are added regularly. The sitemap updates daily. Run on a weekly schedule to stay current.

The run returned 0 results — what happened?

Check the logs. Common causes: (1) invalid agent slugs that don't exist on Agent.ai, (2) agents that have been removed or made private, (3) network timeouts. Try increasing maxRequestRetries to 5.

Can I scrape private or premium agents?

No. This scraper only accesses publicly visible agent pages. Agents with status: private or behind a paywall are not accessible.

My run is slow — how can I speed it up?

The default concurrency is 5 simultaneous requests. For faster runs on large datasets, consider running multiple actors in parallel with non-overlapping agentSlugs lists. Full scrapes of all 2,380 agents take 10–20 minutes.

Related actors

• Apify Store Scraper — Scrape the Apify actor marketplace
• HuggingFace Scraper — Scrape AI models and datasets from HuggingFace
• ProductHunt Scraper — Track AI product launches on Product Hunt