AI Agent Marketplace Scraper

Extract structured data from AI agent marketplaces with ease. This actor crawls marketplace listings and collects detailed information about AI agents, making it ideal for market research, lead generation, competitive analysis, and AI directory creation.

Features

• Scrape AI agent listings
• Extract agent names
• Collect descriptions
• Capture categories
• Get developer/company names
• Extract pricing information
• Collect ratings and reviews
• Extract tags
• Capture website URLs
• Extract demo links
• Collect GitHub repositories
• Extract documentation URLs
• Detect API availability
• Detect open-source projects
• Capture marketplace URLs
• Extract agent images
• Export JSON
• Export CSV
• Access data via API

---

Input

Example:

{
  "startUrls": [
    {
      "url": "https://example.com/marketplace"
    }
  ],
  "maxItems": 100
}

Input Fields

Field	Type	Description
startUrls	Array	Marketplace URLs to scrape
maxItems	Integer	Maximum number of agents to collect

---

Output

Example:

{
  "agentName": "AI Research Assistant",
  "description": "AI-powered research and summarization agent.",
  "category": "Productivity",
  "developer": "Example AI",
  "pricing": "Free",
  "rating": 4.8,
  "reviews": 245,
  "tags": [
    "AI",
    "Research",
    "Automation"
  ],
  "website": "https://example.com",
  "demoUrl": "https://example.com/demo",
  "github": "https://github.com/example/project",
  "documentation": "https://docs.example.com",
  "apiAvailable": true,
  "openSource": true,
  "image": "https://example.com/image.png",
  "marketplaceUrl": "https://example.com/agent"
}

---

Typical Use Cases

• AI market research
• Competitor analysis
• AI directory creation
• Product comparison
• Startup research
• AI trend analysis
• Lead generation
• Dataset creation
• Business intelligence
• Investment research

---

Export Formats

• JSON
• CSV
• Excel
• API

---

Performance

• Fast crawling
• Structured output
• Duplicate reduction
• Reliable extraction
• Scalable for large datasets

---

Requirements

• Valid marketplace URLs
• Internet access during actor execution

---

Limitations

Some marketplaces may:

• Require authentication
• Limit requests
• Block automated traffic
• Load data dynamically

Additional customization may be required for certain websites.

---

Best Practices

• Start with a small maxItems value for testing.
• Verify extracted data before large-scale runs.
• Respect website Terms of Service and robots.txt.
• Avoid excessive request rates.

---

Support

If you encounter issues or have feature requests, please open an issue or contact the actor author.

---

License

MIT License