There's An AI For That Scraper

Scrape AI tools from theresanaiforthat.com with structured output including:

• Tool ID\
• Name\
• Category / task\
• Pricing text\
• Views & saves\
• Rating & review count\
• Overview text\
• Open Graph metadata\
• Screenshot / icon\
• External website URL\
• Top alternatives (name + link)

Built with Apify + Crawlee + Playwright.
Supports proxy configuration and strict crawling mode.

🚀 Features

✅ Strict Mode (Default)

• Crawls only the exact startUrls you provide\
• Does NOT crawl unrelated categories\
• Does NOT follow pagination automatically\
• Enqueues:
  • Tool detail pages found on the page\
  • /agents/period/<month|year>/ links only if displayed on the page

🔎 Supported URL Types

1️⃣ List/Search Pages

https://theresanaiforthat.com/s/aifrontpage/

2️⃣ Agents Period Pages

https://theresanaiforthat.com/agents/period/january/
https://theresanaiforthat.com/agents/period/2020/

3️⃣ Direct Tool Detail Pages

https://theresanaiforthat.com/ai/b12-io/

The actor automatically detects tool detail URLs.

📥 Input Example

{
  "startUrls": [
    "https://theresanaiforthat.com/s/aifrontpage/"
  ],
  "maxItems": 50,
  "fetchDetails": true,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}

📤 Example Output

{
  "id": "4023",
  "name": "PixieBrix",
  "task": "Workflows",
  "taskId": "13243",
  "taskSlug": "workflows",
  "internalUrl": "https://theresanaiforthat.com/ai/pixiebrix/",
  "externalUrl": "https://www.pixiebrix.com/?ref=taaft&utm_source=taaft&utm_medium=referral",
  "iconUrl": "https://media.theresanaiforthat.com/assets/favicon-large.png",
  "description": "Extend your apps, accelerate work",
  "releaseDateRelative": "21h ago",
  "pricingText": "#7 in Trending",
  "views": 53750,
  "saves": 302,
  "sourceListingUrl": "https://theresanaiforthat.com/agents/",
  "detailUrl": "https://theresanaiforthat.com/ai/pixiebrix/",
  "ogTitle": "PixieBrix v3.0.7 - AI Tool For Workflows",
  "ogDescription": "Rapidly build with low-code. Deploy as a context-aware browser extension inside your team’s apps.",
  "ogImage": "https://media.theresanaiforthat.com/pixiebrix.png",
  "ogUrl": "https://theresanaiforthat.com/ai/pixiebrix/",
  "pageTitle": "PixieBrix v3.0.7 - AI Tool For Workflows",
  "pageDescription": "Rapidly build with low-code. Deploy as a context-aware browser extension inside your team’s apps.",
  "screenshotUrl": "https://media.theresanaiforthat.com/pixiebrix.png",
  "rating": 5,
  "review_count": 2,
  "overview_text": "Key FeaturesAgentic AiLow-codeNo-code Playground...",
  "alternatives": [
    {
      "name": "Waveloom",
      "link": "https://theresanaiforthat.com/ai/waveloom/"
    },
    {
      "name": "BotDojo",
      "link": "https://theresanaiforthat.com/ai/botdojo/"
    }
  ]
}

🌐 Proxy Support

{
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}

Supports: - Apify Datacenter proxy\

• Residential proxy\
• Custom proxy groups

⚡ Performance Optimizations

• Blocks images, fonts, and media\
• Uses domcontentloaded instead of full page load\
• Deduplicates tools\
• Stops automatically when maxItems is reached\
• Safely aborts crawler\
• Handles detail URLs directly

🎯 Use Cases

• AI tool directory creation\
• Competitive research\
• Trend monitoring\
• SEO content automation\
• Dataset building\
• Market analysis\
• AI ecosystem tracking

🛠 Tech Stack

• Apify SDK v3\
• Crawlee 3.x\
• Playwright\
• Node.js 22

👨‍💻 Maintainer

Voyn Software