Agentic AI Explorer — Developer Edition

An Apify Actor that scrapes developer-focused Agentic AI sources and converts them into structured insights for developers.

🎯 Overview

This Actor continuously monitors and aggregates the latest developments in agentic AI from multiple sources, transforming raw data into actionable developer insights with code examples and use cases.

📡 Data Sources

The Actor fetches data from:

1. GitHub - Trending repositories related to AI, ML, LLM, and Agent frameworks
2. HuggingFace - Newest models and agent frameworks
3. PapersWithCode - New agentic AI research papers
4. dev.to - Developer articles and tutorials on agentic AI
5. Medium - Blog posts and articles about AI/ML developments
6. LLM-Powered Internet Search - Uses AI search APIs (Tavily, Perplexity, Serper) to find relevant content across the web

🚀 Features

• Multi-source aggregation - Collects data from 6+ sources in parallel
• Developer-focused insights - Transforms raw data into actionable developer information
• Code examples - Generates relevant code examples for each tool/framework
• Multiple output formats - Supports JSON (default) and Markdown output
• Daily subscriptions - Automatically receive fresh insights every day at your preferred time
• Error resilience - Never crashes on partial failures
• Fully typed - Complete TypeScript type safety

📦 Output Structure

Each scraped item produces:

{
  toolName: string;
  source: string;
  summary: string;
  developerUseCase: string;
  codeExample?: string;
  futureOutlook?: string;
}

Markdown Format

When using Markdown output, generates readable sections:

### Tool Name
**Source:** GitHub
**Summary:** ...
**Developer Use Case:** ...
**Code Example:**

// code here

**Future Outlook:** ...

🔑 API Keys & Authentication

Good news: No API keys required for core functionality!

All default data sources work without any API keys:

• ✅ GitHub - Uses public API (no authentication needed)
• ✅ HuggingFace - Uses public API (no authentication needed)
• ✅ PapersWithCode - Uses public API (no authentication needed)
• ✅ dev.to - Uses public API (no authentication needed)
• ✅ Medium - Uses web scraping (no authentication needed)

Optional API keys:

• 🔐 LLM Search - Only required if you enable enableLLMSearch: true
  • Choose one: TAVILY_API_KEY, PERPLEXITY_API_KEY, or SERPER_API_KEY
  • See LLM-Powered Internet Search section for setup

Optional (for higher rate limits):

• 🔐 GitHub Token - Optional GitHub personal access token for higher API rate limits (60/hour → 5000/hour)
  • Set as GITHUB_TOKEN environment variable (not required, Actor works fine without it)

⚙️ Input Parameters

• topics (string[]): Topics to search for (default: ["agentic ai", "llm frameworks"])
• maxItems (number): Maximum number of items to collect per source (default: 20)
• outputFormat ("json" | "markdown"): Output format (default: "json")
• includeCodeExamples (boolean): Whether to include code examples (default: true)
• subscribe (boolean): Enable daily automatic runs with the same query (default: false)
• subscriptionTimeUtc (string): Time of day (UTC) when you want to receive daily updates, e.g. "06:30" (default: "06:30")
• subscriptionName (string): Optional custom name for this subscription. If not provided, a stable name is derived from topics + format (default: "")
• enableLLMSearch (boolean): Enable LLM-powered internet search to find relevant content across the web (default: false)
• llmSearchProvider ("tavily" | "perplexity" | "serper"): Choose the search provider for LLM search (default: "tavily")
• tavilyApiKey (string, secret): Your Tavily API key. Only required if enableLLMSearch is true and llmSearchProvider is "tavily"
• perplexityApiKey (string, secret): Your Perplexity API key. Only required if enableLLMSearch is true and llmSearchProvider is "perplexity"
• serperApiKey (string, secret): Your Serper API key. Only required if enableLLMSearch is true and llmSearchProvider is "serper"
• githubToken (string, secret, optional): Optional GitHub personal access token for higher API rate limits (60/hour → 5000/hour)

🔔 Daily Subscriptions

The Actor supports automatic daily runs through Apify Schedules. When you enable subscriptions:

1. Run once with subscription enabled - Set subscribe: true in your input
2. Actor executes normally - Fetches sources, generates insights, delivers output
3. Schedule is created/updated - After completion, creates or updates an Apify Schedule
4. Daily automatic runs - The schedule runs the actor daily at your specified UTC time
5. Same query, fresh data - Each scheduled run uses the same input parameters (except subscribe is set to false to prevent recursion)

Subscription Features

• Stable schedule names - Re-running with the same subscription updates the existing schedule instead of creating duplicates
• Custom scheduling - Set your preferred UTC time for daily updates
• Automatic delivery - Output is delivered via your configured channels (email/Slack/Discord/webhook/etc.) on each scheduled run
• No recursion - Scheduled runs automatically have subscribe: false to prevent creating schedules of schedules

Example Subscription Input

{
  "topics": ["agentic ai", "llm frameworks"],
  "maxItems": 20,
  "outputFormat": "json",
  "includeCodeExamples": true,
  "subscribe": true,
  "subscriptionTimeUtc": "06:30",
  "subscriptionName": "My Daily AI Updates"
}

This will:

• Run immediately with the provided query
• Create/update a schedule named "My Daily AI Updates"
• Run automatically every day at 06:30 UTC
• Deliver fresh insights via your configured output channels

🔍 LLM-Powered Internet Search

The Actor can use AI-powered search APIs to find relevant content across the internet. This feature is optional and requires API keys.

Supported Search Providers

1. Tavily (Recommended) - Designed specifically for LLM agents, provides high-quality search results
2. Perplexity - AI-powered search with real-time web access
3. Serper - Google Search API with fast response times

Setup Instructions

Step 1: Get Your API Key

Choose one provider and get your API key:

• Tavily (Recommended):
  • Sign up at tavily.com
  • Navigate to your dashboard to get your API key
  • Free tier available
• Perplexity:
  • Visit perplexity.ai/settings/api
  • Generate your API key
• Serper (Google Search API):
  • Sign up at serper.dev
  • Get your API key from the dashboard
  • Free tier: 2,500 searches/month

Step 2: Add API Key (Choose One Method)

Method 1: Direct Input (Easiest - Recommended)

You can enter API keys directly in the Actor input form when running the Actor:

1. When configuring the Actor run, you'll see secret input fields:

   • Tavily API Key - If using Tavily
   • Perplexity API Key - If using Perplexity
   • Serper API Key - If using Serper
   • GitHub Token - Optional, for higher rate limits

2. Simply paste your API key in the appropriate field

3. The field is automatically marked as Secret (encrypted and secure)

4. No additional setup needed!

Method 2: Apify Secrets (For Reusable Keys)

If you want to reuse the same API keys across multiple Actor runs:

1. Go to Apify Console → Settings → Secrets:

   • Direct link: https://console.apify.com/account/integrations/secrets
   • Or: Click your profile icon (top right) → Settings → Integrations → Secrets tab

2. Click Add Secret button

3. Add secrets with these names:

   • TAVILY_API_KEY - For Tavily
   • PERPLEXITY_API_KEY - For Perplexity
   • SERPER_API_KEY - For Serper
   • GITHUB_TOKEN - Optional, for GitHub

4. The Actor will automatically use these secrets if the input fields are empty

Priority Order:

1. Input form secret fields (highest priority)
2. Apify Secrets (fallback)
3. Environment variables (for local development)

Step 3: Enable in Actor Input

When running the Actor, set these input parameters:

{
  "topics": ["agentic ai", "llm frameworks"],
  "enableLLMSearch": true,
  "llmSearchProvider": "serper",
  "serperApiKey": "your-serper-api-key-here"
}

Or for Tavily:

{
  "topics": ["agentic ai", "llm frameworks"],
  "enableLLMSearch": true,
  "llmSearchProvider": "tavily",
  "tavilyApiKey": "your-tavily-api-key-here"
}

Note:

• Make sure llmSearchProvider matches the API key field you fill:
  • "serper" → fill serperApiKey
  • "tavily" → fill tavilyApiKey
  • "perplexity" → fill perplexityApiKey
• If you've set up Apify Secrets, you can leave the API key fields empty and the Actor will use the secrets automatically

Example with LLM Search

{
  "topics": ["langchain", "autonomous agents"],
  "maxItems": 15,
  "outputFormat": "json",
  "enableLLMSearch": true,
  "llmSearchProvider": "tavily"
}

This will search across all sources including the internet using AI-powered search to find the most relevant recent content.

🏗️ Project Structure

.
├── .actor/
│   ├── actor.json          # Actor configuration
│   ├── input_schema.json   # Input validation schema
│   └── output_schema.json  # Output schema
├── src/
│   ├── main.ts             # Main orchestrator
│   ├── types.ts            # TypeScript type definitions
│   ├── transformer.ts      # Data transformation logic
│   ├── fetchers/           # Source-specific fetchers
│   │   ├── index.ts
│   │   ├── fetchGitHub.ts
│   │   ├── fetchHuggingFace.ts
│   │   ├── fetchPapersWithCode.ts
│   │   ├── fetchDevTo.ts
│   │   ├── fetchMedium.ts
│   │   └── fetchLLMSearch.ts
│   ├── output/             # Output generators
│   │   └── generators.ts
│   └── utils/              # Utility functions
│       ├── fetch.ts
│       ├── logger.ts
│       └── subscription.ts # Subscription/schedule management
├── package.json
├── tsconfig.json
└── Dockerfile

🛠️ Development

Prerequisites

• Node.js >= 18.0.0
• npm or yarn
• Apify CLI (npm install -g apify-cli)

Local Development

1. Install dependencies:

$npm install

2. Build the project:

$npm run build

3. Run locally:

$apify run

Testing

The Actor can be tested locally with default input or by providing custom input in storage/key_value_stores/default/INPUT.json:

{
  "topics": ["agentic ai", "llm frameworks"],
  "maxItems": 10,
  "outputFormat": "json",
  "includeCodeExamples": true,
  "subscribe": false,
  "subscriptionTimeUtc": "06:30",
  "subscriptionName": ""
}

Testing with Subscription

To test the subscription feature:

{
  "topics": ["agentic ai"],
  "maxItems": 5,
  "outputFormat": "json",
  "includeCodeExamples": true,
  "subscribe": true,
  "subscriptionTimeUtc": "06:30",
  "subscriptionName": "Test Subscription"
}

This will create a schedule that runs daily at 06:30 UTC. You can view and manage schedules in the Apify Console.

📤 Output

JSON Format (Default)

• Data is pushed to the default dataset (one item per insight)
• A consolidated JSON file is saved to key-value store as output.json

Markdown Format

• A formatted Markdown file is saved to key-value store as output.md

🚢 Deployment

Deploy to Apify Platform

1. Log in to Apify:

$apify login

2. Deploy the Actor:

$apify push

The Actor will be available in your Apify Console.

🔧 Technical Details

• Language: TypeScript
• Runtime: Node.js 18+
• SDK: Apify SDK v3
• Client: Apify Client v2 (for schedule management)
• Module System: ES Modules
• Build: TypeScript compiler

📝 Code Style

• Fully typed with TypeScript
• Modular architecture with reusable components
• Error handling at every level
• Clean, readable code with comments
• Follows Apify best practices

🔄 Future Enhancements

• Additional data sources
• More sophisticated content extraction
• AI-powered summarization
• Custom filtering and ranking
• Webhook notifications
• Multiple subscription frequencies (weekly, monthly)
• Subscription management UI

📚 Resources

• Apify SDK Documentation
• Apify Platform Documentation
• Crawlee Documentation
• Apify Community Discord

📄 License

ISC