# Cross-Platform AI Trending Scraper (`constant_quadruped/cross-platform-ai-trending-scraper`) Actor

Scrapes trending AI repositories, tweets, and papers across GitHub, Twitter, and ArXiv.

- **URL**: https://apify.com/constant\_quadruped/cross-platform-ai-trending-scraper.md
- **Developed by:** [CQ](https://apify.com/constant_quadruped) (community)
- **Categories:** AI, Developer tools, News
- **Stats:** 2 total users, 1 monthly users, 86.2% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md):

```bash
npm install apify-client
```

In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md):

```bash
pip install apify-client
```

In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md):

````bash
# MacOS / Linux
curl -fsSL https://apify.com/install-cli.sh | bash
# Windows
irm https://apify.com/install-cli.ps1 | iex
```bash

In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md).

If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).


# README

## Cross-Platform AI Trending Scraper

Scrapes recent trending AI content from two public sources and filters it by your topics:

- **GitHub** – most-starred repositories created in the last 30 days that match your topics, via the public GitHub Search API.
- **ArXiv** – most recent papers in `cs.AI` / `cs.LG` / `cs.CL` matching your topics, via the public ArXiv Atom API.

Both sources are queried with **no API key required**. A GitHub token can be supplied
optionally to raise the GitHub rate limit, but the actor runs fine without one.

> **Twitter/X is not included.** There is no free, key-less Twitter trending API, so
> this actor does not claim to scrape Twitter. (An earlier version listed Twitter and
> returned fabricated data; that has been removed.)

### Input

| Field | Type | Default | Notes |
|-------|------|---------|-------|
| `topics` | string[] | `["AI","LLM","DeepSeek","Agent"]` | Terms to match in repo/paper name & text. |
| `maxItemsPerSource` | integer | `10` | 1–50 items per platform. |
| `githubToken` | string (secret) | _empty_ | Optional. Raises GitHub rate limit. |

### Output

Each dataset item:

```json
{
  "source": "GitHub",
  "title": "owner/repo",
  "description": "...",
  "url": "https://github.com/owner/repo",
  "stars": 1234,
  "language": "Python",
  "published": "2026-06-01",
  "scrapedAt": "2026-06-20T12:00:00.000Z"
}
````

ArXiv items use `authors` instead of `stars`/`language`. A run summary is also written
to the key-value store under `SUMMARY`.

### Reliability

- Each platform runs independently (`Promise.allSettled`). One source failing (network
  error, rate limit, downtime) does **not** abort the run; remaining sources still
  produce output and the failure is logged.
- Each source request is **attempted up to 4 times** (1 initial try + 3 retries) with a
  growing backoff (3s / 6s / 9s). Public APIs like GitHub and ArXiv intermittently
  rate-limit or block shared datacenter IPs, so a single attempt is flaky from the cloud
  even though it works from a home IP.
- Every request has a 15s timeout (`AbortController`) so a hung endpoint fails fast
  instead of stalling the run.
- A **4-minute wall-clock budget** caps all retries so the run always finishes inside
  Apify's 5-minute quality-test window.
- GitHub is queried with **exactly one request per run** (plus retries on failure) to
  stay well under the unauthenticated rate limit (10 requests/minute).
- If **every** source is temporarily unavailable, the run still finishes **successfully**
  with a single marker record (carrying a `note` explaining the temporary outage) rather
  than an empty dataset or a failed run.

### Usage example

Run with custom topics:

```json
{
  "topics": ["Rust", "WebAssembly"],
  "maxItemsPerSource": 5
}
```

This returns up to 5 GitHub repos and 5 ArXiv papers matching `Rust` or `WebAssembly`.
Leave `topics` empty (or omit it) to use the defaults.

### Limitations

- **Two sources only.** GitHub and ArXiv. No Twitter/X, Reddit, Hacker News, Product Hunt,
  blogs, or other platforms. There is no free, key-less Twitter API, so Twitter is
  deliberately excluded.
- **No AI / semantic analysis.** Filtering is plain keyword matching performed by GitHub's
  and ArXiv's own search APIs — your topic terms are OR'd into a single query per source.
  The actor does not rank, summarize, cluster, embed, or use any ML/LLM. "AI" in the name
  refers to the subject matter (AI repos and papers), not the method.
- **GitHub "trending" is an approximation.** Results are repos **created in the last 30 days**
  (fixed window, by creation date) whose **name or description** matches your topic terms,
  sorted by star count. This is not GitHub's official trending feed and it excludes older
  repos that are gaining stars now. Terms are matched literally, not semantically.
- **ArXiv "trending" = newest, not most-cited.** ArXiv results are the most recently
  submitted papers in `cs.AI` / `cs.LG` / `cs.CL` matching your terms, sorted by submission
  date. There is no citation or popularity signal — recency only. Other ArXiv categories are
  not searched.
- **Volume caps.** At most `maxItemsPerSource` items per source (hard maximum 50). One page
  of results per source — no pagination. GitHub is queried once per run to stay under the
  unauthenticated rate limit (10 requests/minute; 1000 results/query).
- **Shared-IP rate limits.** From Apify's shared datacenter IPs, GitHub and ArXiv
  intermittently return 403/429 or time out. The actor retries (up to 4 attempts each,
  3s/6s/9s backoff, 15s per-request timeout, 4-minute total budget) and never fails the run,
  but a given run can still return fewer items than requested. If **every** source is blocked
  — **or** no items match your topics — the run finishes cleanly with a single marker record
  (`source: "none"`, null `title`/`url`/`published`, plus a `note`) instead of an empty
  dataset. Re-run shortly, or supply a GitHub token / enable a proxy for higher reliability.
- **Field shape varies by source.** GitHub items include `stars` and `language`; ArXiv items
  include `authors` instead. Not every field is present on every record.
- **Minimal ArXiv parsing.** ArXiv Atom XML is parsed with lightweight, dependency-free
  regex targeting ArXiv's standard response format — not a general XML parser.
- **No history or dedup.** Each run is independent. There is no cross-run deduplication or
  persistence beyond the dataset and the per-run `SUMMARY` key-value record.
- **Blank-topic fallback.** If you supply `topics` but every entry is blank, the actor falls
  back to the default topics rather than failing the run.

# Actor input Schema

## `topics` (type: `array`):

Topic terms used to query each source. Repos/papers whose name or text match any term are returned.

## `maxItemsPerSource` (type: `integer`):

Maximum number of items to return from each platform (GitHub, ArXiv). Range 1-50.

## `githubToken` (type: `string`):

Optional GitHub personal access token. Raises the GitHub Search API rate limit. The actor works without it (unauthenticated, low volume). Leave blank to run token-less.

## Actor input object example

```json
{
  "topics": [
    "AI",
    "LLM",
    "DeepSeek",
    "Agent"
  ],
  "maxItemsPerSource": 10
}
```

# Actor output Schema

## `overview` (type: `string`):

Trending items from GitHub and ArXiv with source, title, description, URL, stars, language, authors and dates.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "topics": [
        "AI",
        "LLM",
        "DeepSeek",
        "Agent"
    ],
    "maxItemsPerSource": 10
};

// Run the Actor and wait for it to finish
const run = await client.actor("constant_quadruped/cross-platform-ai-trending-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "topics": [
        "AI",
        "LLM",
        "DeepSeek",
        "Agent",
    ],
    "maxItemsPerSource": 10,
}

# Run the Actor and wait for it to finish
run = client.actor("constant_quadruped/cross-platform-ai-trending-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "topics": [
    "AI",
    "LLM",
    "DeepSeek",
    "Agent"
  ],
  "maxItemsPerSource": 10
}' |
apify call constant_quadruped/cross-platform-ai-trending-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=constant_quadruped/cross-platform-ai-trending-scraper",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

```json
{
    "openapi": "3.0.1",
    "info": {
        "title": "Cross-Platform AI Trending Scraper",
        "description": "Scrapes trending AI repositories, tweets, and papers across GitHub, Twitter, and ArXiv.",
        "version": "2.2",
        "x-build-id": "ORQ9XcdvX91f4IG62"
    },
    "servers": [
        {
            "url": "https://api.apify.com/v2"
        }
    ],
    "paths": {
        "/acts/constant_quadruped~cross-platform-ai-trending-scraper/run-sync-get-dataset-items": {
            "post": {
                "operationId": "run-sync-get-dataset-items-constant_quadruped-cross-platform-ai-trending-scraper",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK"
                    }
                }
            }
        },
        "/acts/constant_quadruped~cross-platform-ai-trending-scraper/runs": {
            "post": {
                "operationId": "runs-sync-constant_quadruped-cross-platform-ai-trending-scraper",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor and returns information about the initiated run in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK",
                        "content": {
                            "application/json": {
                                "schema": {
                                    "$ref": "#/components/schemas/runsResponseSchema"
                                }
                            }
                        }
                    }
                }
            }
        },
        "/acts/constant_quadruped~cross-platform-ai-trending-scraper/run-sync": {
            "post": {
                "operationId": "run-sync-constant_quadruped-cross-platform-ai-trending-scraper",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK"
                    }
                }
            }
        }
    },
    "components": {
        "schemas": {
            "inputSchema": {
                "type": "object",
                "properties": {
                    "topics": {
                        "title": "Trending Topics to Search",
                        "type": "array",
                        "description": "Topic terms used to query each source. Repos/papers whose name or text match any term are returned.",
                        "default": [
                            "AI",
                            "LLM",
                            "DeepSeek",
                            "Agent"
                        ],
                        "items": {
                            "type": "string"
                        }
                    },
                    "maxItemsPerSource": {
                        "title": "Max items per source",
                        "minimum": 1,
                        "maximum": 50,
                        "type": "integer",
                        "description": "Maximum number of items to return from each platform (GitHub, ArXiv). Range 1-50.",
                        "default": 10
                    },
                    "githubToken": {
                        "title": "GitHub token (optional)",
                        "type": "string",
                        "description": "Optional GitHub personal access token. Raises the GitHub Search API rate limit. The actor works without it (unauthenticated, low volume). Leave blank to run token-less."
                    }
                }
            },
            "runsResponseSchema": {
                "type": "object",
                "properties": {
                    "data": {
                        "type": "object",
                        "properties": {
                            "id": {
                                "type": "string"
                            },
                            "actId": {
                                "type": "string"
                            },
                            "userId": {
                                "type": "string"
                            },
                            "startedAt": {
                                "type": "string",
                                "format": "date-time",
                                "example": "2025-01-08T00:00:00.000Z"
                            },
                            "finishedAt": {
                                "type": "string",
                                "format": "date-time",
                                "example": "2025-01-08T00:00:00.000Z"
                            },
                            "status": {
                                "type": "string",
                                "example": "READY"
                            },
                            "meta": {
                                "type": "object",
                                "properties": {
                                    "origin": {
                                        "type": "string",
                                        "example": "API"
                                    },
                                    "userAgent": {
                                        "type": "string"
                                    }
                                }
                            },
                            "stats": {
                                "type": "object",
                                "properties": {
                                    "inputBodyLen": {
                                        "type": "integer",
                                        "example": 2000
                                    },
                                    "rebootCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "restartCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "resurrectCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "computeUnits": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            },
                            "options": {
                                "type": "object",
                                "properties": {
                                    "build": {
                                        "type": "string",
                                        "example": "latest"
                                    },
                                    "timeoutSecs": {
                                        "type": "integer",
                                        "example": 300
                                    },
                                    "memoryMbytes": {
                                        "type": "integer",
                                        "example": 1024
                                    },
                                    "diskMbytes": {
                                        "type": "integer",
                                        "example": 2048
                                    }
                                }
                            },
                            "buildId": {
                                "type": "string"
                            },
                            "defaultKeyValueStoreId": {
                                "type": "string"
                            },
                            "defaultDatasetId": {
                                "type": "string"
                            },
                            "defaultRequestQueueId": {
                                "type": "string"
                            },
                            "buildNumber": {
                                "type": "string",
                                "example": "1.0.0"
                            },
                            "containerUrl": {
                                "type": "string"
                            },
                            "usage": {
                                "type": "object",
                                "properties": {
                                    "ACTOR_COMPUTE_UNITS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_WRITES": {
                                        "type": "integer",
                                        "example": 1
                                    },
                                    "KEY_VALUE_STORE_LISTS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_INTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_EXTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_SERPS": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            },
                            "usageTotalUsd": {
                                "type": "number",
                                "example": 0.00005
                            },
                            "usageUsd": {
                                "type": "object",
                                "properties": {
                                    "ACTOR_COMPUTE_UNITS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_WRITES": {
                                        "type": "number",
                                        "example": 0.00005
                                    },
                                    "KEY_VALUE_STORE_LISTS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_INTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_EXTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_SERPS": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}
```