# Athena Social Scraper (`ranil/athena-social-scraper`) Actor Detects the platform of each public URL (X/Twitter, Instagram, TikTok, Facebook, Reddit, LinkedIn, YouTube, Medium) and scrapes public data itself (HTTP first, Playwright + proxy fallback), without Store actors or logged-in sessions, emitting the shared normalized post output. - **URL**: https://apify.com/ranil/athena-social-scraper.md - **Developed by:** [Rani-L](https://apify.com/ranil) (community) - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing Pay per usage This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## Athena Social Scraper Scrape public posts from **8 social networks with one Actor and one output format**: X (Twitter), Instagram, TikTok, Facebook, Reddit, LinkedIn, YouTube and Medium, **without calling any other Apify Store Actor**. Give it a list of public URLs (profiles, pages, channels, subreddits, posts or videos). For each URL, Athena detects the platform and runs its own chain of scraping techniques: cheap HTTP requests first (page HTML, JSON-LD, RSS, oEmbed, syndication endpoints), then a headless Chromium browser through a proxy as the fallback. Every result is converted to **one normalized post schema**, the same one produced by the sibling Actor *Athena Social Orchestrator*. ### Why use it - **One input, one output.** The same input and output schema for all eight platforms. - **No Store Actors, no third-party bills.** All scraping is done by this Actor. You pay only for its own compute and proxy traffic. - **HTTP first, browser last.** A browser is launched only when an HTTP technique fails or a platform has none, and only if you allow it (`useBrowserFallback`). - **First technique with results wins.** Once one technique returns items for a URL, the remaining ones are not run. - **Honest data.** Unknown counts are `null`, never `0`. Abbreviated counts such as "25K" are not turned into numbers. Dates are ISO 8601 or `null`, never epoch numbers. `url` always points to the post, never to a media file. - **Transparent runs.** A `RUN_SUMMARY` record lists, for every URL, which techniques were tried, how long each took and why each one failed. ### Supported platforms and technique chains Techniques are tried in this order. Techniques ending in `-browser` render the page with Playwright (Chromium) and run only when `useBrowserFallback` is `true`. | Platform / URL | Techniques | Notes | | ------------------------------------------------------------- | ---------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | YouTube video (`watch?v=`, `youtu.be/`, `/shorts/`, `/live/`) | `native:youtube-html` → `native:youtube-browser` | Exact views and likes, publish date. Comments come from the count shown in the comments panel (whether it includes replies is not verified) when it is an exact number (an abbreviated count such as "1.2K" gives `null`). Subscribers are `null`. | | YouTube channel (`@handle`, `/channel/`, `/c/`, `/user/`) | `native:youtube-html` → `native:youtube-browser` | Recent videos from the `/videos` tab, then **one mobile watch page per listed video** (`m.youtube.com`, 3 at a time) for exact views, likes, publish date, comment count (as for a video) and description. A bot-walled watch page is retried once through the input proxy, and the rest of that channel then goes through the proxy ([YouTube channel enrichment](#youtube-channel-enrichment)). A video whose watch page still fails keeps the list data (views, likes and date `null`). Live on build 0.1.6 (2026-09-24, Apify platform): the mobile watch pages were also bot-walled from the datacenter; the proxy retry filled exact views, likes, date and comment count for 3/3 videos in 3.9 s. | | X tweet (`/status/`) | `native:x-syndication` → `native:x-browser` | Text, date, likes, replies and retweets (when present). Views and followers are `null` from syndication; the browser fallback can read them. | | X profile | `native:x-browser` | Expect failures (see Limitations). | | LinkedIn company, person or post | `native:linkedin-jsonld` → `native:linkedin-browser` | Posts from the page's JSON-LD. Shares and views are `null`. A sign-in wall becomes an error. | | Medium author, publication, tag or subdomain | `native:medium-rss` | Recent stories from the RSS feed. Claps and responses are `null`. | | Medium article | `native:medium-rss` → `native:medium-browser` | Found in the author's feed when recent; otherwise the browser reads claps and responses. | | Reddit subreddit, post, user or `redd.it` | `native:reddit-rss` → `native:reddit-browser` | The RSS feed has no score or comment count (`null`); the browser reads them. | | TikTok video | `native:tiktok-oembed` → `native:tiktok-browser` | oEmbed gives title, author and thumbnail only (counts and date are `null`). The browser, which runs only if oEmbed fails, gives stats. | | TikTok profile | `native:tiktok-browser` | Expect failures (see Limitations). | | Instagram profile | `native:instagram-api` → `native:instagram-browser` | Logged out, there is no date, like count or comment count (`null`). The grid can include collaboration posts owned by other accounts. | | Instagram post (`/p/`, `/reel/`) | `native:instagram-browser` | Not verified. | | Facebook page or post | `native:facebook-browser` | A logged-out page usually embeds only its most recent post. | `old.reddit.com` URLs are rewritten to `www.reddit.com`. Any other host is reported as `unsupported`; Athena never guesses the platform. A supported host with an unsupported URL shape (for example `youtube.com/feed/trending`) is reported as `failed` with `unsupported URL shape`. #### First platform run One run on the Apify platform (build 0.1.2, 2026-09-24, Apify `RESIDENTIAL` proxy, `maxItemsPerUrl: 3`, `concurrency: 3`, 54.8 s in total). Values come from its `RUN_SUMMARY`; one run is a snapshot, not a success rate. | URL | Result | Attempts (latency) | Notes | | ------------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------ | | `https://x.com/nasa` | failed | `native:x-browser` HTTP 403 (3.2 s) | | | `https://www.instagram.com/nasa/` | 3 posts | `native:instagram-api` HTTP 401 (1.6 s) → `native:instagram-browser` OK (15.9 s) | `createdAt`, likes and comments `null`; included collaboration posts owned by other accounts (`astro_jessica`, `nasawebb`). | | `https://www.tiktok.com/@nasa` | failed | `native:tiktok-browser` "page is navigating" error (28.6 s) | A retry for this error was added afterwards. Rerun on build 0.1.3: the read error is gone, but the page yielded `no items found` (10.9 s). | | `https://www.facebook.com/NASA` | 1 post | `native:facebook-browser` OK (8.6 s) | Likes, comments, shares and date present. | | `https://www.reddit.com/r/golang/` | failed | `native:reddit-rss` HTTP 403 (0.8 s) → `native:reddit-browser` "Prove your humanity" challenge (8.0 s) | | | `https://www.linkedin.com/company/google/` | 3 posts | `native:linkedin-jsonld` OK (5.0 s) | Likes present; comments and shares `null`. | | `https://www.youtube.com/@GoogleDevelopers` | 3 videos | `native:youtube-html` OK (0.35 s) | Views, likes and `createdAt` `null` on the channel listing (this build had no watch-page enrichment). | | `https://medium.com/@netflixtechblog` | 3 stories | `native:medium-rss` OK (0.8 s) | Full timestamp and author username. | | `https://example.com/not-social` | unsupported | none | | #### YouTube channel enrichment **Live result of build 0.1.5** (platform run `bXo9lNzG1pcDa4FXQ`, 2026-09-24, Apify datacenter IP): build 0.1.5 fetched each video's desktop watch page (`www.youtube.com/watch?v=&hl=en&ucbcb=1`) without a proxy. All 3 watch pages answered `LOGIN_REQUIRED` ("Sign in to confirm you’re not a bot"), so 0 of 3 videos were enriched. The 3 videos were still returned, with views, likes, comments and `createdAt` set to `null`. **Current strategy (not yet verified live):** 1. Each listed video's **mobile** watch page, `https://m.youtube.com/watch?v=&hl=en&ucbcb=1`, is requested first, without a proxy, with generated iPhone Safari headers. The Store Actor `streamers/youtube-scraper` fetched this mobile URL successfully from Apify infrastructure with plain HTTP (its run log, 2026-09-24). A probe from a residential connection returned the full page (HTTP 200, about 0.9 MB, playability `OK`). Whether the mobile host avoids the bot wall from Apify datacenter IPs has not been verified. 2. If a direct request hits the bot wall ("not a bot") or HTTP 429, that video is retried **once through the input proxy** (`proxyConfiguration`, residential by default). From then on, the channel's remaining videos are requested through the proxy directly. Without a configured proxy there is no retry, and the video keeps its list data. 3. Other failures (another HTTP status, a page that does not parse, a page for a different video) are not retried. No item is dropped, and unknown values stay `null`. A log line `youtube watch-page enrichment` reports, per channel, `listed`, `enriched`, `failed`, `skipped` (not started or cancelled at the wind-down), `viaProxy` (videos requested through the proxy), `botWalled` (direct requests that hit the wall or HTTP 429) and the first error. Direct video URLs are unchanged: a single desktop request without a proxy, then the browser fallback. ### Input | Field | Type | Default | Description | | -------------------- | ---------------- | ---------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `urls` | array of strings | required | 1 to 100 public URLs. | | `maxItemsPerUrl` | integer | 20 | Maximum posts per URL (1 to 100). | | `concurrency` | integer | 3 | How many URLs are processed at the same time (1 to 10). Each URL uses at most one browser context at a time. | | `proxyConfiguration` | object | `{ "useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"] }` | Proxy for the browser fallback, for the HTTP techniques that need it (`native:linkedin-jsonld`, `native:reddit-rss`, `native:instagram-api`) and for YouTube channel watch pages that hit the bot wall. `{ "useApifyProxy": false }` disables the proxy. | | `useBrowserFallback` | boolean | true | Render the page with Playwright through the proxy when the HTTP techniques fail or a platform has none. Set to `false` to use only the HTTP techniques; the skipped browser techniques are reported in `RUN_SUMMARY`. | | `perUrlTimeoutSecs` | integer | 90 | Time budget of each URL for its whole technique chain, in seconds (1 to 3600). It is capped to the run's remaining time minus 20 s. At 80% of the budget (but at least 3 s before its end), optional follow-up work (the YouTube channel watch-page enrichment) stops starting new requests and cancels the ones in flight, so the items already collected are kept. When the budget runs out, the current attempt is stopped, its browser context is closed and the rest of the chain is skipped. | Example: ```json { "urls": [ "https://www.youtube.com/@GoogleDevelopers", "https://x.com/jack/status/20", "https://medium.com/@netflixtechblog", "https://www.reddit.com/r/golang/" ], "maxItemsPerUrl": 10 } ``` ### Output Each dataset item is one post: ```json { "platform": "youtube", "inputUrl": "https://www.youtube.com/watch?v=abcdefghijk", "id": "abcdefghijk", "url": "https://www.youtube.com/watch?v=abcdefghijk", "text": "Video title\n\nVideo description...", "createdAt": "2026-09-22T23:00:10.000Z", "author": { "username": "ExampleChannel", "displayName": "Example Channel", "url": "https://www.youtube.com/@ExampleChannel", "followers": null, "verified": true }, "stats": { "likes": 272, "comments": null, "shares": null, "views": 25343 }, "media": ["https://i.ytimg.com/vi_webp/abcdefghijk/maxresdefault.webp"], "source": { "kind": "native", "name": "native:youtube-html" }, "fetchedAt": "2026-09-24T02:40:51.123Z" } ``` The values above illustrate the shape; they are not taken from a real run. `source.name` is the technique that produced the item. The key-value store holds a `RUN_SUMMARY` record with totals and, for each URL: - its platform and status (`succeeded`, `failed` or `unsupported`); - the number of items and the latency; - every attempt, with its technique, result, latency and error (for example `HTTP 403` or `per-URL timeout after 90 s`); - for a failed URL, the reason (for example `run out of time`, `unsupported youtube URL shape` or `...; browser fallback disabled (skipped: native:reddit-browser)`). The record is also written while the run is in progress, so an aborted run still leaves a partial summary. Failures are reported only there and never become dataset rows. One failing URL or platform never stops the run. ### Cost This Actor does not call other Actors, so there are no third-party charges. You pay for: - **Compute units** for this Actor's own run (default memory: 2048 MB). In a local test run, each successful HTTP technique took about 1 s; a browser render can take up to about 45 s for navigation plus 8 s of settling, plus up to about 11 s if the page re-navigates while being read time per URL. - **Proxy traffic.** The default proxy is Apify's residential proxy, which is billed per GB. It is used by the browser fallback, by the LinkedIn, Reddit and Instagram HTTP techniques and by bot-walled YouTube watch pages. To save bandwidth, the browser never downloads images, fonts, media or video streaming segments. See Apify's pricing page for current proxy prices. A YouTube channel URL makes 1 request for the `/videos` tab plus 1 request per listed video (at most `maxItemsPerUrl`) for its watch page, without a proxy. When a watch page hits the bot wall, the channel's watch pages go through the proxy (the walled ones are requested twice, once directly and once through the proxy). Each proxied watch page costs about 0.9 MB of residential proxy traffic. This is an estimate from the uncompressed size of one probe; the traffic actually billed was not measured. For example, with `maxItemsPerUrl: 20` and a bot wall on the first video, a channel can use about 18 MB of proxy traffic. `maxItemsPerUrl`, `perUrlTimeoutSecs` and `useBrowserFallback: false` limit how much each URL can spend. ### Limitations - **Little live testing so far.** Besides manual checks from a residential connection and a local run without a proxy, the Actor has completed a single run on the Apify platform ([First platform run](#first-platform-run)). There, the techniques that by design use no proxy (`native:youtube-html`, `native:medium-rss`) succeeded; the other techniques are designed to use the input proxy (residential in that run). The browser fallback produced items on the platform (Instagram, Facebook); it has not been run locally. - **Some URL types are expected to fail or to return `null` fields**, especially without a residential proxy: - **X profiles:** a logged-out profile page did not load any tweets during manual testing, and `native:x-browser` got HTTP 403 in the platform run. Single tweets work through the syndication endpoint. - **Reddit:** the RSS feed returned 403 and 429 in a local run and 403 in the platform run, where the browser was also shown a "Prove your humanity" challenge. User feeds and `redd.it` links were not verified. - **TikTok profiles:** the video list came back empty behind a captcha during manual testing. In the platform run the browser technique failed while reading the page, because the page was navigating again (Playwright's "page is navigating" error); the fetcher now waits and retries that read. On build 0.1.3 the read error no longer occurs, but the profile still returned `no items found`; the cause (for example a captcha) was not captured. Single videos return metadata from oEmbed, without stats. - **Instagram:** the HTTP API answered `401 require_login` (HTTP 401 again in the platform run). The logged-out browser works on the platform and returns captions and images but no dates, likes or comments; a profile grid can include collaboration posts owned by other accounts. Single post URLs were not verified. - **Facebook:** a logged-out page usually yields only its most recent post (1 post, with likes, comments, shares and date, in the platform run); a login wall becomes an error. Post pages were not verified in a browser. - **LinkedIn person profiles** may hit a sign-in wall after a few requests. - **Medium articles** that are no longer in the author's feed need the browser. - **YouTube channel enrichment:** from an Apify datacenter IP, build 0.1.5 got `LOGIN_REQUIRED` on every desktop watch page (0 of 3 videos enriched). The mobile-first strategy with a proxy retry has not been run live yet ([YouTube channel enrichment](#youtube-channel-enrichment)). - **Platforms change often.** Page structures and endpoints change every few weeks, so techniques break and need maintenance. Each technique fails independently and the failure is reported in `RUN_SUMMARY`. - **Proxy initialization.** On the Apify platform, the run fails if the requested proxy cannot be initialized. When run locally outside the platform, the Apify SDK only logs a warning and the run continues without a proxy. ### Responsible use - Athena **never logs in**: no login cookies are sent, and every browser render uses a fresh context with no cookies carried between URLs. - It **never impersonates search-engine or social-network crawlers** (for example, it does not use Facebook's `facebookexternalhit` user agent). - It collects only publicly visible data. All supported platforms restrict scraping in their terms of service. You are responsible for making sure that your use of this Actor and of the collected data complies with those terms and with the laws that apply to you, including data-protection laws. # Actor input Schema ## `urls` (type: `array`): Public URLs of profiles, channels, subreddits or posts (X/Twitter, Instagram, TikTok, Facebook, Reddit, LinkedIn, YouTube, Medium). ## `maxItemsPerUrl` (type: `integer`): Maximum number of items collected for each URL. ## `proxyConfiguration` (type: `object`): Proxy settings. Ignored by Athena Social Orchestrator (it calls Store actors, which use their own proxies). Athena Social Scraper uses it for its browser fallback and for the HTTP techniques that need a proxy (LinkedIn, Reddit, Instagram). ## `concurrency` (type: `integer`): Number of URLs processed in parallel. ## `useBrowserFallback` (type: `boolean`): When an HTTP technique fails (or a platform has none), render the page with Playwright through the proxy. Disable to measure the HTTP-only techniques. ## `perUrlTimeoutSecs` (type: `integer`): Time budget of each URL across its whole technique chain (HTTP and browser). Capped to the time left in the run. ## Actor input object example ```json { "urls": [ "https://medium.com/@netflixtechblog", "https://www.youtube.com/@GoogleDevelopers" ], "maxItemsPerUrl": 3, "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ] }, "concurrency": 3, "useBrowserFallback": true, "perUrlTimeoutSecs": 90 } ``` # Actor output Schema ## `posts` (type: `string`): Normalized posts: platform, inputUrl, id, url (post link), text, createdAt (ISO 8601 or null), author, stats (null when unknown, never 0 by default), media, source and fetchedAt. Failures never appear here; see Run summary. ## `runSummary` (type: `string`): RUN_SUMMARY record: totals and, for every input URL, its platform, status (succeeded, failed or unsupported), item count, latency and each native technique attempt (HTTP or browser) with its error. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "urls": [ "https://medium.com/@netflixtechblog", "https://www.youtube.com/@GoogleDevelopers" ], "maxItemsPerUrl": 3 }; // Run the Actor and wait for it to finish const run = await client.actor("ranil/athena-social-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "urls": [ "https://medium.com/@netflixtechblog", "https://www.youtube.com/@GoogleDevelopers", ], "maxItemsPerUrl": 3, } # Run the Actor and wait for it to finish run = client.actor("ranil/athena-social-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "urls": [ "https://medium.com/@netflixtechblog", "https://www.youtube.com/@GoogleDevelopers" ], "maxItemsPerUrl": 3 }' | apify call ranil/athena-social-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,ranil/athena-social-scraper" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/QnIpf0u4PdRVpd7cW/builds/SuGTw7uJbhenCYQU2/openapi.json