Athena Social Scraper

Scrape public posts from 8 social networks with one Actor and one output format: X (Twitter), Instagram, TikTok, Facebook, Reddit, LinkedIn, YouTube and Medium, without calling any other Apify Store Actor.

Give it a list of public URLs (profiles, pages, channels, subreddits, posts or videos). For each URL, Athena detects the platform and runs its own chain of scraping techniques: cheap HTTP requests first (page HTML, JSON-LD, RSS, oEmbed, syndication endpoints), then a headless Chromium browser through a proxy as the fallback. Every result is converted to one normalized post schema, the same one produced by the sibling Actor Athena Social Orchestrator.

Why use it

• One input, one output. The same input and output schema for all eight platforms.
• No Store Actors, no third-party bills. All scraping is done by this Actor. You pay only for its own compute and proxy traffic.
• HTTP first, browser last. A browser is launched only when an HTTP technique fails or a platform has none, and only if you allow it (useBrowserFallback).
• First technique with results wins. Once one technique returns items for a URL, the remaining ones are not run.
• Honest data. Unknown counts are null, never 0. Abbreviated counts such as "25K" are not turned into numbers. Dates are ISO 8601 or null, never epoch numbers. url always points to the post, never to a media file.
• Transparent runs. A RUN_SUMMARY record lists, for every URL, which techniques were tried, how long each took and why each one failed.

Supported platforms and technique chains

Techniques are tried in this order. Techniques ending in -browser render the page with Playwright (Chromium) and run only when useBrowserFallback is true.

Platform / URL	Techniques	Notes
YouTube video (watch?v=, youtu.be/, /shorts/, /live/)	native:youtube-html → native:youtube-browser	Exact views and likes, publish date. Comments come from the count shown in the comments panel (whether it includes replies is not verified) when it is an exact number (an abbreviated count such as "1.2K" gives null). Subscribers are null.
YouTube channel (@handle, /channel/, /c/, /user/)	native:youtube-html → native:youtube-browser	Recent videos from the /videos tab, then one mobile watch page per listed video (m.youtube.com, 3 at a time) for exact views, likes, publish date, comment count (as for a video) and description. A bot-walled watch page is retried once through the input proxy, and the rest of that channel then goes through the proxy (YouTube channel enrichment). A video whose watch page still fails keeps the list data (views, likes and date null). Live on build 0.1.6 (2026-09-24, Apify platform): the mobile watch pages were also bot-walled from the datacenter; the proxy retry filled exact views, likes, date and comment count for 3/3 videos in 3.9 s.
X tweet (/status/<id>)	native:x-syndication → native:x-browser	Text, date, likes, replies and retweets (when present). Views and followers are null from syndication; the browser fallback can read them.
X profile	native:x-browser	Expect failures (see Limitations).
LinkedIn company, person or post	native:linkedin-jsonld → native:linkedin-browser	Posts from the page's JSON-LD. Shares and views are null. A sign-in wall becomes an error.
Medium author, publication, tag or subdomain	native:medium-rss	Recent stories from the RSS feed. Claps and responses are null.
Medium article	native:medium-rss → native:medium-browser	Found in the author's feed when recent; otherwise the browser reads claps and responses.
Reddit subreddit, post, user or redd.it	native:reddit-rss → native:reddit-browser	The RSS feed has no score or comment count (null); the browser reads them.
TikTok video	native:tiktok-oembed → native:tiktok-browser	oEmbed gives title, author and thumbnail only (counts and date are null). The browser, which runs only if oEmbed fails, gives stats.
TikTok profile	native:tiktok-browser	Expect failures (see Limitations).
Instagram profile	native:instagram-api → native:instagram-browser	Logged out, there is no date, like count or comment count (null). The grid can include collaboration posts owned by other accounts.
Instagram post (/p/, /reel/)	native:instagram-browser	Not verified.
Facebook page or post	native:facebook-browser	A logged-out page usually embeds only its most recent post.

old.reddit.com URLs are rewritten to www.reddit.com. Any other host is reported as unsupported; Athena never guesses the platform. A supported host with an unsupported URL shape (for example youtube.com/feed/trending) is reported as failed with unsupported <platform> URL shape.

First platform run

One run on the Apify platform (build 0.1.2, 2026-09-24, Apify RESIDENTIAL proxy, maxItemsPerUrl: 3, concurrency: 3, 54.8 s in total). Values come from its RUN_SUMMARY; one run is a snapshot, not a success rate.

URL	Result	Attempts (latency)	Notes
https://x.com/nasa	failed	native:x-browser HTTP 403 (3.2 s)
https://www.instagram.com/nasa/	3 posts	native:instagram-api HTTP 401 (1.6 s) → native:instagram-browser OK (15.9 s)	createdAt, likes and comments null; included collaboration posts owned by other accounts (astro_jessica, nasawebb).
https://www.tiktok.com/@nasa	failed	native:tiktok-browser "page is navigating" error (28.6 s)	A retry for this error was added afterwards. Rerun on build 0.1.3: the read error is gone, but the page yielded no items found (10.9 s).
https://www.facebook.com/NASA	1 post	native:facebook-browser OK (8.6 s)	Likes, comments, shares and date present.
https://www.reddit.com/r/golang/	failed	native:reddit-rss HTTP 403 (0.8 s) → native:reddit-browser "Prove your humanity" challenge (8.0 s)
https://www.linkedin.com/company/google/	3 posts	native:linkedin-jsonld OK (5.0 s)	Likes present; comments and shares null.
https://www.youtube.com/@GoogleDevelopers	3 videos	native:youtube-html OK (0.35 s)	Views, likes and createdAt null on the channel listing (this build had no watch-page enrichment).
https://medium.com/@netflixtechblog	3 stories	native:medium-rss OK (0.8 s)	Full timestamp and author username.
https://example.com/not-social	unsupported	none

YouTube channel enrichment

Live result of build 0.1.5 (platform run bXo9lNzG1pcDa4FXQ, 2026-09-24, Apify datacenter IP): build 0.1.5 fetched each video's desktop watch page (www.youtube.com/watch?v=<id>&hl=en&ucbcb=1) without a proxy. All 3 watch pages answered LOGIN_REQUIRED ("Sign in to confirm you’re not a bot"), so 0 of 3 videos were enriched. The 3 videos were still returned, with views, likes, comments and createdAt set to null.

Current strategy (not yet verified live):

1. Each listed video's mobile watch page, https://m.youtube.com/watch?v=<id>&hl=en&ucbcb=1, is requested first, without a proxy, with generated iPhone Safari headers. The Store Actor streamers/youtube-scraper fetched this mobile URL successfully from Apify infrastructure with plain HTTP (its run log, 2026-09-24). A probe from a residential connection returned the full page (HTTP 200, about 0.9 MB, playability OK). Whether the mobile host avoids the bot wall from Apify datacenter IPs has not been verified.
2. If a direct request hits the bot wall ("not a bot") or HTTP 429, that video is retried once through the input proxy (proxyConfiguration, residential by default). From then on, the channel's remaining videos are requested through the proxy directly. Without a configured proxy there is no retry, and the video keeps its list data.
3. Other failures (another HTTP status, a page that does not parse, a page for a different video) are not retried. No item is dropped, and unknown values stay null.

A log line youtube watch-page enrichment reports, per channel, listed, enriched, failed, skipped (not started or cancelled at the wind-down), viaProxy (videos requested through the proxy), botWalled (direct requests that hit the wall or HTTP 429) and the first error. Direct video URLs are unchanged: a single desktop request without a proxy, then the browser fallback.

Input

Field	Type	Default	Description
urls	array of strings	required	1 to 100 public URLs.
maxItemsPerUrl	integer	20	Maximum posts per URL (1 to 100).
concurrency	integer	3	How many URLs are processed at the same time (1 to 10). Each URL uses at most one browser context at a time.
proxyConfiguration	object	{ "useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"] }	Proxy for the browser fallback, for the HTTP techniques that need it (native:linkedin-jsonld, native:reddit-rss, native:instagram-api) and for YouTube channel watch pages that hit the bot wall. { "useApifyProxy": false } disables the proxy.
useBrowserFallback	boolean	true	Render the page with Playwright through the proxy when the HTTP techniques fail or a platform has none. Set to false to use only the HTTP techniques; the skipped browser techniques are reported in RUN_SUMMARY.
perUrlTimeoutSecs	integer	90	Time budget of each URL for its whole technique chain, in seconds (1 to 3600). It is capped to the run's remaining time minus 20 s. At 80% of the budget (but at least 3 s before its end), optional follow-up work (the YouTube channel watch-page enrichment) stops starting new requests and cancels the ones in flight, so the items already collected are kept. When the budget runs out, the current attempt is stopped, its browser context is closed and the rest of the chain is skipped.

Example:

{
    "urls": [
        "https://www.youtube.com/@GoogleDevelopers",
        "https://x.com/jack/status/20",
        "https://medium.com/@netflixtechblog",
        "https://www.reddit.com/r/golang/"
    ],
    "maxItemsPerUrl": 10
}

Output

Each dataset item is one post:

{
    "platform": "youtube",
    "inputUrl": "https://www.youtube.com/watch?v=abcdefghijk",
    "id": "abcdefghijk",
    "url": "https://www.youtube.com/watch?v=abcdefghijk",
    "text": "Video title\n\nVideo description...",
    "createdAt": "2026-09-22T23:00:10.000Z",
    "author": {
        "username": "ExampleChannel",
        "displayName": "Example Channel",
        "url": "https://www.youtube.com/@ExampleChannel",
        "followers": null,
        "verified": true
    },
    "stats": { "likes": 272, "comments": null, "shares": null, "views": 25343 },
    "media": ["https://i.ytimg.com/vi_webp/abcdefghijk/maxresdefault.webp"],
    "source": { "kind": "native", "name": "native:youtube-html" },
    "fetchedAt": "2026-09-24T02:40:51.123Z"
}

The values above illustrate the shape; they are not taken from a real run. source.name is the technique that produced the item.

The key-value store holds a RUN_SUMMARY record with totals and, for each URL:

• its platform and status (succeeded, failed or unsupported);
• the number of items and the latency;
• every attempt, with its technique, result, latency and error (for example HTTP 403 or per-URL timeout after 90 s);
• for a failed URL, the reason (for example run out of time, unsupported youtube URL shape or ...; browser fallback disabled (skipped: native:reddit-browser)).

The record is also written while the run is in progress, so an aborted run still leaves a partial summary. Failures are reported only there and never become dataset rows. One failing URL or platform never stops the run.

Cost

This Actor does not call other Actors, so there are no third-party charges. You pay for:

• Compute units for this Actor's own run (default memory: 2048 MB). In a local test run, each successful HTTP technique took about 1 s; a browser render can take up to about 45 s for navigation plus 8 s of settling, plus up to about 11 s if the page re-navigates while being read time per URL.
• Proxy traffic. The default proxy is Apify's residential proxy, which is billed per GB. It is used by the browser fallback, by the LinkedIn, Reddit and Instagram HTTP techniques and by bot-walled YouTube watch pages. To save bandwidth, the browser never downloads images, fonts, media or video streaming segments. See Apify's pricing page for current proxy prices.

A YouTube channel URL makes 1 request for the /videos tab plus 1 request per listed video (at most maxItemsPerUrl) for its watch page, without a proxy. When a watch page hits the bot wall, the channel's watch pages go through the proxy (the walled ones are requested twice, once directly and once through the proxy). Each proxied watch page costs about 0.9 MB of residential proxy traffic. This is an estimate from the uncompressed size of one probe; the traffic actually billed was not measured. For example, with maxItemsPerUrl: 20 and a bot wall on the first video, a channel can use about 18 MB of proxy traffic.

maxItemsPerUrl, perUrlTimeoutSecs and useBrowserFallback: false limit how much each URL can spend.

Limitations

• Little live testing so far. Besides manual checks from a residential connection and a local run without a proxy, the Actor has completed a single run on the Apify platform (First platform run). There, the techniques that by design use no proxy (native:youtube-html, native:medium-rss) succeeded; the other techniques are designed to use the input proxy (residential in that run). The browser fallback produced items on the platform (Instagram, Facebook); it has not been run locally.
• Some URL types are expected to fail or to return null fields, especially without a residential proxy:
  • X profiles: a logged-out profile page did not load any tweets during manual testing, and native:x-browser got HTTP 403 in the platform run. Single tweets work through the syndication endpoint.
  • Reddit: the RSS feed returned 403 and 429 in a local run and 403 in the platform run, where the browser was also shown a "Prove your humanity" challenge. User feeds and redd.it links were not verified.
  • TikTok profiles: the video list came back empty behind a captcha during manual testing. In the platform run the browser technique failed while reading the page, because the page was navigating again (Playwright's "page is navigating" error); the fetcher now waits and retries that read. On build 0.1.3 the read error no longer occurs, but the profile still returned no items found; the cause (for example a captcha) was not captured. Single videos return metadata from oEmbed, without stats.
  • Instagram: the HTTP API answered 401 require_login (HTTP 401 again in the platform run). The logged-out browser works on the platform and returns captions and images but no dates, likes or comments; a profile grid can include collaboration posts owned by other accounts. Single post URLs were not verified.
  • Facebook: a logged-out page usually yields only its most recent post (1 post, with likes, comments, shares and date, in the platform run); a login wall becomes an error. Post pages were not verified in a browser.
  • LinkedIn person profiles may hit a sign-in wall after a few requests.
  • Medium articles that are no longer in the author's feed need the browser.
  • YouTube channel enrichment: from an Apify datacenter IP, build 0.1.5 got LOGIN_REQUIRED on every desktop watch page (0 of 3 videos enriched). The mobile-first strategy with a proxy retry has not been run live yet (YouTube channel enrichment).
• Platforms change often. Page structures and endpoints change every few weeks, so techniques break and need maintenance. Each technique fails independently and the failure is reported in RUN_SUMMARY.
• Proxy initialization. On the Apify platform, the run fails if the requested proxy cannot be initialized. When run locally outside the platform, the Apify SDK only logs a warning and the run continues without a proxy.

Responsible use

• Athena never logs in: no login cookies are sent, and every browser render uses a fresh context with no cookies carried between URLs.
• It never impersonates search-engine or social-network crawlers (for example, it does not use Facebook's facebookexternalhit user agent).
• It collects only publicly visible data. All supported platforms restrict scraping in their terms of service. You are responsible for making sure that your use of this Actor and of the collected data complies with those terms and with the laws that apply to you, including data-protection laws.