Bluesky Scraper (AT Protocol)

Search and scrape posts and profiles from Bluesky via the public AT Protocol API. No authentication, no API key, no rate-limit headaches — the bsky.social public AppView serves search and read endpoints for free.

Built as an Apify Actor with pay-per-result pricing: you pay a tiny flat fee per post or profile returned in the dataset, and nothing for empty results.

---

Features

Mode	Endpoint	What it does
posts	app.bsky.feed.searchPosts	Full-text search of public posts
profiles	app.bsky.actor.searchActors	Search actor/profile by name/handle
search	both above	Run posts + profiles searches together
authorFeed	app.bsky.feed.getAuthorFeed	Fetch recent posts from one or more authors
profile	app.bsky.actor.getProfile	Fetch a single profile by handle or DID

Plus filters for date range (since/until), language (lang), images/video only, author DID, linked domain, and sort order (top vs. latest).

---

Input

The actor reads its input from the standard Apify input (.actor/input_schema.json). Key fields:

{
    "mode": "posts",                 // posts | profiles | search | authorFeed | profile
    "queries": ["apify", "scraping"], // for posts/profiles/search
    "authors": ["apify.bsky.social"], // for authorFeed/profile (handle or did:plc:...)
    "maxResults": 100,                // per query/author
    "since": "2024-01-01",            // ISO 8601, posts/search only
    "until": "",
    "lang": "en",
    "hasImages": false,
    "hasVideo": false,
    "authorFilter": "",               // did:plc:... to restrict posts to one author
    "domainFilter": "",               // only posts linking to this domain
    "sort": "top"                     // top | latest
}

See .actor/input_schema.json for the full, validated schema.

---

Output

Each row in the dataset is a flattened Bluesky post or profile with a consistent schema. Posts and profiles share the same column set (profile-only fields are empty on post rows, and vice-versa) so you can mix them in one table.

Selected fields:

• type — post | profile | feedPost
• query / authorHandle — what produced this row
• uri, cid — AT Protocol identifiers
• authorDid, authorHandle, authorDisplayName, authorAvatar
• text — post text / profile description
• indexedAt, createdAt — timestamps
• likeCount, repostCount, replyCount, quoteCount, bookmarkCount
• langs, hasImages, hasVideo, imageCount, imageAlts, imageUrls
• urls — links extracted from post facets/embeds
• replyParentUri, replyRootUri — for threading
• labels — content labels
• followersCount, followsCount, postsCount, viewer, joinedAt — profile-only

---

Pricing

This actor uses Apify's pay-per-event model. A single event type, result, is charged once per item pushed to the dataset. You only pay for data you actually receive.

The events block in .actor/actor.json declares the result event; the SDK's Actor.push_data(rows, 'result') handles both charging and dataset writes, and automatically stops charging once the user's budget is exhausted.

---

Local development

Prerequisites

• Python 3.13+
• uv or pip + venv

Install

cd actors/bluesky-scraper
uv venv && source .venv/bin/activate
uv pip install -r requirements.txt

Run locally with Apify SDK

The Apify Python SDK reads input from a local storage. Set ACTOR_TEST_PAY_PER_EVENT=true to exercise the PPE charging path locally (events default to $1 each in local mode):

export ACTOR_TEST_PAY_PER_EVENT=true
apify run --input '{"mode":"posts","queries":["AI"],"maxResults":5}'

Run the scraper logic standalone (no Apify platform)

For a quick smoke test of just the Bluesky API calls, you can import the client and shape helpers directly:

import asyncio, httpx
from src.main import BlueskyClient, shape_post

async def smoke():
    async with httpx.AsyncClient() as http:
        c = BlueskyClient(http)
        data = await c.search_posts("AI", limit=3)
        for p in data["posts"]:
            row = shape_post(p, query="AI")
            print(row["authorHandle"], "->", row["text"][:60])

asyncio.run(smoke())

---

Deployment

cd actors/bluesky-scraper
apify push

After publishing, enable pay-per-event pricing in the Apify Console and set the price per result event. The actor already declares the event in actor.json, so the Console will detect it automatically.

---

API reference

• AT Protocol searchPosts
• AT Protocol searchActors
• AT Protocol getAuthorFeed
• AT Protocol getProfile
• Apify Python SDK — pay-per-event

---

Notes & limitations

• The Bluesky public API does not require authentication for any endpoint used here.
• searchPosts paginates with a cursor; the actor follows cursors up to maxResults.
• Rate limits are generous on the public AppView but not infinite; the actor uses a polite User-Agent and a 30 s timeout. For heavy loads, configure an Apify proxy in the input.
• The extendOutput input field is reserved for future use and currently ignored.