# Bluesky Scraper - Fast & Monitoring Friendly (`iwc/bluesky-scraper`) Actor

Scrapes Bluesky posts, author feeds, search results, and reply threads via Bluesky's public AppView API. No login, API key, or app password required. Clean dedup keys and ISO timestamps make it ready for scheduled, diffable monitoring.

- **URL**: https://apify.com/iwc/bluesky-scraper.md
- **Developed by:** [Imran](https://apify.com/iwc) (community)
- **Categories:** Social media, Automation, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 post scrapeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Bluesky Scraper — Fast & Monitoring-Friendly

Scrape Bluesky posts, author feeds, search results, and reply threads via Bluesky's
**public AppView API**. **No login, API key, app password, or credentials of any kind are
required** — this works out of the box the moment you hit Run. Built for **speed,
reliability, and monitoring**: every result carries a stable dedup key and an ISO
timestamp, so you can schedule this actor and diff each run's output to catch new posts or
replies.

### What it does

- Scrapes Bluesky search results (`sort: latest` or `top`)
- Scrapes a specific account's post feed (`authors`)
- Scrapes a single post plus its reply thread from a direct link
- Optionally follows discovered posts to pull their replies too
- Talks directly to `https://api.bsky.app` — Bluesky's read-only AppView, which
  needs no authentication for these endpoints — with a descriptive User-Agent, retrying
  429s automatically
- Pushes clean, flat JSON records to the dataset — ready for Excel/Sheets, a database, or a
  diffing script

### No setup required

Unlike most social scrapers, there is nothing to configure before your first run: no
account to create, no app password to generate, no API key to paste in. Bluesky's AppView
(`api.bsky.app`) serves posts, feeds, search, and threads to anyone, unauthenticated
— that's the whole reason this actor is fast to start and simple to operate. Just fill in
what you want to scrape and click Run.

### Input

At least one of `searchQueries`, `authors`, or `startUrls` is required.

| Field | Type | Default | Notes |
|---|---|---|---|
| `searchQueries` | array of strings | `[]` | Keyword searches. If `authors` is also set, each query is searched **scoped to** each author |
| `authors` | array of strings | `[]` | Bluesky handles or DIDs whose feed to scrape, e.g. `"bsky.app"` |
| `startUrls` | array | `[]` | Direct bsky.app URLs: a profile (`https://bsky.app/profile/{handle}`) or a single post (`https://bsky.app/profile/{handle}/post/{rkey}`) |
| `sort` | enum | `latest` | `latest` or `top` — only applies to search; author feeds are always newest-first |
| `maxItems` | integer | `100` | Hard cap on posts scraped this run — your main billing guardrail |
| `includeReplies` | boolean | `false` | Master switch for reply scraping — must be `true` for any reply-thread requests to be made |
| `maxRepliesPerPost` | integer | `50` | Cap on replies per post (keeps billing predictable); `0` = no cap (every reply in that post's thread) |
| `scrapeType` | enum | `posts` | `posts`, `replies`, or `both` — controls what gets **pushed**; replies are still gated by `includeReplies` |
| `proxyConfiguration` | object | `{ "useApifyProxy": true }` | Plain datacenter proxy is fine — the public AppView doesn't IP-block like Reddit does, so residential is not required |

#### Example: monitor a keyword for new posts

```json
{
    "searchQueries": ["nvidia earnings"],
    "sort": "latest",
    "maxItems": 50,
    "scrapeType": "posts"
}
```

#### Example: scrape an account's feed

```json
{
    "authors": ["bsky.app"],
    "maxItems": 25,
    "scrapeType": "posts"
}
```

#### Example: posts + replies for a keyword

```json
{
    "searchQueries": ["apify"],
    "maxItems": 20,
    "scrapeType": "both",
    "includeReplies": true,
    "maxRepliesPerPost": 50
}
```

#### Example: a single post's reply thread

```json
{
    "startUrls": [{ "url": "https://bsky.app/profile/bsky.app/post/3juj4vqfjyk2m" }],
    "scrapeType": "both",
    "includeReplies": true
}
```

### Output

Results are pushed to the actor's default dataset — one JSON object per row.

#### Post record

| Field | Type | Description |
|---|---|---|
| `uri` | string | The post's `at://` URI (globally unique, permanent) |
| `cid` | string | Content hash (CID) of the post record |
| `url` | string | Clickable `https://bsky.app/profile/{handle}/post/{rkey}` permalink |
| `authorHandle` | string | Author's handle, e.g. `alice.bsky.social` |
| `authorDid` | string | Author's DID (permanent identifier) |
| `authorDisplayName` | string | null | Author's display name |
| `text` | string | Post text |
| `createdAt` | string | ISO 8601 — client-declared post creation time (`record.createdAt`) |
| `indexedAt` | string | ISO 8601 — when the AppView indexed the post |
| `likeCount` | number | Like count |
| `repostCount` | number | Repost count |
| `replyCount` | number | Reply count |
| `quoteCount` | number | Quote-post count |
| `langs` | array of strings | Declared post languages |
| `hasMedia` | boolean | Whether the post has an embed (image/video/link/quote) |
| `embedType` | string | null | Embed `$type`, e.g. `app.bsky.embed.images#view` |
| `dedupKey` | string | **Stable dedup key** — the post's `at://` URI |

#### Reply record (when `includeReplies` is on)

Same shape as a post record, plus:

| Field | Type | Description |
|---|---|---|
| `parentUri` | string | null | `at://` URI of the immediate parent post |
| `rootUri` | string | null | `at://` URI of the thread's root post |
| `depth` | number | Nesting depth relative to the post whose thread was fetched (0 = direct reply) |

### Use it for monitoring: schedule + dedupKey

This actor is built to be run on a **schedule** (e.g. every 15–30 minutes with
`sort: "latest"`) and diffed against the previous run:

1. Schedule the actor with the same input each time.
2. Compare each run's dataset against the last one using `dedupKey` (the post's `at://`
   URI) as the join key — any key that's new since the last run is a new post/reply.
3. Because `createdAt`/`indexedAt` are proper ISO timestamps, you can sort/filter
   chronologically without any conversion step, and feed the diff straight into an
   alerting workflow (email, Slack, webhook — bring your own).

`maxItems` is intentionally the primary billing guardrail: it caps posts per run regardless
of how many results the search or feed would otherwise return, so scheduled runs have a
predictable cost ceiling.

### Pricing (pay-per-event)

This actor uses Apify's pay-per-event pricing — no per-compute-unit charges. You're billed:

- **`post-scraped`** — once per post successfully scraped
- **`reply-scraped`** — once per reply successfully scraped (only when `includeReplies`
  is on)

Nothing is charged until an item is actually pushed to the dataset, and charging always
respects `maxItems` / `maxRepliesPerPost` — see `.actor/pay_per_event.json` for current
per-event prices.

### Local development

```bash
npm install
npm start
```

A sample input is provided at `storage/key_value_stores/default/INPUT.json` (searches
"apify", 10 posts, latest sort). No credentials or environment variables are needed —
Bluesky's public AppView is open, but it returns 403 to the `searchPosts` endpoint from cloud/datacenter egress IPs (observed from Apify's own IPs, 2026-08-30). Apify Proxy (datacenter group) is therefore ON by default and required for search; it also gives consistent egress
in production, but light local testing works without it too.

### Notes & known limitations (v1)

- Bluesky's AppView applies its own rate limits; the crawler caps concurrency at 4 and
  retries 429s automatically via Crawlee's session/proxy rotation rather than a
  hand-rolled rate limiter.
- Author feeds skip pure reposts (Bluesky's `reasonRepost` items) since those posts are
  authored by a *different* account than the one you asked to scrape — only the account's
  own posts (including pinned ones) are pushed. Use `scrapeType: "posts"` with
  `searchQueries` + `authors` combined if you want author-scoped keyword matches instead.
- Reply threads are fetched with `depth=10` (deeper than the API's own default of 6);
  `maxRepliesPerPost` caps the flattened, pushed count — not the raw fetch depth — so
  very large/deep threads may need more than one level of nesting to fully exhaust the cap.
- This is a scraper, not an alerting service: it produces alert-friendly output (dedup keys
  - ISO timestamps) but does not itself send notifications. Wire its dataset into your own
    schedule + diff + notify pipeline.

# Actor input Schema

## `searchQueries` (type: `array`):

Keyword searches (app.bsky.feed.searchPosts). If Authors are also provided, each query is searched scoped to each author; otherwise it searches all of Bluesky.

## `authors` (type: `array`):

Bluesky handles or DIDs whose post feed to scrape (app.bsky.feed.getAuthorFeed), e.g. "bsky.app" or "did:plc:z72i7hdynmk6r22z27h6tvur".

## `startUrls` (type: `array`):

Direct bsky.app URLs to start from — a profile (https://bsky.app/profile/{handle}) or a single post (https://bsky.app/profile/{handle}/post/{rkey}).

## `sort` (type: `string`):

Sort order for search results. "latest" is the recommended default for monitoring/alerting use cases since it surfaces the freshest posts first. Only applies to Search queries (author feeds are always returned newest-first by Bluesky).

## `maxItems` (type: `integer`):

Hard cap on the number of posts scraped in this run. This is the main billing guardrail — the run stops once it's reached.

## `includeReplies` (type: `boolean`):

Master switch for reply scraping. When off, no reply-thread requests are made and no reply-scraped events are charged, regardless of Scrape type.

## `maxRepliesPerPost` (type: `integer`):

Cap on replies scraped per post (only used when Include replies is on). Defaults to 50 to keep billing predictable. Set to 0 to remove the cap and scrape every reply in the thread — note that popular posts can have thousands of replies (and reply charges).

## `scrapeType` (type: `string`):

What to push to the output dataset: post records only, reply records only, or both. Replies are only fetched at all if "Include replies" is also on.

## `proxyConfiguration` (type: `object`):

Apify Proxy is ON by default (plain datacenter group) and is required: Bluesky blocks the searchPosts endpoint from cloud/datacenter egress IPs (403) unless requests go through a proxy. Author feeds and post threads work either way. Residential proxy is not needed.

## Actor input object example

```json
{
  "searchQueries": [
    "apify"
  ],
  "authors": [],
  "startUrls": [],
  "sort": "latest",
  "maxItems": 100,
  "includeReplies": false,
  "maxRepliesPerPost": 50,
  "scrapeType": "posts",
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `posts` (type: `string`):

No description

## `postsCsv` (type: `string`):

No description

## `run` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "searchQueries": [
        "apify"
    ],
    "authors": [],
    "startUrls": [],
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("iwc/bluesky-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "searchQueries": ["apify"],
    "authors": [],
    "startUrls": [],
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("iwc/bluesky-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "searchQueries": [
    "apify"
  ],
  "authors": [],
  "startUrls": [],
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call iwc/bluesky-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,iwc/bluesky-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/BTaeUA42ITIdnUz0c/builds/4HRua2uOyIczctmPj/openapi.json