# Lemmy Scraper · Communities, Posts, Comments & Instances (`reapx/lemmy-scraper`) Actor

Scrape Lemmy communities, posts, comments, and instance metadata across the Fediverse. Structured data export by instance domain, community name, and sort type.

- **URL**: https://apify.com/reapx/lemmy-scraper.md
- **Developed by:** [Tarek Etman](https://apify.com/reapx) (community)
- **Categories:** Social media, News
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.30 / 1,000 lemmy item returneds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

![reapX — public sources in, addressable records out](https://reapx.dev/reapx.gif)

## Lemmy Scraper · Communities, Posts, Comments & Instances

Scrape Lemmy communities, posts, discussion comments, and server instance metadata across the decentralized Fediverse. Extract structured JSON records indexed by instance domain host, community name, and listing sort types without requiring authentication or platform tokens.

> Maintained by **reapX**. Every row cites the Apify run that produced it — nothing is inferred, modelled or filled in, and a field absent from the source is absent from the row. The extracted archive for this source is browsable at [reapx.dev/data/lemmy-scraper/](https://reapx.dev/data/lemmy-scraper/) and mirrored as an open dataset on [Hugging Face](https://huggingface.co/datasets/reapxdev/lemmy-scraper) and [Kaggle](https://www.kaggle.com/datasets/reapxdev/lemmy-scraper). Questions: reapxdev@proton.me

***

### ⬇️ Input

The actor accepts structured JSON input options configuring the Lemmy extraction target, Fediverse instance host, listing scope, ranking sort order, and item volume limits.

| Input Parameter | Type | Required | Default / Prefill | Description |
|---|---|---|---|---|
| `mode` | String | Yes | `"posts"` | Extraction target mode: `posts` (post feeds), `communities` (community directory), `comments` (comment threads), or `instance` (server metadata). |
| `instanceHost` | String | Yes | `"lemmy.world"` | Primary Lemmy instance hostname (e.g. `lemmy.world`, `sh.itjust.works`, `beehaw.org`, `feddit.de`, `programming.dev`). |
| `instances` | Array | No | `[]` | List of target Lemmy instance hostnames for multi-instance batch extraction. |
| `type_` | String | Yes | `"All"` | Scope filter for feeds: `All` (federated cross-instance content), `Local` (instance-native only), or `Subscribed`. |
| `sort` | String | Yes | `"Active"` | Sorting algorithm: `Active`, `Hot`, `New`, `Old`, `TopDay`, `TopWeek`, `TopMonth`, `TopYear`, `TopAll`, `MostComments`, or `NewComments`. |
| `communityName` | String | No | `""` | Filter posts or comments by community technical name (e.g. `technology`, `asklemmy`, `linux`, `news`). |
| `maxItems` | Integer | Yes | `100` | Maximum total records to extract during the run. |
| `maxPages` | Integer | No | `5` | Maximum paginated API requests per instance or community query. |
| `requestsPerSecond` | Number | No | `2` | Maximum HTTP requests per second to avoid rate-limiting on volunteer nodes. |

#### Worked Input Example: Posts Feed

```json
{
  "mode": "posts",
  "instanceHost": "lemmy.world",
  "type_": "All",
  "sort": "Hot",
  "communityName": "technology",
  "maxItems": 100
}
```

#### Worked Input Example: Multi-Instance Batch Scrape

```json
{
  "mode": "posts",
  "instanceHost": "lemmy.world",
  "instances": [
    "lemmy.world",
    "sh.itjust.works",
    "beehaw.org",
    "programming.dev",
    "feddit.de"
  ],
  "type_": "All",
  "sort": "Active",
  "maxItems": 250
}
```

#### Worked Input Example: Community Directory

```json
{
  "mode": "communities",
  "instanceHost": "sh.itjust.works",
  "type_": "All",
  "sort": "Hot",
  "maxItems": 100
}
```

***

### ⬆️ Output

Every record is pushed directly to the run's default dataset. Each row contains standard camelCase fields with total coverage on the primary `domain` entity identifier.

#### Dataset Schema Fields

| Field Name | Type | Description | Example Value |
|---|---|---|---|
| `domain` | String | Lemmy server instance domain host serving as primary entity key. | `"lemmy.world"` |
| `communityName` | String | Short technical name of the community. | `"technology"` |
| `communityTitle` | String | Human-readable title of the community. | `"Technology & Computing"` |
| `postTitle` | String | Headline title of the post. | `"Open Source AI Models in 2026"` |
| `postId` | Integer | Numeric post ID on originating instance. | `50206401` |
| `postUrl` | String | External target link or ActivityPub URL. | `"https://lemmy.world/post/50206401"` |
| `apId` | String | Canonical ActivityPub URI. | `"https://lemmy.world/post/50206401"` |
| `body` | String | Markdown post body text. | `"Discussion on modern software..."` |
| `creatorUsername` | String | Username of post or comment author. | `"viking_hippie"` |
| `creatorActorId` | String | Canonical ActivityPub URI of creator. | `"https://lemmy.world/u/viking_hippie"` |
| `publishedAt` | String | ISO 8601 UTC timestamp of publication. | `"2026-08-02T13:17:47Z"` |
| `updatedAt` | String | ISO 8601 UTC timestamp of last update. | `"2026-08-03T10:15:00Z"` |
| `score` | Integer | Net vote score (upvotes minus downvotes). | `831` |
| `commentsCount` | Integer | Total comments count. | `98` |
| `upvotes` | Integer | Total upvotes count. | `855` |
| `downvotes` | Integer | Total downvotes count. | `24` |
| `subscribersCount` | Integer | Total community subscribers count. | `45200` |
| `postsCount` | Integer | Total posts count in community/instance. | `12500` |
| `usersActiveDay` | Integer | Active users in the past 24 hours. | `1420` |
| `usersActiveWeek` | Integer | Active users in past 7 days. | `6800` |
| `usersActiveMonth` | Integer | Active users in past 30 days. | `24500` |
| `commentContent` | String | Text content of comment. | `"Great analysis on Fediverse protocols."` |
| `commentId` | Integer | Numeric comment ID. | `1849201` |
| `instanceName` | String | Server instance title. | `"Lemmy.world"` |
| `instanceDescription` | String | Server description and rules summary. | `"Flagship Lemmy instance."` |
| `version` | String | Lemmy server backend version. | `"0.19.5"` |
| `nsfw` | Boolean | Not Safe For Work flag. | `false` |
| `isLocal` | Boolean | True if item originated on target instance. | `true` |
| `itemType` | String | Entity type (`post`, `community`, `comment`, `instance`). | `"post"` |

#### Sample Output Record (Post)

```json
{
  "domain": "lemmy.world",
  "communityName": "politicalmemes",
  "communityTitle": "Political Memes",
  "postTitle": "Discussion on open infrastructure",
  "postId": 50206401,
  "postUrl": "https://lemmy.world/post/50206401",
  "apId": "https://lemmy.world/post/50206401",
  "body": "Detailed discussion on open protocols...",
  "creatorUsername": "viking_hippie",
  "creatorActorId": "https://lemmy.world/u/viking_hippie",
  "publishedAt": "2026-08-02T13:17:47Z",
  "updatedAt": "2026-08-03T10:15:00Z",
  "score": 831,
  "commentsCount": 98,
  "upvotes": 855,
  "downvotes": 24,
  "nsfw": false,
  "isLocal": true,
  "itemType": "post"
}
```

***

### How it works

1. **Direct HTTP API Scraping**: Communicates directly with Lemmy's official open v3 REST API endpoints (`/api/v3/post/list`, `/api/v3/community/list`, `/api/v3/comment/list`, `/api/v3/site`). No headless browsers or browser automation required.
2. **Fediverse Domain Aggregation**: Extracts canonical ActivityPub domain hosts (`domain`) for every post, community, comment, and instance record, aggregating cross-federated data onto entity pages.
3. **Pay-Per-Event Billing**: Bills strictly per item returned (`lemmy-item-returned` at $0.002 per item) using `Actor.charge()` before pushing to the dataset. Empty or blocked runs cost nothing beyond the platform start fee.
4. **Adaptive Rate Limiting & Backoff**: Automatically handles rate limiting (HTTP 429) and server errors (HTTP 5xx) with exponential backoff delays, respecting volunteer-run Fediverse server capacity.
5. **Streaming Dataset Push**: Pushes items to the default dataset as they arrive rather than buffering, protecting data against network interruptions.

***

### ❓ FAQ

#### Does this scraper require a Lemmy account or API token?

No. All Lemmy v3 API endpoints queried by this actor are public read-only endpoints accessible over plain HTTP without authentication.

#### Which Lemmy instances are supported?

Any public Lemmy instance across the Fediverse is supported, including `lemmy.world`, `sh.itjust.works`, `beehaw.org`, `feddit.de`, `programming.dev`, `lemmy.ml`, `sopuli.xyz`, `lemmy.ca`, `lemmy.zip`, and `reddthat.com`.

#### How does domain aggregation work for federated posts?

Because Lemmy is federated, a post published on `lemmy.dbzer0.com` may appear in the feed of `lemmy.world`. This scraper parses the canonical ActivityPub host URL so that `domain` accurately reflects the originating instance node.

#### How are costs controlled?

You can set `ACTOR_MAX_TOTAL_CHARGE_USD` to cap the maximum cost of any run. If the limit is reached, the scraper gracefully halts and delivers all items collected up to that point.

***

### 💬 Your feedback

Have suggestions for new Lemmy features, schema enhancements, or bugs to report? Contact the maintainers directly at `reapxdev@proton.me` or submit feedback via Apify Console.

***

*Unofficial - not affiliated with Lemmy or any Lemmy instance. Collects public data only. reapx. Contact reapxdev@proton.me.*

### 🧪 Example input

A real, runnable configuration — this is an actual input this Actor has run with.

```json
{
  "instanceHost": "lemmy.world",
  "mode": "posts",
  "type_": "All",
  "sort": "Hot",
  "maxItems": 100
}
```

### 📄 Sample output

One real row from a real run of this Actor, unedited.

```json
{
  "domain": "lemmy.dbzer0.com",
  "communityName": "nottheonion",
  "communityTitle": "Not The Onion",
  "postTitle": "Walmart self-checkout mistake destroys Olympic athlete's career",
  "postId": 50258953,
  "postUrl": "https://boingboing.net/2024/09/25/walmart-self-checkout-mistake-destroys-olympic-athletes-career.html",
  "apId": "https://lemmy.dbzer0.com/post/73364946",
  "body": "",
  "creatorUsername": "irelephant",
  "creatorActorId": "https://lemmy.dbzer0.com/u/irelephant",
  "publishedAt": "2026-08-03T17:34:42.299311Z",
  "updatedAt": "",
  "score": 36,
  "commentsCount": 13,
  "upvotes": 36,
  "downvotes": 0,
  "nsfw": false,
  "isLocal": false,
  "thumbnailUrl": "https://boingboing.net/wp-content/uploads/2024/09/pettipeice.jpg",
  "itemType": "post"
}
```

### ⚠️ Run outcomes and error handling

This Actor reports what happened in the run's **status message**, and it always keeps whatever it collected. These are the outcomes you can get and what each one means.

| Outcome | What it means |
|---|---|
| **Success** | Rows were returned and you were charged `lemmy-item-returned` at $0.002 per row. |
| **No matches** | The source returned nothing for your filters. **Nothing is charged.** Widen the date window or drop a filter. |

#### What is guaranteed either way

- **Every row is pushed as it is built**, not buffered to the end of the run. Anything that buffers output loses everything to a timeout, a block or a migration; this does not.
- **A field absent from the source is absent from the row.** Nothing is inferred, modelled or filled in to make a row look complete.

# Actor input Schema

## `mode` (type: `string`):

Select the Lemmy data extraction target mode: <code>posts</code> (retrieve post feeds), <code>communities</code> (scrape community directory), <code>comments</code> (fetch discussion comments), or <code>instance</code> (retrieve server stats).<br><br><b>Consequence:</b> Determines the primary entity type returned and output schema shape.

## `instanceHost` (type: `string`):

The domain host of the primary Lemmy server instance to target (e.g. <code>lemmy.world</code>, <code>sh.itjust.works</code>, <code>beehaw.org</code>, <code>feddit.de</code>, <code>programming.dev</code>).<br><br><b>Consequence:</b> Specifies the entry node for Fediverse API queries. Malformed hosts cause connection failures.

## `instances` (type: `array`):

List of Lemmy instance hostnames to scrape sequentially in a multi-instance batch run.<br><br><b>Empty value:</b> Leave this field empty to query only the primary domain host specified in <code>instanceHost</code>.<br><br><b>Consequence:</b> Adding multiple instances increases total returned items and domain coverage.

## `type_` (type: `string`):

Filter scope for Lemmy listing feeds: <code>All</code> (federated posts and comments from all connected Fediverse servers), <code>Local</code> (only content created on this instance), or <code>Subscribed</code>.<br><br><b>Consequence:</b> <code>All</code> maximizes cross-instance domain variety across the network.

## `sort` (type: `string`):

Sort order for posts, communities, or comments: <code>Active</code>, <code>Hot</code>, <code>New</code>, <code>Old</code>, <code>TopDay</code>, <code>TopWeek</code>, <code>TopMonth</code>, <code>TopYear</code>, <code>TopAll</code>, <code>MostComments</code>, or <code>NewComments</code>.<br><br><b>Consequence:</b> Controls ranking algorithm for item retrieval.

## `communityName` (type: `string`):

Filter posts or comments by a specific Lemmy community name (e.g. <code>technology</code>, <code>asklemmy</code>, <code>news</code>, <code>linux</code>, <code>memes</code>).<br><br><b>Empty value:</b> Leave this field empty to fetch content across all communities on the target instance.

## `maxItems` (type: `integer`):

Maximum total number of Lemmy posts, communities, comments, or instance records to extract during this run.<br><br><b>Consequence:</b> The higher the limit, the longer the run takes and the more pay-per-event records are billed.<br><br>The higher the number, the longer the run takes and the more you are charged, because you pay per row returned.

## `maxPages` (type: `integer`):

Maximum number of paginated HTTP API requests per instance or community query.<br><br><b>Consequence:</b> Higher page limits retrieve deeper historical feeds but increase execution duration.<br><br>The higher the number, the longer the run takes and the more you are charged, because you pay per row returned.

## `requestsPerSecond` (type: `number`):

Maximum HTTP API requests per second issued to Lemmy servers.<br><br><b>Consequence:</b> Setting a reasonable rate limit prevents HTTP 429 rate limit errors from volunteer-hosted Fediverse nodes.

## Actor input object example

```json
{
  "mode": "posts",
  "instanceHost": "lemmy.world",
  "instances": [],
  "type_": "All",
  "sort": "Active",
  "maxItems": 100,
  "maxPages": 5,
  "requestsPerSecond": 2
}
```

# Actor output Schema

## `results` (type: `string`):

Every Lemmy community, post, comment or instance record extracted by this run, written to the default dataset.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "mode": "posts",
    "instanceHost": "lemmy.world",
    "instances": [],
    "type_": "All",
    "sort": "Active",
    "communityName": "",
    "maxItems": 100,
    "maxPages": 5,
    "requestsPerSecond": 2
};

// Run the Actor and wait for it to finish
const run = await client.actor("reapx/lemmy-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "mode": "posts",
    "instanceHost": "lemmy.world",
    "instances": [],
    "type_": "All",
    "sort": "Active",
    "communityName": "",
    "maxItems": 100,
    "maxPages": 5,
    "requestsPerSecond": 2,
}

# Run the Actor and wait for it to finish
run = client.actor("reapx/lemmy-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "mode": "posts",
  "instanceHost": "lemmy.world",
  "instances": [],
  "type_": "All",
  "sort": "Active",
  "communityName": "",
  "maxItems": 100,
  "maxPages": 5,
  "requestsPerSecond": 2
}' |
apify call reapx/lemmy-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,reapx/lemmy-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/ShfvhQW2rHGp7qyTv/builds/Mpi8LQ1XQ5VUqinOb/openapi.json