Lobsters Scraper — Tech Community Stories, Comments & Tags

Extract clean JSON from Lobste.rs, the invite-only technology community known for high-signal programming, security, infrastructure, and open-source discussions. This Actor uses Lobsters' public JSON endpoints, so it is fast, lightweight, and does not need a browser.

Why use it?

• Track developer conversations — monitor what experienced engineers are discussing before it spreads to larger networks.
• Build research datasets — collect story titles, URLs, scores, submitters, tags, timestamps, and comment counts.
• Find niche technical content — filter by tags such as programming, security, linux, rust, devops, or ai.
• Power RAG and trend pipelines — export structured items to datasets, webhooks, Google Sheets, vector DBs, or your own API.

Input examples

1. Hottest stories

{
  "mode": "hottest",
  "limit": 50
}

2. Newest submissions

{
  "mode": "newest",
  "limit": 30
}

3. Stories by tag

{
  "mode": "tag",
  "tag": "security",
  "limit": 40
}

4. Single story deep scrape

{
  "mode": "story",
  "storyId": "abc123"
}

5. Available tags

{
  "mode": "tags"
}

Input fields

Field	Type	Required	Default	Description
mode	string	Yes	hottest	One of hottest, newest, tag, story, tags.
limit	integer	No	30	Maximum number of stories to return, 1–100.
tag	string	For tag mode	—	Lobsters tag slug, for example programming, security, rust.
storyId	string	For story mode	—	Lobsters story short ID from the URL.

Output

Each dataset item is normalized JSON. Story records can include:

Field	Description
short_id	Lobsters story ID.
url	External link submitted to Lobsters.
comments_url	Lobsters discussion URL.
title	Story title.
description	Story text/description when available.
created_at	Submission timestamp.
score, upvotes, downvotes	Engagement signals exposed by the API.
comment_count	Number of comments.
tags	List of tag objects/labels.
submitter_user	Submitter metadata when present.

Example output item:

{
  "short_id": "abc123",
  "url": "https://example.com/article",
  "comments_url": "https://lobste.rs/s/abc123/example_article",
  "title": "Example technical article",
  "created_at": "2026-04-24T12:00:00.000Z",
  "score": 42,
  "comment_count": 18,
  "tags": ["programming", "security"],
  "submitter_user": "example_user"
}

Use cases

• Developer trend monitoring — daily scrape hottest/newest stories and alert on specific keywords.
• Security research — track security, privacy, and infrastructure discussions.
• Competitive intelligence — monitor how engineering communities react to new tools and releases.
• Content discovery — find high-quality links for newsletters, blogs, podcasts, or social posts.
• LLM/RAG ingestion — feed curated technical links and discussions into knowledge bases.

Integrations

The Actor works with standard Apify integrations:

• Webhooks for automatic downstream processing.
• Google Sheets / Make / Zapier / n8n for no-code workflows.
• Apify API, JavaScript SDK, and Python client for custom apps.
• Dataset export as JSON, CSV, Excel, XML, or RSS.

JavaScript SDK

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('Helpermara/lobsters-scraper').call({
  mode: 'tag',
  tag: 'security',
  limit: 50,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Python client

from apify_client import ApifyClient

client = ApifyClient('YOUR_APIFY_TOKEN')
run = client.actor('Helpermara/lobsters-scraper').call(run_input={
    'mode': 'hottest',
    'limit': 50,
})

items = client.dataset(run['defaultDatasetId']).list_items().items
print(items)

Notes and limits

• Uses Lobsters public JSON endpoints; no login or invite is required for public story data.
• Keep limits reasonable. The Actor is designed for targeted collection, not aggressive crawling.
• Some fields depend on what Lobsters returns for a specific endpoint and story.

FAQ

Does it scrape comments?
The Actor is focused on story, tag, and story metadata. story mode retrieves the public story JSON available from Lobsters, including discussion metadata when exposed by the endpoint.

Do I need a Lobsters account?
No. Public JSON endpoints are used.

Can I filter by multiple tags?
Run the Actor once per tag and merge datasets downstream, or schedule several runs with different inputs.

Is this browser-based?
No. It uses direct HTTP requests, which makes it faster and cheaper than browser automation.

Keywords

Lobsters scraper, lobste.rs API, tech news scraper, developer community scraper, programming links, security news, engineering trends, Apify Actor, JSON API scraper, RAG dataset, developer intelligence.

Changelog

• 2026-04-24 — Rebuilt Actor implementation around real Lobsters JSON endpoints and expanded Store documentation.