Lobsters Stories Scraper: Front Page, Tags & Scores

Pulls stories from the public JSON feeds behind lobste.rs. You get the title, the outbound URL, score, tag list, comment count, submitter and the comments permalink, from the front page, the newest queue, or any tag feed you name.

Lobsters is an invite-only technical community: a fraction of Hacker News' volume with a much higher hit rate per story. This Actor reads structured data directly, so there's no browser, no proxy, no login and no HTML parsing.

Agent skill: SKILL.md

https://api.apify.com/v2/key-value-stores/t7YoTxpZEJOWvw4Ug/records/lobsters-stories-scraper.md

What you get

Field	What it holds
feed	Which feed the story came from (hottest, newest, t/rust)
shortId	Lobsters' short identifier, the stable key for deduplication
title	Story title
url, isTextPost	The outbound link, or null with isTextPost: true for discussion-only posts
score, flags	Net score and flag count
commentCount	Comments at the time of the run
createdAt	Submission time, ISO 8601 with the site's US/Central offset
submitterUser, userIsAuthor	Who submitted it, and whether they wrote the linked thing
tags	Every tag on the story
shortIdUrl, commentsUrl	Short permalink and the full comments URL
description, descriptionPlain	Submitter's note as HTML and as plain text. Usually empty, which Lobsters reports as "" and this Actor stores as null
scrapedAt	Run timestamp

RUN_SUMMARY in the key-value store holds per-run counts, the filters you used, and any feed page that failed.

Use cases

• Engineering digest. Pull the front page every morning above a score threshold and post it into Slack.
• Niche topic tracking. Watch t/rust, t/nix or t/compilers for anything that clears 10 points.
• Hacker News complement. The same story often lands on both, and joining on the outbound URL shows which communities care about what.
• Release monitoring. The release tag is a clean firehose of version announcements for tools that matter.
• Newsletter sourcing. A scored, tagged, deduplicated candidate list without reading a single web page.

Quick start

The current front page, nothing else:

{
 "feeds": ["hottest"]
}

Two tag feeds plus the newest queue, only stories worth reading:

{
 "feeds": ["hottest", "newest", "t/rust,go"],
 "tags": ["rust", "go", "release"],
 "minScore": 5,
 "maxStories": 100,
 "maxPagesPerFeed": 4
}

A deep sweep of a quiet tag. The score filter throws most of it away, so page further:

{
 "feeds": ["t/compilers"],
 "minScore": 10,
 "maxStories": 200,
 "maxPagesPerFeed": 20
}

Input

Field	Type	Default	Notes
feeds	array	["hottest"]	hottest, newest, or a tag feed written as rust, t/rust, or rust,go for a combined feed. Full lobste.rs URLs are accepted and normalised.
tags	array	[]	Keep only stories carrying one of these tags. Applied after fetching, so it filters hottest and newest too. Empty means all.
minScore	integer	0	Drop anything scoring below this.
maxStories	integer	100	Total cap across all feeds. 0 means no limit, bounded only by maxPagesPerFeed.
maxPagesPerFeed	integer	4	Pages to walk per feed, 25 stories each. Max 100.

Nothing is required. Running with no input returns the current front page.

maxStories and maxPagesPerFeed bound the run from two directions, and whichever is hit first stops it. With strict tags or minScore filters, raise maxPagesPerFeed or the run finishes early with very little.

Output example

A real record from the hottest feed:

{
 "feed": "hottest",
 "shortId": "xx83dr",
 "title": "jujutsu 0.44.0",
 "url": "https://github.com/jj-vcs/jj/releases/tag/v0.44.0",
 "isTextPost": false,
 "score": 13,
 "flags": 0,
 "commentCount": 10,
 "createdAt": "2026-08-06T04:03:51.889-05:00",
 "submitterUser": "olex",
 "userIsAuthor": false,
 "tags": ["release", "vcs"],
 "shortIdUrl": "https://lobste.rs/s/xx83dr",
 "commentsUrl": "https://lobste.rs/s/xx83dr/jujutsu_0_44_0",
 "description": null,
 "descriptionPlain": null,
 "scrapedAt": "2026-08-06T11:43:10.687Z"
}

Where a submitter added a note, the two description fields carry it:

{
 "shortId": "kuopn0",
 "title": "Scripting, Programming Languages, Predictions",
 "description": "<p>by Roberto Ierusalimschy, creator of Lua</p>\n",
 "descriptionPlain": "by Roberto Ierusalimschy, creator of Lua",
 "tags": ["lua", "video"]
}

About 5% of stories carry one.

Writing a feed name

Everything below resolves to the same three feed shapes:

You write	Actor reads
hottest, or nothing at all	https://lobste.rs/page/1.json
newest	https://lobste.rs/newest/page/1.json
rust, t/rust, https://lobste.rs/t/rust	https://lobste.rs/t/rust/page/1.json
rust,go	https://lobste.rs/t/rust,go/page/1.json

A comma-separated tag feed is a single request returning stories carrying any of the tags. Listing tags separately in feeds instead gives you one request per tag and a feed field that tells them apart.

API example

curl -X POST "https://api.apify.com/v2/acts/arman-bd~lobsters-stories-scraper/run-sync-get-dataset-items?token=YOUR_TOKEN" \
 -H "Content-Type: application/json" \
 -d '{
 "feeds": ["hottest"],
 "minScore": 10,
 "maxStories": 50
 }'

JavaScript example

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_TOKEN' });
const run = await client.actor('arman-bd/lobsters-stories-scraper').call({
 feeds: ['hottest', 't/rust'],
 minScore: 5,
 maxStories: 60,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const s of items) console.log(`${s.score} ${s.title} [${s.tags.join(', ')}] ${s.commentsUrl}`);

Notes

• Stories are deduplicated by shortId. A story on both hottest and t/rust is saved once, tagged with whichever feed reached it first. The per-feed log line therefore shows a lower "saved" than "matched" on overlapping feeds, which is dedup working rather than a filter.
• Pagination is irregular, and handled for you. The front page pages as /page/N.json while every other feed appends /page/N.json to its own path. Both forms are built from the feed name you give.
• Paging stops early when a page returns fewer than 25 stories, or when maxStories is reached.
• Requests are spaced 300 ms apart. Lobsters is run by volunteers on modest hardware and advertises no rate limit, so this Actor stays deliberately quiet rather than finding the ceiling.
• A bad tag is a 404, recorded in RUN_SUMMARY.failures while other feeds carry on. The run only errors out if every feed fails.
• Scores move. Score and comment count are a snapshot at scrapedAt, not final values. Re-scrape the same shortId later to build a trajectory.

FAQ

Do I need a Lobsters account? No. You supply no credentials.

How far back can I go? Deep pages work, and newest page 50 still returns stories. Lobsters has no date-range parameter though, so historical collection means paging and stopping on createdAt yourself. For a rolling archive, run daily on newest and dedupe on shortId.

Why are scores so low compared with Hacker News? Lobsters is invite-only and much smaller. A score of 10 is a solid story and 50 is a front-page hit, so set minScore accordingly.

Does it fetch the linked articles? No. It returns the url so you can. Fetching every outbound link is a different job with different politeness requirements.

Can I get comments? Not from this Actor. Comment counts are included and commentsUrl points at the thread, but the per-story comment JSON is a separate endpoint.

Can I plug it into something else? Yes. Apify API, the client libraries, webhooks, scheduled runs, dataset exports to JSON, CSV or Excel, or MCP. The output is structured JSON.