# Lobsters Stories Scraper: Front Page, Tags & Scores (`arman-bd/lobsters-stories-scraper`) Actor Scrape Lobsters: story titles, URLs, scores, tags, comment counts and submitter. A smaller, higher-signal alternative to Hacker News for technical content. - **URL**: https://apify.com/arman-bd/lobsters-stories-scraper.md - **Developed by:** [Arman Hossain](https://apify.com/arman-bd) (community) - **Categories:** News, Social media, Developer tools - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.37 / 1,000 story scrapeds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## Lobsters Stories Scraper: Front Page, Tags & Scores ![Lobsters Stories Scraper: Stories from the front page, the newest queue or any tag feed, title, score, tags, comment count and submitter](https://api.apify.com/v2/key-value-stores/ZQOcNAOHrIgTacAmy/records/lobsters-stories-scraper.jpg) Pulls stories from the public JSON feeds behind [lobste.rs](https://lobste.rs). You get the title, the outbound URL, score, tag list, comment count, submitter and the comments permalink, from the front page, the newest queue, or any tag feed you name. Lobsters is an invite-only technical community: a fraction of Hacker News' volume with a much higher hit rate per story. This Actor reads structured data directly, so there's no browser, no proxy, no login and no HTML parsing. **Agent skill: [SKILL.md](https://api.apify.com/v2/key-value-stores/t7YoTxpZEJOWvw4Ug/records/lobsters-stories-scraper.md)** ``` https://api.apify.com/v2/key-value-stores/t7YoTxpZEJOWvw4Ug/records/lobsters-stories-scraper.md ``` ### What you get | Field | What it holds | |---|---| | `feed` | Which feed the story came from (`hottest`, `newest`, `t/rust`) | | `shortId` | Lobsters' short identifier, the stable key for deduplication | | `title` | Story title | | `url`, `isTextPost` | The outbound link, or `null` with `isTextPost: true` for discussion-only posts | | `score`, `flags` | Net score and flag count | | `commentCount` | Comments at the time of the run | | `createdAt` | Submission time, ISO 8601 with the site's US/Central offset | | `submitterUser`, `userIsAuthor` | Who submitted it, and whether they wrote the linked thing | | `tags` | Every tag on the story | | `shortIdUrl`, `commentsUrl` | Short permalink and the full comments URL | | `description`, `descriptionPlain` | Submitter's note as HTML and as plain text. Usually empty, which Lobsters reports as `""` and this Actor stores as `null` | | `scrapedAt` | Run timestamp | `RUN_SUMMARY` in the key-value store holds per-run counts, the filters you used, and any feed page that failed. ### Use cases - **Engineering digest.** Pull the front page every morning above a score threshold and post it into Slack. - **Niche topic tracking.** Watch `t/rust`, `t/nix` or `t/compilers` for anything that clears 10 points. - **Hacker News complement.** The same story often lands on both, and joining on the outbound URL shows which communities care about what. - **Release monitoring.** The `release` tag is a clean firehose of version announcements for tools that matter. - **Newsletter sourcing.** A scored, tagged, deduplicated candidate list without reading a single web page. ### Quick start The current front page, nothing else: ```json { "feeds": ["hottest"] } ``` Two tag feeds plus the newest queue, only stories worth reading: ```json { "feeds": ["hottest", "newest", "t/rust,go"], "tags": ["rust", "go", "release"], "minScore": 5, "maxStories": 100, "maxPagesPerFeed": 4 } ``` A deep sweep of a quiet tag. The score filter throws most of it away, so page further: ```json { "feeds": ["t/compilers"], "minScore": 10, "maxStories": 200, "maxPagesPerFeed": 20 } ``` ### Input | Field | Type | Default | Notes | |---|---|---|---| | `feeds` | array | `["hottest"]` | `hottest`, `newest`, or a tag feed written as `rust`, `t/rust`, or `rust,go` for a combined feed. Full `lobste.rs` URLs are accepted and normalised. | | `tags` | array | `[]` | Keep only stories carrying one of these tags. Applied after fetching, so it filters `hottest` and `newest` too. Empty means all. | | `minScore` | integer | `0` | Drop anything scoring below this. | | `maxStories` | integer | `100` | Total cap across all feeds. 0 means no limit, bounded only by `maxPagesPerFeed`. | | `maxPagesPerFeed` | integer | `4` | Pages to walk per feed, 25 stories each. Max 100. | Nothing is required. Running with no input returns the current front page. `maxStories` and `maxPagesPerFeed` bound the run from two directions, and whichever is hit first stops it. With strict `tags` or `minScore` filters, raise `maxPagesPerFeed` or the run finishes early with very little. ### Output example A real record from the `hottest` feed: ```json { "feed": "hottest", "shortId": "xx83dr", "title": "jujutsu 0.44.0", "url": "https://github.com/jj-vcs/jj/releases/tag/v0.44.0", "isTextPost": false, "score": 13, "flags": 0, "commentCount": 10, "createdAt": "2026-08-06T04:03:51.889-05:00", "submitterUser": "olex", "userIsAuthor": false, "tags": ["release", "vcs"], "shortIdUrl": "https://lobste.rs/s/xx83dr", "commentsUrl": "https://lobste.rs/s/xx83dr/jujutsu_0_44_0", "description": null, "descriptionPlain": null, "scrapedAt": "2026-08-06T11:43:10.687Z" } ``` Where a submitter added a note, the two description fields carry it: ```json { "shortId": "kuopn0", "title": "Scripting, Programming Languages, Predictions", "description": "

by Roberto Ierusalimschy, creator of Lua

\n", "descriptionPlain": "by Roberto Ierusalimschy, creator of Lua", "tags": ["lua", "video"] } ``` About 5% of stories carry one. ### Writing a feed name Everything below resolves to the same three feed shapes: | You write | Actor reads | |---|---| | `hottest`, or nothing at all | `https://lobste.rs/page/1.json` | | `newest` | `https://lobste.rs/newest/page/1.json` | | `rust`, `t/rust`, `https://lobste.rs/t/rust` | `https://lobste.rs/t/rust/page/1.json` | | `rust,go` | `https://lobste.rs/t/rust,go/page/1.json` | A comma-separated tag feed is a single request returning stories carrying any of the tags. Listing tags separately in `feeds` instead gives you one request per tag and a `feed` field that tells them apart. ### API example ```bash curl -X POST "https://api.apify.com/v2/acts/arman-bd~lobsters-stories-scraper/run-sync-get-dataset-items?token=YOUR_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "feeds": ["hottest"], "minScore": 10, "maxStories": 50 }' ``` ### JavaScript example ```js import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: 'YOUR_TOKEN' }); const run = await client.actor('arman-bd/lobsters-stories-scraper').call({ feeds: ['hottest', 't/rust'], minScore: 5, maxStories: 60, }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); for (const s of items) console.log(`${s.score} ${s.title} [${s.tags.join(', ')}] ${s.commentsUrl}`); ``` ### Notes - Stories are deduplicated by `shortId`. A story on both `hottest` and `t/rust` is saved once, tagged with whichever feed reached it first. The per-feed log line therefore shows a lower "saved" than "matched" on overlapping feeds, which is dedup working rather than a filter. - Pagination is irregular, and handled for you. The front page pages as `/page/N.json` while every other feed appends `/page/N.json` to its own path. Both forms are built from the feed name you give. - Paging stops early when a page returns fewer than 25 stories, or when `maxStories` is reached. - Requests are spaced 300 ms apart. Lobsters is run by volunteers on modest hardware and advertises no rate limit, so this Actor stays deliberately quiet rather than finding the ceiling. - A bad tag is a 404, recorded in `RUN_SUMMARY.failures` while other feeds carry on. The run only errors out if every feed fails. - Scores move. Score and comment count are a snapshot at `scrapedAt`, not final values. Re-scrape the same `shortId` later to build a trajectory. ### FAQ **Do I need a Lobsters account?** No. You supply no credentials. **How far back can I go?** Deep pages work, and `newest` page 50 still returns stories. Lobsters has no date-range parameter though, so historical collection means paging and stopping on `createdAt` yourself. For a rolling archive, run daily on `newest` and dedupe on `shortId`. **Why are scores so low compared with Hacker News?** Lobsters is invite-only and much smaller. A score of 10 is a solid story and 50 is a front-page hit, so set `minScore` accordingly. **Does it fetch the linked articles?** No. It returns the `url` so you can. Fetching every outbound link is a different job with different politeness requirements. **Can I get comments?** Not from this Actor. Comment counts are included and `commentsUrl` points at the thread, but the per-story comment JSON is a separate endpoint. **Can I plug it into something else?** Yes. Apify API, the client libraries, webhooks, scheduled runs, dataset exports to JSON, CSV or Excel, or MCP. The output is structured JSON. # Actor input Schema ## `feeds` (type: `array`): Which feeds to read: 'hottest' (front page), 'newest' (submission order), or a tag feed written as 'rust', 't/rust' or 'rust,go' for a combined tag feed. Full lobste.rs URLs are accepted. Defaults to hottest when left empty. ## `tags` (type: `array`): Keep only stories carrying at least one of these tags. Applied after fetching, so it works on 'hottest' and 'newest' too. Leave empty to keep every story. ## `minScore` (type: `integer`): Drop stories scoring below this. Lobsters scores are small: 10 is already a good story, 50 is a hit. Set 0 to keep everything. ## `maxStories` (type: `integer`): Stop once this many stories have been saved across all feeds. Set 0 for no limit, bounded only by 'Max pages per feed'. ## `maxPagesPerFeed` (type: `integer`): How deep to page into each feed. Every page is 25 stories, so 4 pages is 100 stories per feed. Raise it when strict tag or score filters throw most stories away. ## Actor input object example ```json { "feeds": [ "hottest", "newest", "t/rust" ], "tags": [ "rust", "programming" ], "minScore": 0, "maxStories": 100, "maxPagesPerFeed": 4 } ``` # Actor output Schema ## `items` (type: `string`): Every record the run produced. ## `runsummary` (type: `string`): The RUN_SUMMARY record from the run's key-value store. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "feeds": [ "hottest" ] }; // Run the Actor and wait for it to finish const run = await client.actor("arman-bd/lobsters-stories-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "feeds": ["hottest"] } # Run the Actor and wait for it to finish run = client.actor("arman-bd/lobsters-stories-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "feeds": [ "hottest" ] }' | apify call arman-bd/lobsters-stories-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,arman-bd/lobsters-stories-scraper" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/HXsjSvS2wGEQrwQub/builds/8OPKgwM70vvEFqsZ2/openapi.json