Lobsters Scraper - Tech Stories, Scores & Tags

Scrape Lobste.rs tech stories in bulk. Extract title, link, score, comment count, tags, submitter, description and timestamps from the front page or any archive page to CSV/JSON. No API key, no login.

---

What does the Lobsters Scraper do?

This Actor turns any Lobsters search into a structured dataset. You give it the hottest feed or the newest feed, it walks the result pages one after another, and it writes one clean row per storie into your dataset — ready to export as JSON, CSV or Excel, or to pull straight from the Apify API.

Lobste.rs publishes every page of its feeds as JSON at a .json suffix, so the Actor reads typed records straight from the source. Scores, comment counts and tags arrive as data rather than as parsed markup. Pagination is followed automatically until it runs out of results, hits your page limit or hits your Max items cap, whichever comes first. Every row is de-duplicated across the whole run, so you are never billed twice for the same storie.

There is no API key, no login and no browser involved. That keeps runs fast and cheap, and it means you can schedule the Actor without worrying about credentials expiring.

Who is it for?

• Developer-relations teams tracking what the engineering community is reading.
• Content strategists finding which topics earn discussion in a technical audience.
• Researchers studying link-sharing and tagging in a moderated community.
• Newsletter editors sourcing high-signal technical links automatically.
• Product teams monitoring mentions of a language, tool or competitor.

Use cases

• Export the hottest feed and rank stories by score.
• Track which tags dominate discussion week over week.
• Monitor a topic by filtering exported rows on tags.
• Build a curated technical link feed for a newsletter.
• Assemble a dataset of link-sharing behaviour with scores and comment counts.

Why use this Lobsters Scraper?

• 🔑 Keyless — no account, no API token, no cookies to paste.
• 📦 14 fields per storie — everything the result page exposes, already typed.
• 📄 Real pagination — it walks page after page instead of returning the first screen.
• 🎯 Precise caps — Max items stops the run exactly where you want it, so the bill is predictable.
• 📊 Export anywhere — JSON, CSV, Excel or HTML, plus the Apify API and integrations.
• 💸 Pay per result — you pay for rows you actually receive, with no platform fees to calculate.

What data can you extract?

Every run produces one row per storie, with these fields:

Field	Type	Description
storyId	string	Lobsters short ID
title	string	Story title
storyUrl	string	Lobsters discussion page URL
linkUrl	string	External link the story points at
score	number	Net score of the story
commentCount	number	Number of comments
commentsUrl	string	Direct URL to the comment thread
tags	string	Tags, separated by a pipe
submitter	string	Username that submitted the story
description	string	Submitter's description when provided
createdAt	string	ISO timestamp the story was submitted
feed	string	Feed this row came from
page	number	Result page the story appeared on
scrapedAt	string	ISO timestamp of extraction

Output example

{
  "commentCount": 0,
  "commentsUrl": "https://lobste.rs/s/jnutrb/singeli_high_level_interface_for_low",
  "createdAt": "2026-09-13T21:26:52.276-05:00",
  "description": "> Singeli is a domain-specific language for building high-performance algorithms (including SIMD) with flexible abstractions over code that corresp\u2026",
  "feed": "hottest",
  "linkUrl": "https://github.com/mlochbaum/Singeli",
  "page": 1,
  "score": 4,
  "scrapedAt": "2026-09-14T07:56:38.501Z",
  "storyId": "jnutrb",
  "storyUrl": "https://lobste.rs/s/jnutrb",
  "submitter": "veqq",
  "tags": "apl | compilers | performance",
  "title": "Singeli: High-level interface for low-level programming"
}

How to use

Option A — the hottest feed

{
  "feed": "hottest",
  "maxItems": 250,
  "maxPagesPerSearch": 10
}

1. Open the Actor and fill in the the hottest feed field.
2. Set Max pages per search and Max items to bound the run.
3. Click Start, then export from the Output tab.

Option B — the newest feed

{
  "feed": "newest",
  "maxItems": 500,
  "maxPagesPerSearch": 20
}

Paste one or more Lobsters URLs into Start URLs and the Actor paginates each of them independently.

Input parameters

Parameter	Type	Default	Description
feed	string	hottest	Which Lobsters feed to read.
maxPagesPerSearch	integer	6	How many result pages to walk for each search.
maxItems	integer	200	Stop after this many stories.
maxConcurrency	integer	2	Parallel requests.
proxyConfiguration	object	{"useApifyProxy": true}	Proxy used to fetch pages.

Tips for best results

• hottest is the ranked front page; newest is better for monitoring.
• Lobsters returns 25 stories per page, so a 10-page run yields roughly 250 rows.
• tags is pipe-separated so it survives a CSV round trip; split it after import.
• Sort by score after export to separate signal from volume.
• linkUrl is the external article; storyUrl is the discussion — they are different things.
• Schedule a daily run to build an archive the site itself does not expose as a bulk feed.
• An empty description simply means the submitter linked without commentary.
• Keep Max concurrency low — Lobsters is a small, volunteer-run community site.
• Archive pages go back a long way, so deep page counts still return real data.
• Pair with the Hacker News archive scraper to compare what each community surfaces.

Integrations

Send results straight into the tools you already use: Google Sheets, Slack, Zapier, Make, Airtable or any Webhook. You can also schedule the Actor to run hourly, daily or weekly and have each run append to the same dataset, which is how you build a price or availability history rather than a one-off snapshot.

API usage

Run the Actor and collect results from any language. Replace <YOUR_TOKEN> with your Apify API token.

cURL

curl -X POST "https://api.apify.com/v2/acts/logiover~lobsters-tech-news-scraper/run-sync-get-dataset-items?token=<YOUR_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"feed": "hottest", "maxItems": 250, "maxPagesPerSearch": 10}'

Node.js

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_TOKEN>' });
const run = await client.actor('logiover/lobsters-tech-news-scraper').call({"feed": "hottest", "maxItems": 250, "maxPagesPerSearch": 10});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Python

from apify_client import ApifyClient

client = ApifyClient('<YOUR_TOKEN>')
run = client.actor('logiover/lobsters-tech-news-scraper').call(run_input={"feed": "hottest", "maxItems": 250, "maxPagesPerSearch": 10})
for item in client.dataset(run['defaultDatasetId']).iterate_items():
    print(item)

Use with AI agents (MCP)

This Actor is available through the Apify MCP server, so an AI agent can call it as a tool. Point your agent at https://mcp.apify.com and it can run the Lobsters Scraper on demand — for example: "Pull the first 500 stories from Lobsters and summarise the price distribution." The agent receives the same structured rows you would get from the UI.

FAQ

Do I need a Lobsters account or API key?

No. The Actor reads publicly available pages only. There is nothing to authenticate and no credentials to rotate.

How many stories can I get in one run?

As many as the search exposes. Raise Max pages per search and Max items together; the run stops at whichever limit it reaches first.

Why did I get fewer rows than I asked for?

The search ran out of stories. That is normal for narrow queries — broaden the search or add more searches to one run.

Are results de-duplicated?

Yes. Each storie is emitted once per run, even when it appears on several pages, so you are never billed twice for the same record.

Why are some fields empty?

Lobsters does not publish every attribute for every storie. Empty means the source did not show it, not that extraction failed.

What export formats are supported?

JSON, CSV, Excel, HTML and RSS from the Output tab, plus the Apify API and any integration you connect.

How fast is it?

It is pure HTTP with no browser, so a page of results typically takes a second or two. Raise Max concurrency carefully — the source rate-limits aggressive crawling.

Can I schedule it?

Yes. Use the Apify scheduler to run it on any interval and append each run to the same dataset for time-series analysis.

Does it work behind a proxy?

It uses Apify Proxy automatically. You can switch groups or supply your own proxies in Proxy configuration.

How often does the data change?

Lobsters updates continuously. Re-run whenever you need current data; the Actor always reads the live pages, never a cache.

Is the output schema stable?

Yes. Field names and types are fixed, so downstream pipelines will not break between runs.

What if the site changes its layout?

Open an issue on the Issues tab and it gets fixed. The Actor is actively maintained.

Is it legal?

This Actor reads only publicly available pages on Lobsters — the same content any visitor sees without logging in. It does not bypass authentication, does not collect private data and does not attempt to defeat access controls. You are responsible for how you use the output: respect the source's terms of service, applicable copyright, and data-protection law such as GDPR where personal data is involved. Scraping public data is generally lawful in the EU and the US, but the responsibility for the downstream use of that data sits with you.

Related scrapers

• Hacker News Archive — Tech discussion history
• Reddit Search Scraper — Reddit posts and comments
• Lemmy Scraper — Fediverse discussion
• dev.to Scraper — Developer articles