Lobsters Scraper

Scrape tech stories from Lobsters, a computing-focused link aggregation community. Get titles, scores, tags, comment counts, and discussion links from hottest, newest, or tag-filtered feeds.

What does Lobsters Scraper do?

Lobsters Scraper uses the Lobsters JSON API to extract stories with full metadata. It collects story titles, URLs, scores, flag counts, comment counts, submitter usernames, tags, and timestamps.

Filter stories by tag (e.g. rust, python, security, web) or browse sections like hottest, newest, or active.

Why scrape Lobsters?

Lobsters is an invite-only, developer-focused community known for high-quality technical discussions. Unlike larger aggregators, Lobsters uses a transparent moderation model and mandatory tagging system.

Key reasons to scrape it:

• Technical trend tracking — Monitor what technologies developers are discussing
• Content discovery — Find high-quality articles curated by developers
• Community research — Analyze discussion patterns in a focused tech community
• Competitive intelligence — Track mentions of your product or technology
• Developer marketing — Identify trending topics for content strategy

Use cases

• Developer advocates tracking trending technologies and discussion topics
• Content marketers finding high-engagement technical content ideas
• Engineering managers staying current on community-vetted tools and practices
• Researchers studying developer community dynamics and preferences
• Product teams monitoring how their tools are received by developers
• Newsletter curators sourcing high-quality technical content

How to scrape Lobsters

1. Go to Lobsters Scraper on Apify Store
2. Choose a section (hottest, newest, or active)
3. Optionally filter by tag (e.g. rust, python, web)
4. Set result limits
5. Click Start and wait for results
6. Download data as JSON, CSV, or Excel

Input parameters

Parameter	Type	Default	Description
section	string	"hottest"	Feed section: hottest, newest, active
tag	string	""	Filter by tag (e.g. rust, python, security)
maxStories	integer	50	Maximum stories to return
maxPages	integer	3	Max pages (~25 stories per page)

Input example

{
    "section": "hottest",
    "tag": "rust",
    "maxStories": 25,
    "maxPages": 1
}

Output

Each story in the dataset contains:

Field	Type	Description
shortId	string	Unique Lobsters story ID
title	string	Story title
url	string	Link URL (external article)
score	number	Net upvotes
flags	number	Number of flags
commentCount	number	Number of comments
description	string	Optional story description
submitter	string	Username who submitted the story
userIsAuthor	boolean	Whether the submitter is the author
tags	string[]	Topic tags (e.g. ["rust", "web"])
lobstersUrl	string	Lobsters story page URL
commentsUrl	string	Lobsters comments page URL
createdAt	string	Submission timestamp
scrapedAt	string	ISO timestamp of extraction

Output example

{
    "shortId": "yyqowj",
    "title": "GRAM: A Zed fork without all the AI",
    "url": "https://gram.liten.app/",
    "score": 134,
    "flags": 2,
    "commentCount": 58,
    "description": "",
    "submitter": "patrik",
    "userIsAuthor": false,
    "tags": ["editors", "vibecoding"],
    "lobstersUrl": "https://lobste.rs/s/yyqowj",
    "commentsUrl": "https://lobste.rs/s/yyqowj/gram_zed_fork_without_all_ai",
    "createdAt": "2026-03-02T02:12:13.000-06:00",
    "scrapedAt": "2026-03-03T03:19:32.123Z"
}

Pricing

Lobsters Scraper uses pay-per-event pricing:

Event	Price
Run started	$0.001
Story extracted	$0.001 per story

Cost examples

Scenario	Stories	Cost
Quick check	25	$0.026
Daily digest	75	$0.076
Full archive	200	$0.201

Platform costs are negligible — typically under $0.001 per run.

Using Lobsters Scraper with the Apify API

Node.js

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_API_TOKEN' });

const run = await client.actor('automation-lab/lobsters-scraper').call({
    section: 'hottest',
    tag: 'rust',
    maxStories: 25,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(`Found ${items.length} stories`);
items.forEach(story => {
    console.log(`[${story.score}] ${story.title} (${story.tags.join(', ')})`);
});

Python

from apify_client import ApifyClient

client = ApifyClient('YOUR_API_TOKEN')

run = client.actor('automation-lab/lobsters-scraper').call(run_input={
    'section': 'hottest',
    'tag': 'rust',
    'maxStories': 25,
})

dataset = client.dataset(run['defaultDatasetId']).list_items().items
print(f'Found {len(dataset)} stories')
for story in dataset:
    print(f"[{story['score']}] {story['title']} ({', '.join(story['tags'])})")

Integrations

Lobsters Scraper works with all Apify integrations:

• Scheduled runs — Build a daily/weekly digest of trending tech stories
• Webhooks — Get notified when a scrape completes
• API — Trigger runs and fetch results programmatically
• Google Sheets — Export story data to a spreadsheet
• Slack — Share trending stories with your team

Connect to Zapier, Make, or Google Sheets for automated workflows.

Tips

• Filter by tag to focus on a specific technology (e.g. rust, python, security)
• Use "newest" section to catch stories before they trend
• Compare scores and flags — high flags relative to score may indicate controversial topics
• Check userIsAuthor to find original content vs. shared links
• Monitor specific tags on a schedule to build a focused tech newsletter
• Export to Google Sheets for team sharing and trend analysis

FAQ

What tags are available? Lobsters has dozens of tags including ai, rust, python, web, security, networking, programming, culture, hardware, math, and many more. Use the tag parameter to filter by any of them.

How many stories per page? Each page returns approximately 25 stories.

Is Lobsters invite-only? Yes, Lobsters requires an invitation to post and comment, but all stories and discussions are publicly readable.

How often should I scrape? The hottest page updates continuously. For daily tracking, one run per day is sufficient. For real-time monitoring, consider scheduling runs every few hours.