Lobsters Scraper

Scrape tech stories from Lobsters, a computing-focused link aggregation community. Get titles, scores, tags, comment counts, and discussion links from hottest, newest, or tag-filtered feeds.

What does Lobsters Scraper do?

Lobsters Scraper uses the Lobsters JSON API to extract stories with full metadata. It collects story titles, URLs, scores, flag counts, comment counts, submitter usernames, tags, and timestamps.

Filter stories by tag (e.g. rust, python, security, web) or browse sections like hottest, newest, or active.

Who is it for?

• 💻 Software engineers — tracking trending technical discussions and curated links
• 📊 Technology analysts — monitoring developer community sentiment and topic trends
• 📝 Technical writers — finding popular technical topics for content ideas
• 🎓 Developer advocates — sourcing high-quality technical content for newsletters
• 🤖 NLP researchers — collecting curated developer discussion data for analysis

Why scrape Lobsters?

Lobsters is an invite-only, developer-focused community known for high-quality technical discussions. Unlike larger aggregators, Lobsters uses a transparent moderation model and mandatory tagging system.

Key reasons to scrape it:

• Technical trend tracking — Monitor what technologies developers are discussing
• Content discovery — Find high-quality articles curated by developers
• Community research — Analyze discussion patterns in a focused tech community
• Competitive intelligence — Track mentions of your product or technology
• Developer marketing — Identify trending topics for content strategy

Use cases

• Developer advocates tracking trending technologies and discussion topics
• Content marketers finding high-engagement technical content ideas
• Engineering managers staying current on community-vetted tools and practices
• Researchers studying developer community dynamics and preferences
• Product teams monitoring how their tools are received by developers
• Newsletter curators sourcing high-quality technical content

How to scrape Lobsters

1. Go to Lobsters Scraper on Apify Store
2. Choose a section (hottest, newest, or active)
3. Optionally filter by tag (e.g. rust, python, web)
4. Set result limits
5. Click Start and wait for results
6. Download data as JSON, CSV, or Excel

Input parameters

Parameter	Type	Default	Description
section	string	"hottest"	Feed section: hottest, newest, active
tag	string	""	Filter by tag (e.g. rust, python, security)
maxStories	integer	50	Maximum stories to return
maxPages	integer	3	Max pages (~25 stories per page)

Input example

{
    "section": "hottest",
    "tag": "rust",
    "maxStories": 25,
    "maxPages": 1
}

Output

Each story in the dataset contains:

Field	Type	Description
shortId	string	Unique Lobsters story ID
title	string	Story title
url	string	Link URL (external article)
score	number	Net upvotes
flags	number	Number of flags
commentCount	number	Number of comments
description	string	Optional story description
submitter	string	Username who submitted the story
userIsAuthor	boolean	Whether the submitter is the author
tags	string[]	Topic tags (e.g. ["rust", "web"])
lobstersUrl	string	Lobsters story page URL
commentsUrl	string	Lobsters comments page URL
createdAt	string	Submission timestamp
scrapedAt	string	ISO timestamp of extraction

Output example

{
    "shortId": "yyqowj",
    "title": "GRAM: A Zed fork without all the AI",
    "url": "https://gram.liten.app/",
    "score": 134,
    "flags": 2,
    "commentCount": 58,
    "description": "",
    "submitter": "patrik",
    "userIsAuthor": false,
    "tags": ["editors", "vibecoding"],
    "lobstersUrl": "https://lobste.rs/s/yyqowj",
    "commentsUrl": "https://lobste.rs/s/yyqowj/gram_zed_fork_without_all_ai",
    "createdAt": "2026-03-02T02:12:13.000-06:00",
    "scrapedAt": "2026-03-03T03:19:32.123Z"
}

How much does it cost to scrape Lobsters?

Lobsters Scraper uses pay-per-event pricing:

Event	Price
Run started	$0.001
Story extracted	$0.001 per story

Cost examples

Scenario	Stories	Cost
Quick check	25	$0.026
Daily digest	75	$0.076
Full archive	200	$0.201

Platform costs are negligible — typically under $0.001 per run.

Using Lobsters Scraper with the Apify API

Node.js

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_API_TOKEN' });

const run = await client.actor('automation-lab/lobsters-scraper').call({
    section: 'hottest',
    tag: 'rust',
    maxStories: 25,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(`Found ${items.length} stories`);
items.forEach(story => {
    console.log(`[${story.score}] ${story.title} (${story.tags.join(', ')})`);
});

Python

from apify_client import ApifyClient

client = ApifyClient('YOUR_API_TOKEN')

run = client.actor('automation-lab/lobsters-scraper').call(run_input={
    'section': 'hottest',
    'tag': 'rust',
    'maxStories': 25,
})

dataset = client.dataset(run['defaultDatasetId']).list_items().items
print(f'Found {len(dataset)} stories')
for story in dataset:
    print(f"[{story['score']}] {story['title']} ({', '.join(story['tags'])})")

Use with Claude AI (MCP)

This actor is available as a tool in Claude AI through the Model Context Protocol (MCP). Add it to Claude Desktop, Cursor, Windsurf, or any MCP-compatible client.

Setup for Claude Code

$claude mcp add --transport http apify "https://mcp.apify.com"

Setup for Claude Desktop, Cursor, or VS Code

Add this to your MCP config file:

{
    "mcpServers": {
        "apify": {
            "url": "https://mcp.apify.com"
        }
    }
}

Example prompts

• "What are the hottest stories on Lobsters right now? Give me the top 10 with scores."
• "Find the latest Rust-related articles on Lobsters and summarize the most discussed ones."
• "Show me trending security stories on Lobsters from the past week."

Learn more in the Apify MCP documentation.

Integrations

Lobsters Scraper works with all Apify integrations:

• Scheduled runs — Build a daily/weekly digest of trending tech stories
• Webhooks — Get notified when a scrape completes
• API — Trigger runs and fetch results programmatically
• Google Sheets — Export story data to a spreadsheet
• Slack — Share trending stories with your team

Connect to Zapier, Make, or Google Sheets for automated workflows.

Tips

• Filter by tag to focus on a specific technology (e.g. rust, python, security)
• Use "newest" section to catch stories before they trend
• Compare scores and flags — high flags relative to score may indicate controversial topics
• Check userIsAuthor to find original content vs. shared links
• Monitor specific tags on a schedule to build a focused tech newsletter
• Export to Google Sheets for team sharing and trend analysis

Legality

Scraping publicly available data is generally legal according to the US Court of Appeals ruling (HiQ Labs v. LinkedIn). This actor only accesses publicly available information and does not require authentication. Always review and comply with the target website's Terms of Service before scraping. For personal data, ensure compliance with GDPR, CCPA, and other applicable privacy regulations.

FAQ

What tags are available? Lobsters has dozens of tags including ai, rust, python, web, security, networking, programming, culture, hardware, math, and many more. Use the tag parameter to filter by any of them.

How many stories per page? Each page returns approximately 25 stories.

Is Lobsters invite-only? Yes, Lobsters requires an invitation to post and comment, but all stories and discussions are publicly readable.

How often should I scrape? The hottest page updates continuously. For daily tracking, one run per day is sufficient. For real-time monitoring, consider scheduling runs every few hours.

My tag filter returns no results. Lobsters tags are lowercase and specific. Check available tags at https://lobste.rs/tags to verify the exact tag name before using it.

Why do some stories have a high flag count? Flagged stories indicate community disagreement or off-topic content. A high flag-to-score ratio typically means the story is controversial. This is normal Lobsters moderation behavior.

Other news and community scrapers

• Hacker News Scraper -- scrape stories and discussions from Hacker News
• Dev.to Scraper -- extract articles and posts from Dev.to
• TechCrunch Scraper -- scrape tech news articles from TechCrunch
• Substack Scraper -- extract newsletter posts from Substack publications