Lobsters Scraper — Developer News Stories & Discussions

Extract stories, votes, comments, tags, and metadata from Lobsters, a computing-focused community centered around link aggregation and discussion. This Actor lets you monitor trending developer topics, track engagement metrics, and analyze the content that matters to the programming community.

Use Cases

• Trend Monitoring — Track which technologies and topics gain traction among developers over time
• Content Curation — Build curated newsletters or feeds from Lobsters front-page stories
• Competitive Analysis — Monitor discussions around your product, framework, or technology
• Research — Analyze engagement patterns (votes vs. comments) across different domains and tags
• SEO & Content Strategy — Identify high-performing domains and story types in the developer space

Input

Field	Type	Required	Description
startUrls	Array	Yes	Lobsters URLs to scrape (e.g., https://lobste.rs/, https://lobste.rs/active)
maxItems	Number	No	Maximum stories to extract. Default: 100. Use 0 for unlimited.
maxPages	Number	No	Maximum pages to crawl. Default: 5. Use 0 for unlimited.
proxyConfiguration	Object	No	Proxy configuration. Recommended for production.

Output

The Actor outputs a dataset with the following fields for each story:

{
  "id": "wskhre",
  "title": "linux 0-day, access root-owned files as an unprivileged user",
  "url": "https://github.com/0xdeadbeefnetwork/ssh-keysign-pwn/",
  "votes": 61,
  "commentsCount": 25,
  "tags": ["linux", "security"],
  "domain": "github.com/0xdeadbeefnetwork",
  "author": "dzwdz",
  "publishedAt": "2026-05-14 20:14:21",
  "hasDescription": false,
  "lobstersUrl": "https://lobste.rs/s/wskhre"
}

Field Descriptions

• id — Short story identifier
• title — Story headline
• url — Link to the external or internal resource
• votes — Number of upvotes received
• commentsCount — Number of comments in the discussion
• tags — Array of topic tags assigned to the story
• domain — Source domain (e.g., github.com)
• author — Username of the submitter
• publishedAt — ISO-formatted submission timestamp
• hasDescription — Whether the story includes a text description
• lobstersUrl — Direct link to the story page on Lobsters

Pricing

Pay per event: $0.001 per story scraped.

No monthly subscription. You only pay for the stories extracted. Perfect for one-off research or scheduled monitoring.

Limitations

• Lobsters does not require authentication for front-page content
• Rate limiting is handled with automatic retries (3 attempts)
• Internal Lobsters discussion links (/s/...) are returned as-is in the url field
• Story descriptions (text body) are not extracted; only metadata and engagement metrics

FAQ

Q: Can I scrape user profiles or comments? A: This Actor focuses on front-page story listings. Comment extraction is not supported in this version.

Q: Do I need a Lobsters account? A: No. All data is publicly available without authentication.

Q: What happens if I hit a rate limit? A: The Actor automatically retries failed requests up to 3 times with exponential backoff.

Changelog

• v1.0.0 — Initial release. Story metadata extraction with pagination, tags, votes, and comments count.