Lobsters Scraper
Pricing
from $19.00 / 1,000 results
Lobsters Scraper
Scrapes Lobsters tech news stories from the hottest or newest feed, optionally filtered by tag. Returns each story as a flat row with title, score, tags, comments, and author.
Pricing
from $19.00 / 1,000 results
Rating
0.0
(0)
Developer
ParseForge
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
7 days ago
Last modified
Categories
Share
Lobsters Scraper
Scrape Lobsters tech news stories from the hottest or newest feed, filtered by tag, up to a million per run. Every story comes with its title, score, tags, comments, and author. No login or API key. Export to CSV, JSON, Excel, or XML.
Lobsters is a focused tech news community where stories are submitted and voted on by developers, and the front page is a tight signal of what engineers care about right now. This Actor reads the public hottest and newest feeds directly, so you can collect stories by tag without building a scraper or dealing with rate limits.
| Who uses it | What they scrape Lobsters for |
|---|---|
| Developer advocates | Which tools and languages the community is discussing this week |
| Market researchers | What topics and tags are trending among technical audiences |
| Content marketers | Story ideas and discussion angles that resonate with developers |
| Data analysts | A clean dataset of tech news stories for trend analysis |
What it does
This Actor collects Lobsters tech news stories from the hottest or newest feed, optionally filtered by tag, and returns each story as a flat row.
- 🔥 Hottest feed: stories ranked by score, the same list you see on the Lobsters front page.
- 🆕 Newest feed: stories in chronological order, for catching submissions as they appear.
- 🏷️ Tag filter: narrow to a single tag like programming, javascript, python, security, linux, web, ask, or show.
- 📦 Flat rows: every story is one row with title, score, tags, comments, and author, ready for CSV or JSON.
Results export to CSV, JSON, Excel, or XML, or straight from the API.
What you can do with Lobsters data
📈 Track trending tech topics.
A developer advocate runs the Actor on the hottest feed each morning to see which tags and stories are gaining traction, then writes a newsletter that matches the community's current interests.
🔍 Monitor a specific tag.
A security researcher filters by the security tag and pulls the newest stories daily to stay on top of disclosed vulnerabilities and discussions before they hit mainstream news.
📊 Build a dataset for analysis.
A data analyst collects a month of stories with scores and tags, then analyzes which topics get the most engagement and how that changes over time.
✍️ Find content ideas.
A content marketer scrapes the ask tag to see what questions developers are asking, then writes blog posts that answer those questions directly.
Why choose this scraper
| What you get | |
|---|---|
| No API key | Reads the public Lobsters feeds directly, no registration or OAuth. |
| Tag filtering | Collect only stories from one tag, like security or ask, instead of the whole feed. |
| Flat schema | Every story is one row with the same fields, so you can append runs and build a history. |
| Scale | Paid users can pull up to a million stories per run for large archives. |
How it compares
This Actor focuses on the hottest and newest feeds with tag filtering, while the other Lobsters scraper also mentions search results.
| Feature | ParseForge | Lobsters Scraper - Tech News & Discussions |
|---|---|---|
| Hottest feed | Yes | Yes |
| Newest feed | Yes | Yes |
| Tag filter | Yes | Not listed |
| Search results | Not listed | Yes |
| Flat row output | Yes | Not listed |
Configure the run
Drive the Actor from the hottest or newest feed, optionally filtered by a single tag, and set a maximum number of stories so only the matches you need reach your dataset. The Input tab lists every parameter.
A first run with the defaults:
A larger pull:
Pricing
Pay-per-result: $0.021 per result collected. You pay only for the results written to your dataset.
| Results collected | Approximate cost |
|---|---|
| 100 results | $2.10 |
| 1,000 results | $21.00 |
| 10,000 results | $210.00 |
New Apify accounts start with $5 in free credit.
Free users
Free-plan runs return up to 10 results as a preview. Upgrade your Apify plan to collect up to 1,000,000 results per run.
Run it
- Create a free Apify account with $5 in credit.
- Open the Lobsters Scraper.
- Set your inputs and any filters, then click Start.
- Export the results as CSV, Excel, JSON, or XML from the Dataset tab.
Run it programmatically through the Apify API (run-sync-get-dataset-items) or the ApifyClient for JavaScript and Python.
Use with AI agents (MCP)
Give an AI agent live access to Lobsters through the Model Context Protocol. Add the Actor to Claude, Cursor, or any MCP client:
Then prompt it in plain language to run the scraper and read back the results.
Troubleshooting
Why am I getting no results?
Check your tag filter. If you set a tag that does not exist or has no recent stories, the Actor returns an empty dataset. Try leaving the tag empty or using a common tag like programming.
Why am I only getting 10 items?
Free users are limited to 10 items as a preview. To get more, upgrade to a paid plan and set maxItems to a higher number.
Why are the results different from the Lobsters front page?
The hottest feed is ranked by score, which can change as users vote. If you run the Actor at different times, the order and set of stories may differ.
Can I get more than one tag at a time?
No, the tag input accepts a single tag. To collect multiple tags, run the Actor once per tag and merge the datasets.
Why is the author field empty for some stories?
Some stories may be submitted by users who have deleted their accounts or by anonymous users. In those cases, the author field is empty.
FAQ
| Question | Answer |
|---|---|
| What is Lobsters? | Lobsters is a technology-focused link aggregation site where users submit and vote on stories. It is similar to Hacker News but with a smaller, more curated community and a strong tagging system. |
| Do I need an API key or login? | No. This Actor reads the public Lobsters feeds directly, so you do not need to register an app or provide credentials. |
| Which feeds can I scrape? | You can scrape the hottest feed, which is ranked by score, or the newest feed, which is chronological. Both are available from the feedType input. |
| Can I filter by tag? | Yes. Set the tag input to a single tag like programming, javascript, python, security, linux, web, ask, or show. Leave it empty to get all stories. |
| What data do I get for each story? | Each story is returned as a flat row with its title, score, tags, comments, and author, among other fields. The exact fields are shown in the sample output. |
| How many stories can I scrape? | Free users are limited to 10 items as a preview. Paid users can set maxItems up to 1,000,000 per run. |
| What formats can I export to? | You can export the results to CSV, JSON, Excel, or XML from the Apify dataset. |
| Is this the same as the Hacker News scraper? | No. Hacker News and Lobsters are different sites with different communities and data. This Actor scrapes Lobsters only. |
| Can I scrape search results? | This Actor scrapes the hottest and newest feeds, not the search page. If you need search results, you may need a different tool. |
| How often should I run this? | It depends on your use case. For trending topics, run it daily on the hottest feed. For monitoring, run it hourly on the newest feed with a tag filter. |
Related actors
Browse the full ParseForge collection for more scrapers.
🆘 Need help? Email parseforge@protonmail.com with your run ID, your input, and what you expected.
⚠️ Disclaimer. This Actor is unofficial and is not affiliated with, endorsed by, or sponsored by Lobsters. It collects only publicly available data. You are responsible for using the collected data in compliance with the source's terms of service and applicable data-protection laws, including GDPR, CCPA, and PIPL. Do not use it to collect personal data unlawfully.
