Lobsters News Scraper
Pricing
from $10.00 / 1,000 results
Lobsters News Scraper
Lobsters Tech News Scraper
What Does Lobsters Tech News Scraper Do?
Lobsters Tech News Scraper is an Apify actor that scrapes the Lobste.rs technology news aggregator for curated programming articles, software engineering discussions, and tech news links. It extracts story titles, URLs, authors, scores, comment counts, and tags from this invite-only community known for its high signal-to-noise ratio.
Why Use This Lobsters Scraper?
Lobste.rs is a computing-focused community link aggregation site with a strong emphasis on quality technical content. Unlike larger aggregators, Lobste.rs requires invitations and uses fine-grained tagging. This scraper lets you programmatically monitor what the technical community is reading, track emerging technologies, analyze discussion trends, and build content feeds for newsletters or research pipelines.
How to Scrape Lobsters News
- Set the number of pages to scrape (each page contains approximately 25 stories).
- Configure the maximum number of results you want.
- Run the actor and download the structured dataset of tech news stories.
The scraper visits each page of Lobste.rs sequentially, extracting story metadata from the clean HTML using a Cheerio-based parser.
Input Parameters
| Parameter | Type | Description | Default |
|---|---|---|---|
pages | integer | Number of pages to scrape (25 stories per page) | 3 |
maxResults | integer | Maximum stories to return | 75 |
Output Data
Each story in the output dataset contains:
- title - Story headline
- url - Link to the original article
- domain - Source domain of the article
- author - Lobsters username who submitted the story
- score - Upvote score
- commentCount - Number of comments
- commentUrl - Link to the discussion
- tags - Array of topic tags (e.g., programming, security, web)
- scrapedAt - Timestamp of scrape
Cost of Usage
This actor is lightweight and affordable:
- Per result: $0.01
- Per 1,000 results: $10
- Actor start cost: $0.005
A typical run scraping 3 pages costs a fraction of a cent in platform resources.
Tips and Best Practices
- The default 3 pages gives you the latest 75 stories, covering roughly 2-3 days of content.
- Increase the page count to build a larger historical dataset.
- Tags are excellent for filtering results after extraction. Use them to focus on specific topics.
- Schedule the actor to run daily to build a continuous tech news archive.
Explore related scrapers:
- Wikipedia Article Extractor - Extract structured Wikipedia content
- HN Who Is Hiring Scraper - Job postings from Hacker News
- NPR Article Scraper - NPR news articles
