Lobsters Stories Scraper: Hottest, Newest, and Tagged Tech Stories

Overview

Lobsters Stories Scraper collects stories from the public Lobsters technology community lists, including Hottest, Newest, and individual tag lists. Each story is saved as a structured dataset record containing its headline, outbound link, score, flags, comment count, submitter, tags, and related Lobsters links.

The Actor does not require login credentials. It supports configurable pagination and item limits, making it suitable for collecting current technology discussions and maintaining recurring story archives.

Features

• Collects Hottest, Newest, and tag-specific story lists.
• Extracts headlines, outbound links, scores, and comment counts.
• Captures submitter information and author-status flags.
• Collects story tags and descriptions when available.
• Includes story and comment links for convenient navigation.
• Supports configurable page limits and maximum item counts.
• Removes duplicate stories using their short IDs.
• Retries temporary download failures and rate-limit responses.
• Records errors for invalid inputs and unsuccessful requests.
• Supports JSON, CSV, and Excel dataset exports.
• Includes request delays to reduce unnecessary load on the source.

Use Cases

• Daily reading lists: Collect popular technology stories for review.
• Tag monitoring: Track Python or other supported topics.
• Research and analysis: Compare story scores, flags, and comment counts.
• Discussion tracking: Keep story and comment links for relevant discussions.
• News archiving: Maintain a record of stories appearing across recurring runs.
• Dataset creation: Export structured technology community data for spreadsheet analysis.

Filtering Options

The Actor provides three collection modes through the feed parameter.

Mode	Description
hottest	Collects stories from the Hottest list.
newest	Collects stories from the Newest list.
tag	Collects stories associated with the specified tag.

When using tag mode, provide the desired tag through the tag parameter. For example, python collects stories associated with the Python tag.

Input Parameters

Parameter	Type	Default	Description
feed	String	hottest	Collection mode: hottest, newest, or tag.
tag	String	python	Tag to collect when feed is set to tag.
maxPages	Integer	4	Maximum number of pages to process. Each page can contain up to 25 stories.
maxItems	Integer	100	Maximum number of story records to save. The value is limited to 1–1,000.
timeoutSecs	Integer	300	Maximum overall run duration in seconds.
requestTimeoutSecs	Integer	30	Maximum duration of an individual request, in seconds.

Example Input

The following example collects Python-tagged stories from up to two pages, with a maximum of 50 records.

{
  "feed": "tag",
  "tag": "python",
  "maxPages": 2,
  "maxItems": 50,
  "timeoutSecs": 300,
  "requestTimeoutSecs": 30
}

To collect popular stories, set feed to hottest. To collect the newest stories, set it to newest.

Extracted Data

Each successful record represents one story and includes its identifier, creation time, headline, outbound link, engagement statistics, description, submitter information, tags, related Lobsters links, source list, and collection timestamp.

Descriptions are cleaned of HTML markup. Some stories may not contain descriptions, author information, or other optional details.

Example Output

{
  "short_id": "u9ue6c",
  "created_at": "2026-10-11T04:37:56.100-05:00",
  "storyTitle": "What's Wrong with GitHub",
  "url": "ARTICLE_URL",
  "score": 46,
  "flags": 2,
  "comment_count": 29,
  "storyDescription": "",
  "storyDescriptionPlain": "",
  "submitter_user": "tusharhero",
  "user_is_author": false,
  "tags": [
    "vcs"
  ],
  "short_id_url": "LOBSTERS_STORY_URL",
  "comments_url": "LOBSTERS_COMMENTS_URL",
  "scrapedAt": "2026-10-12T00:00:00+00:00"
}

The example illustrates the record structure. Actual values depend on the stories available in the selected list and tag.

Field Reference

Field	Type	Description
short_id	String	Short identifier assigned to the story. Used for deduplication.
created_at	String	Story creation timestamp provided by the source.
storyTitle	String	Story headline.
url	String	Outbound story link, or the Lobsters story URL when the outbound link is missing.
score	Number	Story score.
flags	Number	Number of flags associated with the story.
comment_count	Number	Number of comments associated with the story.
storyDescription	String	Story description with HTML markup removed.
storyDescriptionPlain	String	Plain-text story description with HTML markup removed.
submitter_user	String	Username of the story submitter.
user_is_author	Boolean	Indicates whether the submitter is identified as the story's author.
tags	Array	Tags associated with the story.
short_id_url	String	Link to the Lobsters story page.
comments_url	String	Link to the story's discussion page.
scrapedAt	String	UTC timestamp indicating when the record was saved.
error	String	Error description included in an error record.
input	Object or other input value	Input snapshot associated with an error record.

Some source fields may be empty when the corresponding information is unavailable.

Error Records

An invalid feed mode, unsuccessful response, or other processing failure can produce an error record instead of a story record.

{
  "error": "ERROR_DESCRIPTION",
  "input": {
    "feed": "INVALID_FEED"
  }
}

The exact structure of the input value depends on the failed input. An individual error does not necessarily stop the entire run.

Dataset Views and Exports

Collected story records are saved to the Actor's Apify dataset.

Supported export formats include:

• JSON
• CSV
• Excel

The exports can be used to build reading lists, review story engagement, track selected tags, or compare collected stories across runs.

Collection and Rate Limits

Pagination

The Actor supports multiple pages for Hottest, Newest, and tag-specific lists. Each page can contain up to 25 stories.

The number of pages processed is controlled by maxPages, while maxItems limits the number of saved records. Collection may stop when either limit is reached or when no additional stories are available.

Some Hottest pages may repeat stories from earlier pages. The Actor skips duplicate short IDs and continues processing within the configured limits.

Request frequency

The Actor waits at least one second before making the next request. This helps limit repeated requests to the source.

The source's public information does not specify a formal request-rate limit. Users should avoid excessive collection frequency and respect any applicable source restrictions.

Retry behavior

The Actor retries HTTP 429 responses, HTTP 5xx errors, and timeouts up to three additional times, with delays of approximately 2, 4, and 8 seconds. When the source provides a Retry-After value, that value may be used instead.

Timeouts

• Overall run timeout: 300 seconds by default.
• Individual request timeout: 30 seconds by default.

Data freshness

The lists reflect currently available stories and are not a complete historical archive. Schedule recurring runs if you need to maintain a longer-term collection.

Attribution and Source Restrictions

Lobsters is a community-driven technology discussion site. The source code for the site is licensed under BSD terms, while individual stories and linked content may remain subject to their respective authors' rights.

When using collected data:

• Preserve links to original stories and discussions.
• Attribute stories to Lobsters and their original sources where appropriate.
• Respect applicable source terms and restrictions.
• Do not assume that the site's software license grants permission to republish user-submitted content.
• Handle usernames and other potentially personal information responsibly.

The Actor collects story metadata and descriptions available in the public lists. Users are responsible for determining whether their intended use complies with applicable terms and rights.

FAQ

Which collection modes are supported?

The Actor supports hottest, newest, and tag. The tag mode requires a tag value.

How many stories can one run save?

The default maximum is 100 stories, and the hard maximum is 1,000. The actual number depends on the selected list, available pages, and configured limits.

What happens if the feed mode is invalid?

The Actor creates an error record containing information about the failed input. The run can continue rather than stopping entirely.

Why do some Hottest pages repeat stories?

Stories can appear on more than one page. The Actor deduplicates records using short_id and continues within the configured page and item limits.

Can I schedule recurring collection?

Yes. You can schedule the Actor through Apify and export the dataset after each run.

Are all story descriptions available?

No. Some stories do not provide a description. In those cases, the description fields may be empty.

How are duplicate stories identified?

The Actor uses short_id to identify duplicate stories and prevent repeated records within a run.

Can I export the results to CSV or Excel?

Yes. Dataset exports support JSON, CSV, and Excel formats.

Trademark Notice

Lobsters and all associated names, logos, and trademarks belong to their respective owners. They are referenced solely to identify the source of the collected information. This Actor is independent and is not affiliated with, sponsored by, endorsed by, or officially associated with Lobsters or its owners.

Disclaimer

This Actor is an independent tool for collecting publicly available story metadata. Source content, list availability, and structure may change over time, and uninterrupted or complete collection is not guaranteed.

Users are responsible for complying with applicable laws, source terms, attribution requirements, copyright restrictions, and data-handling obligations when collecting, storing, using, or redistributing the results.

Is Web Scraping Legal?

Collecting information that is already public is generally legal. Scraping is not the same thing as hacking, and there is no single law that bans it. The limits are the usual ones: do not break into accounts, do not take private or confidential data, and be careful with personal data and copyrighted works. A site's terms can also restrict how you access it. Facts such as a public name, price, or rating are not the same thing as republishing someone's article or photo.

This actor collects information from public pages. It does not log in and it does not bypass access controls.

For a longer explanation, see Apify's article Is web scraping legal?.