# Lobsters Stories Scraper (`zerobreak/lobsters-stories-scraper`) Actor

Lobsters Stories Scraper is a Python tool designed to extract submitted articles, discussion threads, comment trees, tags, and user metadata from Lobste.rs. It exports structured technical news data into clean CSV or JSON formats for analysis and research.

- **URL**: https://apify.com/zerobreak/lobsters-stories-scraper.md
- **Developed by:** [ZeroBreak](https://apify.com/zerobreak) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.99 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Lobsters Stories Scraper: Hottest, Newest, and Tagged Tech Stories

<p align="center">
  <img src="https://imgur.com/kv4DANa.png" alt="Lobsters Stories Scraper" width="800" />
</p>

### Overview

Lobsters Stories Scraper collects stories from the public Lobsters technology community lists, including Hottest, Newest, and individual tag lists. Each story is saved as a structured dataset record containing its headline, outbound link, score, flags, comment count, submitter, tags, and related Lobsters links.

The Actor does not require login credentials. It supports configurable pagination and item limits, making it suitable for collecting current technology discussions and maintaining recurring story archives.

### Features

- Collects Hottest, Newest, and tag-specific story lists.
- Extracts headlines, outbound links, scores, and comment counts.
- Captures submitter information and author-status flags.
- Collects story tags and descriptions when available.
- Includes story and comment links for convenient navigation.
- Supports configurable page limits and maximum item counts.
- Removes duplicate stories using their short IDs.
- Retries temporary download failures and rate-limit responses.
- Records errors for invalid inputs and unsuccessful requests.
- Supports JSON, CSV, and Excel dataset exports.
- Includes request delays to reduce unnecessary load on the source.

### Use Cases

- **Daily reading lists:** Collect popular technology stories for review.
- **Tag monitoring:** Track Python or other supported topics.
- **Research and analysis:** Compare story scores, flags, and comment counts.
- **Discussion tracking:** Keep story and comment links for relevant discussions.
- **News archiving:** Maintain a record of stories appearing across recurring runs.
- **Dataset creation:** Export structured technology community data for spreadsheet analysis.

### Filtering Options

The Actor provides three collection modes through the `feed` parameter.

| Mode | Description |
|---|---|
| `hottest` | Collects stories from the Hottest list. |
| `newest` | Collects stories from the Newest list. |
| `tag` | Collects stories associated with the specified tag. |

When using `tag` mode, provide the desired tag through the `tag` parameter. For example, `python` collects stories associated with the Python tag.

### Input Parameters

| Parameter | Type | Default | Description |
|---|---|---|---|
| `feed` | String | `hottest` | Collection mode: `hottest`, `newest`, or `tag`. |
| `tag` | String | `python` | Tag to collect when `feed` is set to `tag`. |
| `maxPages` | Integer | `4` | Maximum number of pages to process. Each page can contain up to 25 stories. |
| `maxItems` | Integer | `100` | Maximum number of story records to save. The value is limited to 1–1,000. |
| `timeoutSecs` | Integer | `300` | Maximum overall run duration in seconds. |
| `requestTimeoutSecs` | Integer | `30` | Maximum duration of an individual request, in seconds. |

### Example Input

The following example collects Python-tagged stories from up to two pages, with a maximum of 50 records.

```json
{
  "feed": "tag",
  "tag": "python",
  "maxPages": 2,
  "maxItems": 50,
  "timeoutSecs": 300,
  "requestTimeoutSecs": 30
}
```

To collect popular stories, set `feed` to `hottest`. To collect the newest stories, set it to `newest`.

### Extracted Data

Each successful record represents one story and includes its identifier, creation time, headline, outbound link, engagement statistics, description, submitter information, tags, related Lobsters links, source list, and collection timestamp.

Descriptions are cleaned of HTML markup. Some stories may not contain descriptions, author information, or other optional details.

### Example Output

```json
{
  "short_id": "u9ue6c",
  "created_at": "2026-10-11T04:37:56.100-05:00",
  "storyTitle": "What's Wrong with GitHub",
  "url": "ARTICLE_URL",
  "score": 46,
  "flags": 2,
  "comment_count": 29,
  "storyDescription": "",
  "storyDescriptionPlain": "",
  "submitter_user": "tusharhero",
  "user_is_author": false,
  "tags": [
    "vcs"
  ],
  "short_id_url": "LOBSTERS_STORY_URL",
  "comments_url": "LOBSTERS_COMMENTS_URL",
  "scrapedAt": "2026-10-12T00:00:00+00:00"
}
```

The example illustrates the record structure. Actual values depend on the stories available in the selected list and tag.

### Field Reference

| Field | Type | Description |
|---|---|---|
| `short_id` | String | Short identifier assigned to the story. Used for deduplication. |
| `created_at` | String | Story creation timestamp provided by the source. |
| `storyTitle` | String | Story headline. |
| `url` | String | Outbound story link, or the Lobsters story URL when the outbound link is missing. |
| `score` | Number | Story score. |
| `flags` | Number | Number of flags associated with the story. |
| `comment_count` | Number | Number of comments associated with the story. |
| `storyDescription` | String | Story description with HTML markup removed. |
| `storyDescriptionPlain` | String | Plain-text story description with HTML markup removed. |
| `submitter_user` | String | Username of the story submitter. |
| `user_is_author` | Boolean | Indicates whether the submitter is identified as the story's author. |
| `tags` | Array | Tags associated with the story. |
| `short_id_url` | String | Link to the Lobsters story page. |
| `comments_url` | String | Link to the story's discussion page. |
| `scrapedAt` | String | UTC timestamp indicating when the record was saved. |
| `error` | String | Error description included in an error record. |
| `input` | Object or other input value | Input snapshot associated with an error record. |

Some source fields may be empty when the corresponding information is unavailable.

#### Error Records

An invalid feed mode, unsuccessful response, or other processing failure can produce an error record instead of a story record.

```json
{
  "error": "ERROR_DESCRIPTION",
  "input": {
    "feed": "INVALID_FEED"
  }
}
```

The exact structure of the `input` value depends on the failed input. An individual error does not necessarily stop the entire run.

### Dataset Views and Exports

Collected story records are saved to the Actor's Apify dataset.

Supported export formats include:

- JSON
- CSV
- Excel

The exports can be used to build reading lists, review story engagement, track selected tags, or compare collected stories across runs.

### Collection and Rate Limits

#### Pagination

The Actor supports multiple pages for Hottest, Newest, and tag-specific lists. Each page can contain up to 25 stories.

The number of pages processed is controlled by `maxPages`, while `maxItems` limits the number of saved records. Collection may stop when either limit is reached or when no additional stories are available.

Some Hottest pages may repeat stories from earlier pages. The Actor skips duplicate short IDs and continues processing within the configured limits.

#### Request frequency

The Actor waits at least one second before making the next request. This helps limit repeated requests to the source.

The source's public information does not specify a formal request-rate limit. Users should avoid excessive collection frequency and respect any applicable source restrictions.

#### Retry behavior

The Actor retries HTTP 429 responses, HTTP 5xx errors, and timeouts up to three additional times, with delays of approximately 2, 4, and 8 seconds. When the source provides a `Retry-After` value, that value may be used instead.

#### Timeouts

- Overall run timeout: 300 seconds by default.
- Individual request timeout: 30 seconds by default.

#### Data freshness

The lists reflect currently available stories and are not a complete historical archive. Schedule recurring runs if you need to maintain a longer-term collection.

### Attribution and Source Restrictions

Lobsters is a community-driven technology discussion site. The source code for the site is licensed under BSD terms, while individual stories and linked content may remain subject to their respective authors' rights.

When using collected data:

- Preserve links to original stories and discussions.
- Attribute stories to Lobsters and their original sources where appropriate.
- Respect applicable source terms and restrictions.
- Do not assume that the site's software license grants permission to republish user-submitted content.
- Handle usernames and other potentially personal information responsibly.

The Actor collects story metadata and descriptions available in the public lists. Users are responsible for determining whether their intended use complies with applicable terms and rights.

### FAQ

#### Which collection modes are supported?

The Actor supports `hottest`, `newest`, and `tag`. The `tag` mode requires a tag value.

#### How many stories can one run save?

The default maximum is 100 stories, and the hard maximum is 1,000. The actual number depends on the selected list, available pages, and configured limits.

#### What happens if the feed mode is invalid?

The Actor creates an error record containing information about the failed input. The run can continue rather than stopping entirely.

#### Why do some Hottest pages repeat stories?

Stories can appear on more than one page. The Actor deduplicates records using `short_id` and continues within the configured page and item limits.

#### Can I schedule recurring collection?

Yes. You can schedule the Actor through Apify and export the dataset after each run.

#### Are all story descriptions available?

No. Some stories do not provide a description. In those cases, the description fields may be empty.

#### How are duplicate stories identified?

The Actor uses `short_id` to identify duplicate stories and prevent repeated records within a run.

#### Can I export the results to CSV or Excel?

Yes. Dataset exports support JSON, CSV, and Excel formats.

### Trademark Notice

Lobsters and all associated names, logos, and trademarks belong to their respective owners. They are referenced solely to identify the source of the collected information. This Actor is independent and is not affiliated with, sponsored by, endorsed by, or officially associated with Lobsters or its owners.

### Disclaimer

This Actor is an independent tool for collecting publicly available story metadata. Source content, list availability, and structure may change over time, and uninterrupted or complete collection is not guaranteed.

Users are responsible for complying with applicable laws, source terms, attribution requirements, copyright restrictions, and data-handling obligations when collecting, storing, using, or redistributing the results.

### Is Web Scraping Legal?

Collecting information that is already public is generally legal. Scraping is not the same thing as hacking, and there is no single law that bans it. The limits are the usual ones: do not break into accounts, do not take private or confidential data, and be careful with personal data and copyrighted works. A site's terms can also restrict how you access it. Facts such as a public name, price, or rating are not the same thing as republishing someone's article or photo.

This actor collects information from public pages. It does not log in and it does not bypass access controls.

For a longer explanation, see Apify's article [Is web scraping legal?](https://blog.apify.com/is-web-scraping-legal/).

# Actor input Schema

## `feed` (type: `string`):

Which public list to read: hottest, newest, or tag.

## `tag` (type: `string`):

Lobsters tag used when feed is tag. Letters, numbers, plus, underscore, and hyphen only.

## `maxPages` (type: `integer`):

How many list pages to request. Each page holds up to 25 stories.

## `maxItems` (type: `integer`):

Free cap on stories saved in one run. Values outside 1 to 1000 are clamped.

## `timeoutSecs` (type: `integer`):

Maximum seconds the run may spend fetching stories before it stops.

## `requestTimeoutSecs` (type: `integer`):

Seconds to wait for each HTTP response before retrying or recording an error.

## `proxyConfiguration` (type: `object`):

Optional proxy for requests. Leave Apify Proxy off unless the site blocks your IP.

## Actor input object example

```json
{
  "feed": "hottest",
  "tag": "python",
  "maxPages": 4,
  "maxItems": 100,
  "timeoutSecs": 300,
  "requestTimeoutSecs": 30,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `dataset` (type: `string`):

No description

## `datasetUrl` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "feed": "hottest",
    "tag": "python",
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("zerobreak/lobsters-stories-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "feed": "hottest",
    "tag": "python",
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("zerobreak/lobsters-stories-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "feed": "hottest",
  "tag": "python",
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call zerobreak/lobsters-stories-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,zerobreak/lobsters-stories-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/L2EeIv2DgXj7RgAwz/builds/LvtjWexh2KtCtHwGN/openapi.json