🦞 Lobste.rs Tech News Scraper

🚀 Export curated tech stories from Lobste.rs in seconds. Get titles, scores, tags, authors, and discussion links. no account, no auth, no rate limits.

🕒 Last updated: 2026-05-22 · 📊 14 fields per record · Unlimited items · Global coverage

Lobste.rs is an invite-only, curated tech community where experts share and discuss links across programming, security, systems, web development, and more. Unlike mainstream aggregators, every submitter is accountable. real usernames, real reputations. This scraper taps Lobste.rs's fully public JSON API to export stories in real time, with zero authentication or proxies required.

Whether you're tracking technology trends, monitoring specific tags like security or python, or building a developer-focused news feed, this actor delivers clean, structured data in seconds.

Coverage: Thousands of curated tech stories across 50+ community-maintained tags, covering programming languages, operating systems, web, security, mathematics, hardware, and more. Every record includes the story's score, flag count, comment count, author username, tags, and optionally the full description for self-posts.

Target Audience / Use Cases

Audience	Use Case
Researchers & Analysts	Track technology trends and community sentiment over time
Content Teams	Monitor developer community discussions for content ideas
Data Scientists	Build datasets of high-quality, expert-curated tech links
News Aggregators	Power developer-focused newsletters or Slack digests
Security Teams	Watch the security tag for emerging vulnerability discussions
Educators	Find the most-discussed topics in specific programming domains

📋 What the Lobste.rs Scraper does

• Fetches stories from the hottest feed (ranked by community score) or newest feed (chronological)
• Filters by any Lobste.rs tag: programming, javascript, python, security, linux, web, ask, show, and 50+ more
• Paginates automatically to collect as many stories as you need (up to 1,000,000)
• Returns structured records with title, URL, score, flags, comment count, tags, author, and timestamps
• Captures full description text for self-posts (ask/show posts written directly on Lobste.rs)
• Uses a polite 500ms delay between pages to respect the community server

💡 Why it matters: Lobste.rs is one of the most signal-rich tech communities on the internet. Unlike Hacker News, every member is vouched by a real person. The stories that trend here represent genuine expert consensus. not viral noise.

🎬 Full Demo

🚧 Coming soon

⚙️ Input

Field	Type	Default	Description
feedType	select	hottest	Feed to scrape: hottest (by score) or newest (chronological)
tag	string	(empty)	Filter by tag slug (e.g. programming, security, linux). Leave empty for all stories.
maxItems	integer	10	Max stories to return. Free: limited to 10. Paid: up to 1,000,000.

Example 1 - Top security stories:

{
    "feedType": "hottest",
    "tag": "security",
    "maxItems": 100
}

Example 2 - Latest programming stories:

{
    "feedType": "newest",
    "tag": "programming",
    "maxItems": 50
}

⚠️ Good to Know: The tag field accepts any tag slug that Lobste.rs uses in their URLs (e.g. https://lobste.rs/t/security). Common tags include: programming, javascript, python, security, linux, web, ask, show, go, rust, c, devops, math, hardware, browsers, practices. Self-posts (ask/show) include description text; link posts have description: null.

📊 Output

Field	Type	Description
📰 title	string	Story headline
🔗 url	string	External URL (or lobste.rs URL for self-posts)
🦞 lobstersUrl	string	Lobste.rs discussion page URL
⭐ score	integer	Community score (upvotes minus flags, weighted)
🚩 flags	integer	Number of flags (downvote signals) from the community
💬 commentCount	integer	Number of comments on the story
🏷️ tags	array	Community tags assigned to the story
👤 submittedBy	string	Username of the person who submitted the story
📝 description	string or null	Full self-post body (HTML); null for link posts
📅 createdAt	string	ISO 8601 timestamp of submission
🔗 commentsUrl	string	Direct URL to the comments section
🔑 shortId	string	Lobste.rs short ID (used in URLs)
🕒 scrapedAt	string	ISO 8601 timestamp of when this record was scraped
❌ error	string or null	Error message if something went wrong; always last

Sample record (link post):

{
    "title": "Chromium publishes fixed exploit 4 years later, turns out it's actually unfixed",
    "url": "https://infosec.exchange/@rebane2001/116606719764376414",
    "lobstersUrl": "https://lobste.rs/s/e7lsqn",
    "score": 97,
    "flags": 1,
    "commentCount": 13,
    "tags": ["browsers", "security"],
    "submittedBy": "hugoarnal",
    "description": null,
    "createdAt": "2026-05-20T15:29:05.000-05:00",
    "commentsUrl": "https://lobste.rs/s/e7lsqn/chromium_publishes_fixed_exploit_4_years",
    "shortId": "e7lsqn",
    "scrapedAt": "2026-05-22T01:43:03.583Z",
    "error": null
}

Sample record (self-post with description):

{
    "title": "Internships for early university / no former employment",
    "url": "https://lobste.rs/s/r87zln",
    "lobstersUrl": "https://lobste.rs/s/r87zln",
    "score": 28,
    "flags": 0,
    "commentCount": 15,
    "tags": ["ask", "job"],
    "submittedBy": "runxiyu",
    "description": "<p>As a rising freshman beginning an undergraduate computer science degree...</p>",
    "createdAt": "2026-05-21T10:09:32.000-05:00",
    "commentsUrl": "https://lobste.rs/s/r87zln/internships_for_early_university_no",
    "shortId": "r87zln",
    "scrapedAt": "2026-05-22T01:43:03.583Z",
    "error": null
}

✨ Why choose this Actor

Feature	Detail
🚀 No auth required	Lobste.rs JSON API is fully public. no login, no token
⚡ Real-time data	Always fetches live data, never cached responses
🏷️ Tag filtering	Narrow to any of Lobste.rs's 50+ community tags in one field
📄 Full pagination	Collects across multiple pages automatically
🛡️ Polite crawling	500ms delay between pages, respects the community server
🔍 Clean output	All fields normalized, null for genuinely absent data
💸 Pay-per-event pricing	Only pay for what you collect
🌐 Zero proxy cost	Plain fetch. no proxy needed, no IP rotation overhead

📈 How it compares to alternatives

Feature	This Actor	Manual API calls	RSS feeds
Structured JSON output	✅	Requires custom parsing	Partial
Tag filtering	✅	✅	Limited
Automatic pagination	✅	Manual	No
Apify dataset integration	✅	No	No
Webhook / schedule support	✅	No	Limited
CSV / Excel export	✅	No	No

🚀 How to use

1. Create a free Apify account (includes $5 free credit)
2. Go to the Lobste.rs Tech News Scraper page
3. Click Try for free
4. Set your feedType (hottest or newest), optional tag, and maxItems
5. Click Start and wait a few seconds
6. Download your dataset as JSON, CSV, Excel, or XML

💼 Business use cases

Tech Trend Analysis

Monitor which topics are gaining traction in the developer community. Track score velocity on security or ai tags over days or weeks. Correlate community interest spikes with product launch cycles.

Content Intelligence

Know what developers are actually reading and discussing. Use the hottest feed as an editorial signal for blog posts, videos, or podcast topics. Identify underserved niches by watching what generates high comment counts with low scores.

Competitive and Threat Intelligence

Security teams can monitor the security tag in real time. When a CVE or novel attack technique surfaces on Lobste.rs, it often trends before mainstream outlets. Get structured, timestamped records to feed into SIEM or alerting pipelines.

Developer Community Research

Academic and industry researchers can build longitudinal datasets of expert-curated tech links. Study how community reception (score, flags, comments) correlates with article quality, recency, or domain authority.

🔌 Automating Lobste.rs Scraper

Connect this actor to your workflows with no code:

• Make (Integromat): Schedule runs, then send new stories to Slack, Notion, Airtable, or Google Sheets
• Zapier: Trigger a Zap when a run finishes - push top stories to your CMS or email list
• Slack: Pipe the hottest security stories into a #security-feed channel every morning
• Google Sheets: Build a living spreadsheet of tech trends with daily scheduled runs
• Webhooks: Use Apify's webhook system to POST results to any internal API or dashboard

🌟 Beyond business use cases

Research

Build citation-quality datasets of expert-curated tech literature. Lobste.rs's invite-only model means submitters are accountable - the links represent genuine expert consensus, not algorithmic noise.

Creative

Run weekly "best of Lobste.rs" newsletters or zines. Use the description field of self-posts to surface community-written content. Build a reading list generator that matches your tag interests.

Non-profit and Open Source

Track open-source project discussions by watching tags like open-source or freebsd. Monitor community health signals across sub-communities by comparing score distributions over time.

Experimentation

Build a personal recommendation engine trained on Lobste.rs community signals. Train a classifier to predict story success from title alone, using score as the label.

🤖 Ask an AI assistant about this scraper

"I want to track security vulnerabilities discussed by expert developers. How can I use the Lobste.rs Scraper to build a daily digest?"

"Show me how to export the top 500 Python stories from Lobste.rs this month as a CSV."

"I'm building a developer newsletter. How do I connect the Lobste.rs Scraper to Mailchimp via Zapier?"

Paste any of these into ChatGPT, Claude, or Gemini alongside the actor URL for step-by-step integration guidance.

❓ Frequently Asked Questions

🔑 Do I need an account to use this? No. Lobste.rs has a completely public JSON API. This actor requires no credentials, no auth tokens, and no proxies.

📄 How many stories can I get per run? Paid users can collect up to 1,000,000 stories. Free users get a 10-item preview. Lobste.rs paginates in batches of ~25 stories per page.

🏷️ What tags are available? Lobste.rs maintains 50+ community tags. Common ones: programming, javascript, python, rust, go, c, security, linux, web, devops, math, hardware, browsers, practices, ask, show. Browse all tags at https://lobste.rs/tags.

📝 Why is description null for most stories? Only self-posts (ask/show posts written directly on Lobste.rs) have a description. Link posts that point to external URLs have description: null - this is correct behavior from the source.

⭐ What is score vs flags? score is the net community rating. flags counts how many users flagged a story as off-topic or low quality. A story with high score and low flags is strongly endorsed by the community.

🔄 How fresh is the data? Data is fetched in real time on every run. There is no caching. Results reflect Lobste.rs's current state at the moment you start the actor.

📊 Can I get comments, not just stories? This actor returns story-level data including comment counts and links to the comments page. For full comment text, you would need to fetch individual story detail pages.

🗓️ Can I schedule daily runs? Yes. Use Apify's built-in scheduler to run this actor on a cron schedule (e.g., every morning at 8am UTC) and accumulate a longitudinal dataset.

🔌 Can I send results directly to Google Sheets? Yes. Use the Apify-Google Sheets integration or connect via Make or Zapier to push each run's results into a spreadsheet automatically.

🚦 Is there a rate limit risk? The actor uses a polite 500ms delay between page requests and sends a standard User-Agent header. Lobste.rs's public API is designed for programmatic access.

💸 How is pricing calculated? This actor uses pay-per-event pricing - you only pay for the stories you actually collect. There are no run-time fees for idle pages or failed requests.

🔌 Integrate with any app

Export your Lobste.rs dataset to: JSON - CSV - Excel - XML - Google Sheets - Airtable - Notion - Slack - Zapier - Make - Power Automate - REST API - Webhooks - S3 - BigQuery - Snowflake - PostgreSQL - MongoDB - Elasticsearch

🔗 Recommended Actors

Actor	Description
Hacker News Scraper	Export stories and comments from Hacker News
GitHub Trending Scraper	Track trending repositories by language and time period
Dev.to Article Scraper	Export developer articles from Dev.to by tag or author

💡 Pro Tip: browse the complete ParseForge collection for 50+ specialized data extractors covering developer tools, tech communities, job boards, and more.

---

This actor is not affiliated with or endorsed by Lobste.rs. It uses only publicly available data from the official Lobste.rs JSON API. Use responsibly and in accordance with Lobste.rs's terms of service.