Hacker News Search & Front Page Scraper

What is Hacker News Search & Front Page Scraper?

Hacker News Search & Front Page Scraper is an Apify Actor that pulls stories, comments, Show HN, Ask HN and poll items straight from the free Algolia Hacker News Search API — no API key, no scraping of news.ycombinator.com, no anti-bot walls. Give it one or more search terms and it returns one clean row per matched item; leave a query empty (or type front_page) and it returns the current Hacker News front page instead. Every row carries points, author, comment count, the external URL and — for comments and Ask HN posts — the full text. Runs on Apify's schedule, webhook and API infrastructure, so a daily "new mentions of my product" search is a five-minute setup.

Why use Hacker News Search & Front Page Scraper?

• Founders and marketers track every Hacker News mention of their product, company or competitor, with points and comment count to gauge how much it landed.
• Researchers and journalists pull the front page on a schedule to build a historical archive of what topped Hacker News each day.
• Developer-relations teams search show_hn or ask_hn for a technology name to find community feedback and feature requests.
• Sentiment and trend analysis feeds the text field of matched comments into a summarizer or classifier without writing any HN API client code.

How to use Hacker News Search & Front Page Scraper

1. Fill Search queries with one or more search terms, one per line — or leave a row empty (or type front_page) to fetch the current front page.
2. Set Content type to story, comment, show_hn, ask_hn, poll or all to filter which kind of item comes back (ignored for front_page queries).
3. Choose Sort by: relevance (Algolia's default ranking), date (newest first) or points (highest score first).
4. Optionally set Minimum points, an Author username filter, and Only items from the last N days.
5. Click Start and export the dataset as JSON, CSV, Excel or HTML from the Output tab.

Example input

{
    "queries": ["apify", "front_page"],
    "contentType": "story",
    "sortBy": "points",
    "minPoints": 20,
    "maxResults": 50
}

Example output

{
    "objectId": "18064493",
    "type": "story",
    "title": "Show HN: Apify SDK – A scalable web crawling and scraping library for JavaScript",
    "url": "https://github.com/apifytech/apify-js",
    "hnUrl": "https://news.ycombinator.com/item?id=18064493",
    "domain": "github.com",
    "author": "jancurn",
    "points": 78,
    "numComments": 8,
    "createdAt": "2018-09-25T08:24:40.000Z",
    "text": null,
    "storyId": "18064493",
    "storyTitle": null,
    "tags": ["story", "author_jancurn", "story_18064493", "show_hn"],
    "query": "apify",
    "error": null,
    "scrapedAt": "2026-09-13T00:00:00.000Z"
}

Data table

Hacker News Search & Front Page Scraper extracts 16 fields per row:

Field	Type	Description
objectId	string	Hacker News item id, same id the Firebase HN API and news.ycombinator.com use
type	string	story, comment, poll, pollopt or job, derived from Algolia's tags
title, url, domain	string	Story headline, its external link (if any) and the link's hostname
hnUrl	string	The item's own discussion page at news.ycombinator.com/item?id=…
author, points, numComments	string, number	Poster's username, current score and comment count
createdAt	string	When the item was posted (ISO 8601 UTC)
text	string	Comment body, or the self-text of an Ask HN / Show HN post
storyId, storyTitle	string	The parent story a comment belongs to, and its title
tags	array	Raw Algolia tags, e.g. ["story","author_pg","story_18064493"]
query, error, scrapedAt	string	The input entry that produced the row, any failure reason and the run timestamp

Input parameters

Parameter	Type	Default	Description
queries	array	["front_page"]	Search terms, or an empty row / front_page for the current front page
contentType	string	story	story, comment, show_hn, ask_hn, poll or all
sortBy	string	relevance	relevance, date (newest first) or points (highest score first)
minPoints	integer	0	Minimum score to include (0 = no filter)
maxResults	integer	50	Max rows per query entry (1-1000)
author	string	(empty)	Only include items posted by this Hacker News username
sinceDays	integer	0	Only include items posted in the last N days (0 = no filter)

Pricing

Hacker News Search & Front Page Scraper uses pay-per-event pricing: $0.001 per result row, plus a negligible actor-start fee. Fetching the front page (about 30 items) costs about three cents; a search returning 50 stories costs about five cents. Set Maximum cost per run and the Actor trims the work to what the budget covers instead of overspending.

Hacker News Search & Front Page Scraper vs. news.ycombinator.com

The Hacker News website shows one page of results at a time and has no built-in export. This Actor turns any number of search terms — or the live front page — into one flat, typed dataset, ready to schedule, join with other data or feed into a monitoring or sentiment pipeline, all without touching HTML.

Using Hacker News Search & Front Page Scraper with AI agents and MCP

Hacker News Search & Front Page Scraper is pay-per-event with limited permissions — the two requirements for an Actor to be callable through the Apify MCP server at mcp.apify.com. An agent passes a search term or front_page and gets structured story and comment rows back, ready to summarise what's trending or check for new mentions of a product.

FAQ

Does this need a Hacker News or Algolia API key? No. It uses the free, public Algolia Hacker News Search API (hn.algolia.com/api/v1/search), the same API news.ycombinator.com's own search box uses, with no authentication.

Why is text null for most stories? text only holds a value for comments (comment_text) and for Ask HN / Show HN self-posts that include a text body (story_text). A story that just links to an external URL never had a text body.

Why does front_page ignore contentType and sortBy? The Hacker News front page is one specific, already-ranked list of stories. Filtering it by content type would be redundant (it's stories only), and it has no separate "sort by date/points" ranking — it is what it is right now.

Why did my search return fewer rows than maxResults? Algolia returns whatever actually matches the query, tags and filters; a narrow term, a high minPoints, or a short sinceDays window can simply have fewer real hits.

What are the limitations? numComments and points are only meaningful on stories, polls and jobs — comment rows carry points: null since Algolia does not track comment scores. The API is rate-limited; if a run reports HTTP 429, wait a moment and retry.

Is this legal to run? Yes. The Algolia Hacker News Search API is a public, unauthenticated endpoint explicitly built for third-party search integrations, and no personal data beyond publicly posted usernames and comments is collected.

Related Actors

Part of the webdatatools web-intelligence suite — every Actor is pay-per-event, reads public data without a login, and returns one clean row per entity:

Browse the whole suite at webdatatools, or call ten of these Actors straight from Claude, Cursor or Cline with the webdatatools MCP server.

Website & domain intelligence

• Email Extractor — Website Contact & Social Finder — e-mails, phones and social profiles per domain
• Tech Stack Detector — Wappalyzer & BuiltWith Alternative — CMS, e-commerce, analytics, pixels and payments per domain
• Domain DNS & Email Security Checker — SPF, DKIM, DMARC, MX provider, registrar and domain age
• Domain Security Audit (TLS, HTTP headers, redirects, robots) — TLS expiry, security headers, redirect chain, robots and llms.txt
• Subdomain Finder (Certificate Transparency) — every subdomain seen in CT logs, with a live DNS check
• Bulk Core Web Vitals & PageSpeed Audit — Lighthouse scores, LCP, CLS, INP and top fixes per URL
• On-Page SEO Audit — title, meta, headings, links, images and schema issues per page
• Sitemap URL Extractor & Change Monitor — every sitemap URL, or new and removed pages between runs
• Wayback Machine Snapshot & Page Change Tracker — how a page changed over time, or every archived snapshot
• Bulk Domain WHOIS & RDAP Lookup — registrar, dates, status and nameservers per domain
• Web Scraper — CSS Selector & Data Extractor — pull any CSS selector off any page, one row per URL
• Website Screenshot Generator — full-page or viewport PNG/JPEG screenshots of any URL

Content for AI, LLMs and RAG

• AI Web Search & Read: Google results as clean Markdown — a query turned into clean Markdown from the top search results
• Website to Markdown — Content Crawler for LLM & RAG — any site as clean Markdown per page, no browser
• Article & News Extractor (clean text, author, date, markdown) — clean article text, author, date and Markdown per URL
• Structured Data & JSON-LD Extractor (Schema.org, Open Graph) — Schema.org and Open Graph data from any page
• Google News Scraper (RSS search by keyword, topic, site) — news results by keyword, topic or site
• Press Release Monitor: PR Newswire, BusinessWire, GlobeNewswire — PR Newswire, Business Wire and GlobeNewswire releases

Search, video and social

• YouTube Shorts Scraper — Shorts from channels, hashtags and searches with view counts
• Pinterest Pins Scraper — latest pins of public Pinterest profiles and boards
• YouTube Transcript Scraper — captions and subtitles as text + timed segments, per video or channel
• Google Search Results Scraper — SERP API — organic SERP results per keyword and country
• YouTube Comments Scraper — Comments & Replies — comments and replies with likes, no API key
• YouTube Channel Latest Videos (RSS, no API key) — the latest 15 videos of any channel from RSS
• YouTube Channel Scraper (videos, shorts, live) — a channel's full video, shorts and stream list
• YouTube Search Results Scraper (videos, channels, no API key) — videos, channels and playlists per query
• YouTube Video Details Scraper (views, likes, description, tags) — views, likes, description, tags and chapters per video
• Apple Podcasts Lookup & Episodes Scraper — podcast metadata and episodes from iTunes and RSS
• Bluesky Post, Search & Profile Scraper — posts, profiles, followers and threads from the AT Protocol API
• Telegram Channel Posts Scraper — posts, views and media flags from any public channel
• Substack Publication & Posts Scraper — archive, authors and paywall status per publication
• Google Play Reviews Scraper — reviews, ratings, replies and app versions per app
• App Store Reviews Scraper — iOS reviews and ratings per app and country
• Google Trends Scraper — interest over time, by region, and related queries per keyword
• Google Ads Transparency Scraper — ads any advertiser runs on Google, with format and dates
• Keyword Suggestions Scraper (Google, YouTube, Amazon, Bing) — autocomplete keyword ideas from four search engines
• Bilibili Scraper (Videos, Search, Popular) — Chinese video platform: views, likes, coins, danmaku, uploader
• Mastodon Scraper (Hashtags, Accounts, Trending) — public fediverse posts by hashtag, account or trending
• Meetup Events Scraper (Search by Keyword & City) — upcoming events with RSVPs, fees, venues and groups
• Eventbrite Scraper (Events by Keyword & City) — events by keyword and city with venue, dates and organizer

Leads, jobs and company data

• Career Site Jobs API (Greenhouse, Lever, Ashby, Workday +1) — company domains in, their open jobs out, ATS detected automatically
• Workday Jobs Scraper — jobs with full descriptions from any Workday career site
• Google Maps Scraper — businesses with phone, website, address, rating and coordinates per search
• LinkedIn Jobs Scraper — job titles, companies, locations and full descriptions from LinkedIn job search
• Company 360: full company profile from a domain — one row per domain: contacts, tech, security, hiring and company facts
• Hiring Signals Scraper (Greenhouse, Lever, Ashby, Workable) — open jobs and hiring velocity from 10 public ATS boards
• Y Combinator Companies & Founders Scraper — YC startups by batch, industry and hiring status
• Wikidata Entity & Company Enrichment (facts, IDs, links) — HQ, founders, employees, revenue and social IDs per company
• Email Validator & Verifier — Bulk Email Check — syntax, MX, disposable, role and free-provider checks
• OpenStreetMap POI Extractor (Overpass API: shops, amenities) — shops and amenities by radius, bbox or area
• Stock, Crypto & FX Quotes — one row per symbol from Yahoo, Binance and ECB rates
• Remote Jobs Aggregator (RemoteOK, WWR, Hacker News) — one clean row per remote job, de-duplicated across feeds
• Greenhouse Jobs Scraper — jobs with descriptions from any Greenhouse job board
• Lever Jobs Scraper — jobs with descriptions from any Lever careers page
• Ashby Jobs Scraper — jobs, salaries and descriptions from any Ashby job board
• SmartRecruiters Jobs Scraper — jobs with descriptions from any SmartRecruiters company
• Seek Jobs Scraper (Australia & New Zealand) — Seek job ads with salary, work type and location
• Dice Jobs Scraper — US tech jobs from Dice with salary and remote flag
• AutoScout24 Scraper — European car listings with price, mileage and seller
• Rightmove Scraper — UK property for sale or rent with price and agent
• Wellfound Jobs Scraper (AngelList Startup Jobs) — startup jobs with salary and equity ranges, company size and stage
• Yandex Maps Scraper (Places, Ratings, Phones) — businesses in Russia, Türkiye and the CIS with phones, ratings, hours
• Craigslist Scraper (Listings, Prices, Locations) — listings in any area and category with price, date and coordinates
• JobStreet Scraper (Malaysia, Singapore, PH, ID + JobsDB) — JobStreet and JobsDB jobs in 6 Asian countries with parsed salaries
• InfoJobs Scraper (Spain Jobs, Salaries, Companies) — Spanish jobs with salary range, contract type and full description
• Redfin Scraper (Homes for Sale, Prices, Details) — US homes for sale or sold from any Redfin search, with price and details
• Kleinanzeigen Scraper (Ads, Prices, Locations) — German classifieds with price, VB flag, ZIP, city and seller type

Developer, app and research data

• npm, PyPI & Crates.io Package Health Checker — releases, downloads, deprecation and a health score
• GitHub Repository Health & Activity Report — stars, commits, contributors and risk flags per repo
• VS Code Marketplace Extension Scraper (installs, ratings) — installs, ratings and versions per extension
• Chrome Web Store Extension Scraper (installs, ratings) — users, rating, version and developer per extension
• Google Play Scraper — apps, ratings, installs, developer contact and reviews
• App Store (iOS) App Metadata, Ratings & Top Charts Lookup — ratings, price, version and charts per app
• CrossRef DOI & Citation Metadata Lookup — papers, authors, journals and citation counts
• FDA Recalls & Adverse Events Monitor (openFDA) — food, drug and device recalls from openFDA
• iCal / ICS Calendar Feed to Events Extractor — any public calendar feed as event rows
• Shopify Store Products Scraper — catalog, prices, variants and stock per store
• GitHub Trending Repositories Scraper — trending repos and developers by language and period
• Stack Overflow & Stack Exchange Q&A Scraper — questions, answers and scores by query, tag or site
• Bulk Image Downloader — download image URLs to storage with size, dimensions and a ZIP
• Google Flights Scraper (Prices, Airlines, Stops) — flight prices, airlines, times, stops and CO2 by route and date
• Google Hotels Scraper (Prices, Ratings, Reviews) — hotel prices per night, stars, rating and reviews by city and dates
• AliExpress Scraper (Search Products & Prices) — AliExpress search results with USD price, discount and rank
• Lazada Scraper (Products, Prices, Sold, Ratings) — Lazada products in 6 countries with price, rating, units sold and seller

Support and feedback

Found a query shape or content-type edge case it should handle better? Open an issue on the Issues tab.