Hacker News Deep Scraper

Hacker News Deep Scraper

Extract stories, points, authors, comment counts, and timestamps from Hacker News with full metadata. Track tech trends, monitor competitor mentions, or export structured HN data for analysis.

Why use this scraper?

• Export HN front-page and paginated stories as structured JSON
• Track technology trends over time by re-running on a schedule
• Monitor mentions of specific companies, products, or keywords
• Feed downstream research pipelines and data warehouses

How to use

1. Open the Hacker News Deep Scraper in Apify Console
2. Enter one or more start URLs (default: https://news.ycombinator.com)
3. Set maxPages for how many paginated pages to crawl
4. Click Run and watch the Output tab populate in real time

Input

Field	Type	Description	Default
startUrls	array	HN pages or sections to scrape	https://news.ycombinator.com
maxPages	integer	Max paginated pages to follow	10

Output

{
  "type": "story",
  "title": "Show HN: My new project",
  "url": "https://example.com",
  "score": "142",
  "author": "joshua",
  "num_comments": "56",
  "created": "2 hours ago",
  "page": 1,
  "source": "https://news.ycombinator.com",
  "scraped_at": "2026-07-05T17:00:00.000Z"
}

You can download the dataset in JSON, CSV, Excel, or XML from the Apify Console.

Pricing

Pay-per-event: $0.01 per extracted result. A typical front-page crawl costs less than $0.20.

Tips

• Start with maxPages: 3 to validate selectors before scaling
• Combine with the Apify Scheduler to run hourly for trend tracking
• Use the title and url fields as keys for deduplication across runs

FAQ

Is it okay to scrape Hacker News? HN is generally scraper-friendly, but respect rate limits and do not hammer the site with concurrent requests.

Where does the data go? Results are stored in an Apify dataset you can export or connect to Make/Zapier/Google Sheets.