Hacker News Api Scraper

Extract top stories, comments, and engagement data from Hacker News without API keys. This scraper uses the official Hacker News Firebase API as primary source with HTML fallback for maximum reliability.

Features

• ✅ Zero Authentication - No API keys or login required
• ✅ Official API First - Uses Hacker News Firebase API
• ✅ HTML Fallback - Alternative frontend mirrors when API is slow
• ✅ Real-time Data - Live stories, comments, and engagement metrics
• ✅ Multiple Story Types - Top, New, Best, Ask HN, Show HN, Jobs
• ✅ Comment Extraction - Optional top comments for each story
• ✅ Fast & Reliable - Requests-based, no browser automation

Use Cases

• Tech Trend Monitoring - Track emerging technologies and discussions
• Content Analysis - Analyze viral content and engagement patterns
• Market Research - Monitor startup launches and product announcements
• Competitive Intelligence - Track industry news and competitor mentions
• Data Journalism - Extract data for research and reporting

Input Parameters

• storyType: Type of stories to scrape (top, new, best, ask, show, job)
• maxResults: Number of stories to extract (1-200)
• includeComments: Whether to fetch top comments for each story
• maxComments: Maximum comments per story (0-10)

Output Fields

Each story includes:

• title: Story title
• url: Story URL
• author: Submitting user
• points: Upvote score
• comments: Number of comments
• rank: Position in list
• type: Story type
• created: Timestamp
• comments_list: Top comments (if enabled)

API Endpoints Used

Primary: https://hacker-news.firebaseio.com/v0/

• /topstories - Top stories
• /newstories - New stories
• /beststories - Best stories
• /askstories - Ask HN stories
• /showstories - Show HN stories
• /jobstories - Job stories
• /item/{id} - Story/comment details

Fallback Mirrors

• https://bhn.vercel.app - Alternative Hacker News frontend
• https://news.ycombinator.com - Official site

Performance

• Speed: ~2-3 seconds for 25 stories
• Reliability: 99.9% uptime with API + fallback
• Cost: $0.003 per result (compute costs only)

Pricing Model

Pay-per-result pricing with no API key requirements:

• Base Cost: $0.003 per story extracted
• Comments: No extra cost when enabled
• Maximum: 200 stories per run

Technical Details

• Architecture: Python requests + BeautifulSoup
• Rate Limiting: Built-in delays to respect HN guidelines
• Error Handling: Automatic fallback API → HTML
• Data Validation: Structured JSON output with type checking

Deployment

Built for Apify platform with:

• Docker containerization
• Automatic scaling
• Built-in monitoring
• Export to JSON/CSV

---

No API keys required - Built with the Reddit hybrid approach for maximum reliability!