HackerNoon Scraper

Extract articles from HackerNoon[https://hackernoon.com/] across 22 different categories (AI, Web3, Business, Finance and many more). Discover high quality tech stories from +45000 contributing writers.

🚀 Features

• Multi-Category Support: Scrape from 22 different HackerNoon categories
• Rich Data Extraction: Extracts comprehensive article metadata including content, author information, images, and engagement metrics
• Configurable Limits: Set maximum number of articles to scrape

📊 Supported Categories

The scraper supports all major HackerNoon categories:

• Technology: AI, Programming, Tech Companies, Tech Stories, Cybersecurity, Cloud, Data Science
• Business: Business, Finance, Startups, Management, Product Management
• Lifestyle: Life Hacking, Remote Work, Gaming, Writing
• Science & Innovation: Science, Futurism, Web3
• Community: HackerNoon, Society, Media
• Special: Top Stories (trending articles)

📋 Data Fields Extracted

Each scraped article includes the following information:

Article Metadata

• id - Unique article identifier
• title - Article title
• slug - URL slug
• link - Full article URL
• excerpt - Article excerpt/summary
• tldr - Too Long; Didn't Read summary
• articleBody - Full article content
• createdAt - Publication date and time
• parentCategory - Primary category
• tags - Array of article tags
• commentsCount - Number of comments
• pageViews - Number of page views/reads
• arweave - Arweave blockchain reference

Media Information

• mainImage - Main article image URL
• mainImageHeight - Main image height (pixels)
• mainImageWidth - Main image width (pixels)
• socialPreviewImage - Social media preview image

Author Information

• author_name - Author display name
• author_handle - Author username/handle
• author_avatar - Author profile picture URL
• author_bio - Author biography
• author_isBrand - Whether the author is a brand account
• author_isTrusted - Whether the author is verified/trusted

⚙️ Configuration

Input Parameters

The scraper accepts the following input parameters:

{
  "category": "AI",
  "max_posts": 100
}

• category (required): Choose from available categories

  • Default: "Top Stories"
  • Options: See full list of supported categories above

• max_posts (optional): Maximum number of articles to scrape

  • Default: No limit (up to 500 articles max)
  • Range: 50-500
  • ⚠️ Note: For "Top Stories" category, maximum output is limited to 150 articles due to the time it takes to render compared to the other categories.

📊 Output Format

The scraper outputs data in JSON format, with each article as a separate record:

{
  "id": "rxhvxiLNxsRwnMGivMFc",
  "title": "The Future of AI in Healthcare",
  "slug": "the-future-of-ai-in-healthcare",
  "link": "https://hackernoon.com/the-future-of-ai-in-healthcare",
  "excerpt": "Exploring how AI is revolutionizing medical diagnosis...",
  "tldr": "AI is transforming healthcare through improved diagnostics and personalized treatment.",
  "articleBody": "Full article content here...",
  "createdAt": "2023-09-30T21:36:56.367Z",
  "mainImage": "https://hackernoon.imgix.net/images/...",
  "mainImageHeight": 1024,
  "mainImageWidth": 1536,
  "socialPreviewImage": "https://hackernoon.imgix.net/images/...",
  "parentCategory": "ai",
  "tags": ["artificial-intelligence", "healthcare", "machine-learning"],
  "commentsCount": 15,
  "pageViews": 348080,
  "arweave": "QuKn6Hew8wrwpJ9Zt0OFoeVt5yQwBQyZf30TtejOOno",
  "author_name": "Dr. Sarah Johnson",
  "author_handle": "sarahj_ai",
  "author_avatar": "https://cdn.hackernoon.com/images/...",
  "author_bio": "AI researcher and healthcare innovation expert",
  "author_isBrand": false,
  "author_isTrusted": true
}