Hacker News Scraper

What does hacker-news-scraper do?

hacker-news-scraper uses the shared live-site runtime to find matching articles and article details on news.ycombinator.com. It builds requests from a target-site URL or search query and records only values received from the live page or public response. The output URLs retain the target site as their source. No sample or static records are inserted.

Current setup: ready to request live pages from news.ycombinator.com. The prefilled query is artificial intelligence. Results depend on the current response from that website and may be empty or blocked by the site.

Why use hacker-news-scraper?

Use this actor when you want a repeatable local run for articles on the actor's configured website. The input keeps the actor name, target domain, target type, and query together so the request can be checked before it runs. The crawler can follow discovered item links for detail enrichment within the configured limit. Each record keeps its source URL and collection time for review.

The run depends on the target site's current availability and public response. Some sites limit automated traffic, require a sign-in, or return a verification page. In those cases the run reports a blocked status or no results instead of inventing records.

What data can hacker-news-scraper extract?

The exact fields depend on what the target site exposes for the selected articles. Common output fields are:

Field	Description
actorName	The actor identity recorded for each live item.
sourceUrl	The live target search or listing URL that produced the item.
itemUrl	The matching target-site item URL when one is exposed.
title	The title or name parsed from the live page or response.
description / text	Visible summary or content returned by the target.
searchQuery	The query used for this item, when the run used search.
scrapedAt	The time the live item was collected.
rawMetadata	Additional source fields retained when available.
text	A target-specific value, included when exposed by the live articles source.
author	A target-specific value, included when exposed by the live articles source.
postedDate	A target-specific value, included when exposed by the live articles source.
storyId	Hacker News item identifier returned by its linked search API.
author	Public Hacker News submitter username.
points	Story points returned by the live response.
commentCount	Number of comments reported for the story.
publishedAt	Item creation timestamp.
outboundUrl	Original link submitted to Hacker News.
commentIds	Child comment identifiers returned by the API.
comments	Public child comments returned by the item-detail API.

Type-specific values are retained when they appear in the live response; unavailable fields are left empty or omitted.

How to scrape news.ycombinator.com

1. Open the input settings and confirm the target domain is news.ycombinator.com.
2. Supply a direct URL from that website in startUrls, or edit searchQueries to the exact term you want to search.
3. Set maxItems to control the number of records. Production/schema defaults request 20 valid unique items (up to 50 when explicitly changed). Local input.json explicitly requests 10 for a smoke test.
4. Run npm start from this actor folder.
5. Review the dataset and the run summary under storage/; each output item includes its source URL and timestamp.

Cost and run size

A local run uses your machine, network connection, and the configured request limits. This project does not add a per-result fee. If you run it on a hosted Actor platform, charges depend on the current account plan and the run's actual compute and proxy settings. Check the platform's current pricing before increasing the item count or frequency. Keeping maxItems, maxPages, and detailPages low is useful for a small validation run.

Input

• Local input file: input.json is used for local npm start; the batch runner mirrors it to storage/key_value_stores/default/INPUT.json.
• Actor: hacker-news-scraper (folder: hacker-news-scraper).
• Target type: articles.
• Target domain: news.ycombinator.com.
• Prefilled query: artificial intelligence.
• Direct URLs: add one or more live URLs from the target website to startUrls.
• Local smoke-test limit: 10; production default: 20; maximum allowed by the schema: 50.
• Detail enrichment: detailPages limits how many discovered item pages are opened.

See the input tab for the full configuration. Use a URL and query that belong to the actor's target website; results are filtered to the requested target and query.

Output

The dataset contains live records collected during the current run. Fields are populated only when the target page or response provides them, so two runs can return different items and optional fields. The actor does not seed the dataset with fixed examples. Download or inspect the dataset in JSON, CSV, or another format supported by the runner.

Proxy and retry configuration

Production defaults use the configured public API directly when applicable, automatic Apify Proxy for other websites, or GOOGLE_SERP for supported Google search endpoints. The input tab shows the exact default. Set RESIDENTIAL only when appropriate and permitted by your account/budget. Requests rotate proxy sessions on retry, respect per-host request delays, and use bounded exponential backoff. A failed proxy access check is reported as a failure, not silently bypassed. CAPTCHA and login walls remain access limitations; no fake records replace blocked responses.

Tips and troubleshooting

• Use a direct results URL when the website's search route needs extra filters or location settings.
• If the run reports no results, check that the query matches the target site's own search syntax and that the URL opens publicly.
• If the run reports blocked, review the target site's response and provide a permitted public URL. The crawler will not treat a verification page as a result.
• Keep the detail page limit low when validating a new query, then increase it if the target exposes useful item pages.

Responsible use and support

Use this actor only with pages you are permitted to access and in line with the target website's terms. It is designed for public information and does not request private account data. Public pages can still contain personal information; collect and retain only what you have a legitimate reason to use, and follow applicable privacy rules. For issues, include the actor name, target domain, query or URL, and run summary when reporting a problem.