Pricing

from $1.00 / 1,000 hn item returneds

Hacker News Search Scraper Stories, Comments, Show HN, Ask HN

Search Hacker News stories, comments, Show HN, Ask HN, polls, and jobs by keyword, author, date range, points, and comment count. Full text and engagement metrics. No login.

Pricing

from $1.00 / 1,000 hn item returneds

Rating

0.0

(0)

Developer

Andrew

Actor stats

Bookmarked

Total users

Monthly active users

2 days ago

Last modified

Hacker News Search — Stories, Comments & Developer Sentiment

ryanclinton/hackernews-search

Search and extract stories, comments, polls, Show HN, and Ask HN posts from Hacker News. This actor uses the Algolia HN Search API to find content by keyword, filter by author, date range, minimum points, and comment count -- then returns clean, structured JSON ready for analysis, monitoring, or ...

Ryan Clinton

Hacker News Enhanced Scraper - Stories, Comments & Search

hata1234/hn-scraper

Scrape Hacker News stories, comments, and search results via official Firebase and Algolia APIs. No proxy needed. Supports top, best, new, Ask HN, Show HN, job stories, full-text search, comment extraction, and advanced filtering by points, date, and domain.

Moris Chao

Hacker News Scraper — Stories, Comments & Jobs

cryptosignals/hackernews-scraper

Scrape Hacker News stories, comments, and user profiles — extract title, URL, score, author, comment threads, and submission time. CSV/JSON output.

Web Data Labs

Hackernews Intelligence

viralanalyzer/hackernews-intelligence

Scrape Hacker News stories, comments, and discussions. Track tech trends, startup news, and developer community sentiment.

viralanalyzer

5.0

(3)

Hacker News Scraper

parseforge/hacker-news-scraper

Extract stories, comments, and user data from Hacker News. Browse 6 feed types (Top, New, Best, Ask HN, Show HN, Jobs) or search with filters for points, comments, and date ranges. Get nested comment threads with depth control and author karma scores. Perfect for tech trends monitoring and analysis.

ParseForge

Hacker News Scraper

gentle_cloud/hacker-news-scraper

Scrape Hacker News stories, comments, and user data. Supports top/new/best/ask/show/job story feeds and full-text keyword search via the Algolia API. Extract titles, URLs, scores, authors, comment counts, and timestamps.

Monkey Coder

Hacker News Scraper

automation-lab/hackernews-scraper

Scrape Hacker News stories from any section: front page, newest, Ask HN, Show HN, jobs, and best. Extracts titles, URLs, points, authors, comment counts, and post ages. Great for tech trend detection, content curation, and newsletter creation. Export to JSON, CSV, or Excel.

Stas Persiianenko

Hacker News Scraper - Stories, Comments & AI Digest

harvestlab/hacker-news-scraper

Scrape HN top stories, search, Ask HN, Show HN, job posts and user profiles via Algolia & Firebase APIs. Zero anti-bot, no API key. AI topic digest (themes, trends, TLDR) via 5 LLM providers. x402-ready. $0.001/item.

Nick

🔶 Hacker News Scraper — Stories & Tech Trends

nexgendata/hacker-news-scraper

Scrape Hacker News stories — top, new, best, ask, show, jobs. Engagement tracking, trend analysis, and tech topic monitoring.

Stephan Corbeil

Hacker News Scraper

cloud9_ai/hackernews-scraper

Scrape Hacker News stories, comments, and user profiles via official Firebase API. Get top, new, best, ask, show stories with scores, comments, and author data.

cloud9

FROM apify/actor-node:22 COPY package*.json ./ RUN npm --quiet set progress=false && npm install --include=dev --no-optional COPY tsconfig.json ./ COPY src ./src RUN npm run build && npm prune --omit=dev CMD node dist/main.js

{ "name": "hn-search-scraper", "version": "1.0.0", "type": "module", "main": "dist/main.js", "scripts": { "build": "tsc", "start": "tsc && node dist/main.js", "dev": "tsx src/main.ts" }, "dependencies": { "apify": "^3.0.0", "dotenv": "^17.3.1" }, "devDependencies": { "@types/node": "^20.0.0", "tsx": "^4.0.0", "typescript": "^5.0.0" } }

{ "compilerOptions": { "target": "ES2022", "module": "NodeNext", "moduleResolution": "NodeNext", "lib": ["ES2022"], "outDir": "dist", "rootDir": "src", "strict": true, "esModuleInterop": true, "skipLibCheck": true, "resolveJsonModule": true }, "include": ["src"] }

{ "actorSpecification": 1, "name": "hn-search-scraper", "title": "Hacker News Search Scraper — Stories, Comments, Show HN, Ask HN", "description": "Search Hacker News stories, comments, Show HN, Ask HN, polls, and jobs by keyword, author, date range, points, and comment count. Full text and engagement metrics. No login.", "categories": ["NEWS", "DEVELOPER_TOOLS", "LEAD_GENERATION"], "version": "1.0", "buildTag": "latest", "dockerfile": "./Dockerfile", "input": "./input_schema.json", "output": "./output_schema.json" }

{ "title": "Hacker News Search Scraper Input", "type": "object", "schemaVersion": 1, "properties": { "query": { "title": "Search query", "type": "string", "editor": "textfield", "description": "Keyword(s) to search across titles, URLs, story text, and comment text. Leave empty to fetch the most recent items in the chosen tag.", "prefill": "openai" }, "tag": { "title": "Item type", "type": "string", "editor": "select", "enum": ["any", "story", "comment", "show_hn", "ask_hn", "poll", "job", "front_page"], "enumTitles": ["Any", "Stories", "Comments", "Show HN", "Ask HN", "Polls", "Jobs", "Front Page Stories"], "default": "story", "description": "Restrict results to a single item type. 'Front Page Stories' = stories that have appeared on the HN front page." }, "author": { "title": "Author", "type": "string", "editor": "textfield", "description": "Optional. HN username to restrict results to (case-sensitive). Leave empty to search all authors.", "prefill": "" }, "sortBy": { "title": "Sort by", "type": "string", "editor": "select", "enum": ["relevance", "date"], "enumTitles": ["Relevance (popularity)", "Date (newest first)"], "default": "relevance", "description": "Relevance favors highly-upvoted matches; Date returns the most recent items first." }, "dateAfter": { "title": "Created after (YYYY-MM-DD)", "type": "string", "editor": "textfield", "description": "Optional. Only return items created on or after this UTC date.", "prefill": "" }, "dateBefore": { "title": "Created before (YYYY-MM-DD)", "type": "string", "editor": "textfield", "description": "Optional. Only return items created on or before this UTC date.", "prefill": "" }, "minPoints": { "title": "Minimum points", "type": "integer", "editor": "number", "default": 0, "minimum": 0, "description": "Only return stories with at least this many points. Ignored for comments." }, "minComments": { "title": "Minimum comments", "type": "integer", "editor": "number", "default": 0, "minimum": 0, "description": "Only return stories with at least this many comments. Ignored for comments." }, "maxItems": { "title": "Max items", "type": "integer", "editor": "number", "default": 100, "minimum": 0, "description": "Maximum results to return. 0 = no cap. Algolia caps relevance results at ~1000; date sort can return more." }, "pageId": { "title": "Page ID (pagination)", "type": "string", "editor": "textfield", "description": "Optional. Paste the NEXT_PAGE_ID from a previous run's Key-value store to resume from where you left off." } }, "required": [] }

{ "actorSpecification": 1, "actorOutputSchemaVersion": 1, "title": "Hacker News Search Output", "description": "One HN item per dataset row. Resume cursor stored in the default key-value store.", "type": "object", "properties": { "results": { "title": "Items dataset", "type": "string", "description": "One row per HN item: objectID, type, title, url, author, points, numComments, storyText, commentText, parent story info, createdAt, hnUrl, tags.", "template": "{{links.apiDefaultDatasetUrl}}/items" }, "nextPageId": { "title": "Next page cursor", "type": "string", "description": "NEXT_PAGE_ID record in the default key-value store. Paste into Page ID on the next run to resume; null when results are exhausted.", "template": "{{links.apiDefaultKeyValueStoreUrl}}/records/NEXT_PAGE_ID" } } }

1import "dotenv/config"; 2import { Actor, log } from "apify"; 3const NEXT_PAGE_ID_KEY = "NEXT_PAGE_ID"; 4const VALID_TAGS = new Set(["any", "story", "comment", "show_hn", "ask_hn", "poll", "job", "front_page"]); 5const VALID_SORT = new Set(["relevance", "date"]); 6const TYPE_TAGS = ["story", "comment", "show_hn", "ask_hn", "poll", "pollopt", "job"]; 7const PAGE_SIZE = 100; 8const MAX_PAGES = 1000; 9function deriveType(tags) { 10 for (const t of tags) { 11 if (TYPE_TAGS.includes(t)) 12 return t; 13 } 14 return null; 15} 16function parseDate(raw, endOfDay) { 17 const s = raw.trim(); 18 if (!s) 19 return null; 20 const m = /^(\d{4})-(\d{2})-(\d{2})$/.exec(s); 21 if (!m) 22 return null; 23 const y = parseInt(m[1], 10); 24 const mo = parseInt(m[2], 10) - 1; 25 const d = parseInt(m[3], 10); 26 const ts = endOfDay 27 ? Date.UTC(y, mo, d, 23, 59, 59) 28 : Date.UTC(y, mo, d, 0, 0, 0); 29 if (Number.isNaN(ts)) 30 return null; 31 return Math.floor(ts / 1000); 32} 33function buildTagsFilter(tag, author) { 34 const parts = []; 35 if (tag !== "any") 36 parts.push(tag); 37 if (author) 38 parts.push(`author_${author}`); 39 return parts.length > 0 ? parts.join(",") : null; 40} 41function buildNumericFilters(afterUnix, beforeUnix, minPoints, minComments) { 42 const parts = []; 43 if (afterUnix !== null) 44 parts.push(`created_at_i>=${afterUnix}`); 45 if (beforeUnix !== null) 46 parts.push(`created_at_i<=${beforeUnix}`); 47 if (minPoints > 0) 48 parts.push(`points>=${minPoints}`); 49 if (minComments > 0) 50 parts.push(`num_comments>=${minComments}`); 51 return parts.length > 0 ? parts.join(",") : null; 52} 53function mapHit(h) { 54 const tags = h._tags ?? []; 55 return { 56 objectID: h.objectID, 57 type: deriveType(tags), 58 title: h.title ?? null, 59 url: h.url ?? null, 60 author: h.author ?? null, 61 points: h.points ?? null, 62 numComments: h.num_comments ?? null, 63 storyText: h.story_text ?? null, 64 commentText: h.comment_text ?? null, 65 storyId: h.story_id ?? null, 66 storyTitle: h.story_title ?? null, 67 storyUrl: h.story_url ?? null, 68 parentId: h.parent_id ?? null, 69 createdAt: h.created_at ?? null, 70 createdAtUnix: h.created_at_i ?? null, 71 hnUrl: `https://news.ycombinator.com/item?id=${h.objectID}`, 72 tags, 73 }; 74} 75async function fetchPage(endpoint, query, tagsFilter, numericFilters, page) { 76 const url = new URL(`https://hn.algolia.com/api/v1/${endpoint}`); 77 if (query) 78 url.searchParams.set("query", query); 79 if (tagsFilter) 80 url.searchParams.set("tags", tagsFilter); 81 if (numericFilters) 82 url.searchParams.set("numericFilters", numericFilters); 83 url.searchParams.set("hitsPerPage", String(PAGE_SIZE)); 84 url.searchParams.set("page", String(page)); 85 const response = await fetch(url, { 86 headers: { accept: "application/json" }, 87 signal: AbortSignal.timeout(30_000), 88 }); 89 if (!response.ok) { 90 throw new Error(`Algolia ${response.status}: ${await response.text().catch(() => "")}`); 91 } 92 return (await response.json()); 93} 94await Actor.init(); 95const input = (await Actor.getInput()) ?? {}; 96const query = (input.query ?? "").trim(); 97const tag = (input.tag ?? "story").toLowerCase(); 98if (!VALID_TAGS.has(tag)) { 99 await Actor.fail(`Input 'tag' must be one of: ${[...VALID_TAGS].join(", ")}.`); 100 process.exit(1); 101} 102const sortBy = (input.sortBy ?? "relevance").toLowerCase(); 103if (!VALID_SORT.has(sortBy)) { 104 await Actor.fail(`Input 'sortBy' must be 'relevance' or 'date'.`); 105 process.exit(1); 106} 107const author = (input.author ?? "").trim(); 108const dateAfterRaw = (input.dateAfter ?? "").trim(); 109const dateBeforeRaw = (input.dateBefore ?? "").trim(); 110const afterUnix = dateAfterRaw ? parseDate(dateAfterRaw, false) : null; 111const beforeUnix = dateBeforeRaw ? parseDate(dateBeforeRaw, true) : null; 112if (dateAfterRaw && afterUnix === null) { 113 await Actor.fail(`Input 'dateAfter' must be in YYYY-MM-DD format (got '${dateAfterRaw}').`); 114 process.exit(1); 115} 116if (dateBeforeRaw && beforeUnix === null) { 117 await Actor.fail(`Input 'dateBefore' must be in YYYY-MM-DD format (got '${dateBeforeRaw}').`); 118 process.exit(1); 119} 120const minPoints = Math.max(0, input.minPoints ?? 0); 121const minComments = Math.max(0, input.minComments ?? 0); 122const maxItems = Math.max(0, input.maxItems ?? 100); 123const startPage = input.pageId ? parseInt(input.pageId.trim(), 10) : 0; 124if (Number.isNaN(startPage) || startPage < 0) { 125 await Actor.fail(`Input 'pageId' must be a non-negative integer (got '${input.pageId}').`); 126 process.exit(1); 127} 128const endpoint = sortBy === "date" ? "search_by_date" : "search"; 129const tagsFilter = buildTagsFilter(tag, author); 130const numericFilters = buildNumericFilters(afterUnix, beforeUnix, minPoints, minComments); 131const dataset = await Actor.openDataset(); 132const timeoutAt = Actor.getEnv().timeoutAt; 133const deadlineMs = timeoutAt ? timeoutAt.getTime() - 60_000 : null; 134let pushedTotal = 0; 135let page = startPage; 136let lastGoodPage = startPage; 137let pages = 0; 138log.info(`[hn] query='${query}' tag=${tag}${author ? ` author=${author}` : ""} sort=${sortBy} startPage=${startPage} max=${maxItems || "unlimited"}`); 139try { 140 while (true) { 141 if (maxItems > 0 && pushedTotal >= maxItems) 142 break; 143 if (pages >= MAX_PAGES) { 144 log.warning(`[hn] hit MAX_PAGES safety cap.`); 145 break; 146 } 147 if (deadlineMs && Date.now() > deadlineMs) { 148 log.warning(`[hn] approaching actor timeout. Stopping early; resume with NEXT_PAGE_ID.`); 149 break; 150 } 151 const data = await fetchPage(endpoint, query, tagsFilter, numericFilters, page); 152 const hits = data.hits ?? []; 153 pages++; 154 let pushedThisPage = 0; 155 for (const h of hits) { 156 if (maxItems > 0 && pushedTotal >= maxItems) 157 break; 158 await dataset.pushData(mapHit(h)); 159 pushedTotal++; 160 pushedThisPage++; 161 } 162 log.info(`[hn] page ${page} (${pages} fetched): +${pushedThisPage} (total ${pushedTotal}/${data.nbHits ?? "?"})`); 163 const nbPages = data.nbPages ?? 0; 164 if (hits.length === 0 || page + 1 >= nbPages) { 165 lastGoodPage = null; 166 break; 167 } 168 page++; 169 lastGoodPage = page; 170 } 171} 172catch (err) { 173 log.error(`[hn] failed: ${err.message}. Pushed ${pushedTotal} so far.`); 174} 175await Actor.setValue(NEXT_PAGE_ID_KEY, lastGoodPage === null ? null : String(lastGoodPage)); 176log.info(`[hn] done. items=${pushedTotal} ${NEXT_PAGE_ID_KEY}=${lastGoodPage ?? "null"}`); 177await Actor.exit();

1import "dotenv/config"; 2import { Actor, log } from "apify"; 3 4interface Input { 5 query?: string; 6 tag?: string; 7 author?: string; 8 sortBy?: string; 9 dateAfter?: string; 10 dateBefore?: string; 11 minPoints?: number; 12 minComments?: number; 13 maxItems?: number; 14 pageId?: string; 15} 16 17interface AlgoliaHit { 18 objectID: string; 19 created_at?: string; 20 created_at_i?: number; 21 author?: string; 22 title?: string | null; 23 url?: string | null; 24 story_text?: string | null; 25 comment_text?: string | null; 26 story_id?: number | null; 27 story_title?: string | null; 28 story_url?: string | null; 29 parent_id?: number | null; 30 points?: number | null; 31 num_comments?: number | null; 32 _tags?: string[]; 33} 34 35interface AlgoliaResponse { 36 hits: AlgoliaHit[]; 37 nbHits?: number; 38 page?: number; 39 nbPages?: number; 40 hitsPerPage?: number; 41} 42 43interface ItemRecord { 44 objectID: string; 45 type: string | null; 46 title: string | null; 47 url: string | null; 48 author: string | null; 49 points: number | null; 50 numComments: number | null; 51 storyText: string | null; 52 commentText: string | null; 53 storyId: number | null; 54 storyTitle: string | null; 55 storyUrl: string | null; 56 parentId: number | null; 57 createdAt: string | null; 58 createdAtUnix: number | null; 59 hnUrl: string; 60 tags: string[]; 61} 62 63const NEXT_PAGE_ID_KEY = "NEXT_PAGE_ID"; 64const VALID_TAGS = new Set(["any", "story", "comment", "show_hn", "ask_hn", "poll", "job", "front_page"]); 65const VALID_SORT = new Set(["relevance", "date"]); 66const TYPE_TAGS = ["story", "comment", "show_hn", "ask_hn", "poll", "pollopt", "job"]; 67const PAGE_SIZE = 100; 68const MAX_PAGES = 1000; 69 70function deriveType(tags: string[]): string | null { 71 for (const t of tags) { 72 if (TYPE_TAGS.includes(t)) return t; 73 } 74 return null; 75} 76 77function parseDate(raw: string, endOfDay: boolean): number | null { 78 const s = raw.trim(); 79 if (!s) return null; 80 const m = /^(\d{4})-(\d{2})-(\d{2})$/.exec(s); 81 if (!m) return null; 82 const y = parseInt(m[1], 10); 83 const mo = parseInt(m[2], 10) - 1; 84 const d = parseInt(m[3], 10); 85 const ts = endOfDay 86 ? Date.UTC(y, mo, d, 23, 59, 59) 87 : Date.UTC(y, mo, d, 0, 0, 0); 88 if (Number.isNaN(ts)) return null; 89 return Math.floor(ts / 1000); 90} 91 92function buildTagsFilter(tag: string, author: string): string | null { 93 const parts: string[] = []; 94 if (tag !== "any") parts.push(tag); 95 if (author) parts.push(`author_${author}`); 96 return parts.length > 0 ? parts.join(",") : null; 97} 98 99function buildNumericFilters( 100 afterUnix: number | null, 101 beforeUnix: number | null, 102 minPoints: number, 103 minComments: number, 104): string | null { 105 const parts: string[] = []; 106 if (afterUnix !== null) parts.push(`created_at_i>=${afterUnix}`); 107 if (beforeUnix !== null) parts.push(`created_at_i<=${beforeUnix}`); 108 if (minPoints > 0) parts.push(`points>=${minPoints}`); 109 if (minComments > 0) parts.push(`num_comments>=${minComments}`); 110 return parts.length > 0 ? parts.join(",") : null; 111} 112 113function mapHit(h: AlgoliaHit): ItemRecord { 114 const tags = h._tags ?? []; 115 return { 116 objectID: h.objectID, 117 type: deriveType(tags), 118 title: h.title ?? null, 119 url: h.url ?? null, 120 author: h.author ?? null, 121 points: h.points ?? null, 122 numComments: h.num_comments ?? null, 123 storyText: h.story_text ?? null, 124 commentText: h.comment_text ?? null, 125 storyId: h.story_id ?? null, 126 storyTitle: h.story_title ?? null, 127 storyUrl: h.story_url ?? null, 128 parentId: h.parent_id ?? null, 129 createdAt: h.created_at ?? null, 130 createdAtUnix: h.created_at_i ?? null, 131 hnUrl: `https://news.ycombinator.com/item?id=${h.objectID}`, 132 tags, 133 }; 134} 135 136async function fetchPage( 137 endpoint: "search" | "search_by_date", 138 query: string, 139 tagsFilter: string | null, 140 numericFilters: string | null, 141 page: number, 142): Promise<AlgoliaResponse> { 143 const url = new URL(`https://hn.algolia.com/api/v1/${endpoint}`); 144 if (query) url.searchParams.set("query", query); 145 if (tagsFilter) url.searchParams.set("tags", tagsFilter); 146 if (numericFilters) url.searchParams.set("numericFilters", numericFilters); 147 url.searchParams.set("hitsPerPage", String(PAGE_SIZE)); 148 url.searchParams.set("page", String(page)); 149 150 const response = await fetch(url, { 151 headers: { accept: "application/json" }, 152 signal: AbortSignal.timeout(30_000), 153 }); 154 155 if (!response.ok) { 156 throw new Error(`Algolia ${response.status}: ${await response.text().catch(() => "")}`); 157 } 158 159 return (await response.json()) as AlgoliaResponse; 160} 161 162await Actor.init(); 163 164const input = (await Actor.getInput<Input>()) ?? {}; 165 166const query = (input.query ?? "").trim(); 167const tag = (input.tag ?? "story").toLowerCase(); 168if (!VALID_TAGS.has(tag)) { 169 await Actor.fail(`Input 'tag' must be one of: ${[...VALID_TAGS].join(", ")}.`); 170 process.exit(1); 171} 172 173const sortBy = (input.sortBy ?? "relevance").toLowerCase(); 174if (!VALID_SORT.has(sortBy)) { 175 await Actor.fail(`Input 'sortBy' must be 'relevance' or 'date'.`); 176 process.exit(1); 177} 178 179const author = (input.author ?? "").trim(); 180const dateAfterRaw = (input.dateAfter ?? "").trim(); 181const dateBeforeRaw = (input.dateBefore ?? "").trim(); 182const afterUnix = dateAfterRaw ? parseDate(dateAfterRaw, false) : null; 183const beforeUnix = dateBeforeRaw ? parseDate(dateBeforeRaw, true) : null; 184 185if (dateAfterRaw && afterUnix === null) { 186 await Actor.fail(`Input 'dateAfter' must be in YYYY-MM-DD format (got '${dateAfterRaw}').`); 187 process.exit(1); 188} 189if (dateBeforeRaw && beforeUnix === null) { 190 await Actor.fail(`Input 'dateBefore' must be in YYYY-MM-DD format (got '${dateBeforeRaw}').`); 191 process.exit(1); 192} 193 194const minPoints = Math.max(0, input.minPoints ?? 0); 195const minComments = Math.max(0, input.minComments ?? 0); 196const maxItems = Math.max(0, input.maxItems ?? 100); 197const startPage = input.pageId ? parseInt(input.pageId.trim(), 10) : 0; 198if (Number.isNaN(startPage) || startPage < 0) { 199 await Actor.fail(`Input 'pageId' must be a non-negative integer (got '${input.pageId}').`); 200 process.exit(1); 201} 202 203const endpoint = sortBy === "date" ? "search_by_date" : "search"; 204const tagsFilter = buildTagsFilter(tag, author); 205const numericFilters = buildNumericFilters(afterUnix, beforeUnix, minPoints, minComments); 206 207const dataset = await Actor.openDataset<ItemRecord>(); 208const timeoutAt = Actor.getEnv().timeoutAt; 209const deadlineMs = timeoutAt ? timeoutAt.getTime() - 60_000 : null; 210 211let pushedTotal = 0; 212let page = startPage; 213let lastGoodPage: number | null = startPage; 214let pages = 0; 215 216log.info( 217 `[hn] query='${query}' tag=${tag}${author ? ` author=${author}` : ""} sort=${sortBy} startPage=${startPage} max=${maxItems || "unlimited"}`, 218); 219 220try { 221 while (true) { 222 if (maxItems > 0 && pushedTotal >= maxItems) break; 223 if (pages >= MAX_PAGES) { 224 log.warning(`[hn] hit MAX_PAGES safety cap.`); 225 break; 226 } 227 if (deadlineMs && Date.now() > deadlineMs) { 228 log.warning(`[hn] approaching actor timeout. Stopping early; resume with NEXT_PAGE_ID.`); 229 break; 230 } 231 232 const data = await fetchPage(endpoint, query, tagsFilter, numericFilters, page); 233 const hits = data.hits ?? []; 234 pages++; 235 236 let pushedThisPage = 0; 237 for (const h of hits) { 238 if (maxItems > 0 && pushedTotal >= maxItems) break; 239 await dataset.pushData(mapHit(h)); 240 pushedTotal++; 241 pushedThisPage++; 242 } 243 244 log.info( 245 `[hn] page ${page} (${pages} fetched): +${pushedThisPage} (total ${pushedTotal}/${data.nbHits ?? "?"})`, 246 ); 247 248 const nbPages = data.nbPages ?? 0; 249 if (hits.length === 0 || page + 1 >= nbPages) { 250 lastGoodPage = null; 251 break; 252 } 253 254 page++; 255 lastGoodPage = page; 256 } 257} catch (err) { 258 log.error(`[hn] failed: ${(err as Error).message}. Pushed ${pushedTotal} so far.`); 259} 260 261await Actor.setValue(NEXT_PAGE_ID_KEY, lastGoodPage === null ? null : String(lastGoodPage)); 262 263log.info(`[hn] done. items=${pushedTotal} ${NEXT_PAGE_ID_KEY}=${lastGoodPage ?? "null"}`); 264 265await Actor.exit();

Hacker News Search Scraper Stories, Comments, Show HN, Ask HN

Hacker News Search — Stories, Comments & Developer Sentiment

Hacker News Enhanced Scraper - Stories, Comments & Search

Hacker News Scraper — Stories, Comments & Jobs

Hackernews Intelligence

Hacker News Scraper

Hacker News Scraper

Hacker News Scraper

Hacker News Scraper - Stories, Comments & AI Digest

🔶 Hacker News Scraper — Stories & Tech Trends

Hacker News Scraper

Dockerfile

package.json

tsconfig.json

.actor/actor.json

.actor/input_schema.json

.actor/output_schema.json

dist/main.js

icon/hn-search-scraper.png

src/main.ts

Hacker News Search Scraper Stories, Comments, Show HN, Ask HN

You might also like

Hacker News Search — Stories, Comments & Developer Sentiment

Hacker News Enhanced Scraper - Stories, Comments & Search

Hacker News Scraper — Stories, Comments & Jobs

Hackernews Intelligence

Hacker News Scraper

Hacker News Scraper

Hacker News Scraper

Hacker News Scraper - Stories, Comments & AI Digest

🔶 Hacker News Scraper — Stories & Tech Trends

Hacker News Scraper

Dockerfile

package.json

tsconfig.json

.actor/actor.json

.actor/input_schema.json

.actor/output_schema.json

dist/main.js

icon/hn-search-scraper.png

src/main.ts