# Hacker News Search Scraper (`fetch_cat/hacker-news-search-scraper`) Actor Search Hacker News stories, comments, Ask HN, Show HN, polls, and authors. Export clean HN result data for trend, product, and market research. - **URL**: https://apify.com/fetch\_cat/hacker-news-search-scraper.md - **Developed by:** [Hanna Nosova](https://apify.com/fetch_cat) (community) - **Categories:** News - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.03 / 1,000 hn result saveds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Hacker News Search Scraper Search Hacker News stories, comments, Ask HN posts, Show HN launches, polls, and author activity. Export clean, structured HN search results for market research, product monitoring, recruiting, developer relations, and trend analysis. ### What does Hacker News Search Scraper do? This actor finds public Hacker News content that matches your query, tags, author, or date range. It returns normalized dataset rows with titles, links, authors, timestamps, points, comment counts, story IDs, parent IDs, cleaned comment text, and tags. Use it when you need repeatable Hacker News monitoring without copying results by hand. ### Who is it for? - 🚀 Founders tracking launch feedback and competitor mentions. - 📈 Investors watching market signals and technical trends. - 🧑‍💻 Developer relations teams monitoring product discussions. - 🧲 Recruiters finding active technical communities and hiring threads. - 📰 Content teams researching developer pain points. - 🔍 Analysts collecting public technology discussion data. ### Why use it? Hacker News discussions are valuable but hard to export consistently. This scraper gives you: - Keyword search across stories and comments. - Tag filters for stories, comments, Ask HN, Show HN, polls, and authors. - Date filtering for recent or historical monitoring. - Clean URLs for both the external page and Hacker News item. - Structured JSON, CSV, Excel, XML, and API access through Apify datasets. ### Common use cases - Track mentions of your startup or product. - Find competitor discussions and launch reactions. - Monitor developer sentiment around a technology. - Build a list of recent Show HN launches in a niche. - Export Ask HN discussions for content research. - Collect Who is Hiring style author activity by tag or user. - Watch security, AI, infrastructure, or open-source trend keywords. ### What data can you extract? | Field | Description | | --- | --- | | `objectId` | Hacker News / search object identifier. | | `type` | Story, comment, poll, job, or unknown. | | `title` | Story title or parent story title. | | `url` | External URL when available. | | `hnUrl` | Direct Hacker News item URL. | | `author` | Hacker News username. | | `createdAt` | Creation timestamp. | | `points` | Story points when available. | | `numComments` | Story comment count when available. | | `storyId` | Parent story identifier. | | `parentId` | Parent item identifier. | | `commentText` | Cleaned comment body for comments. | | `highlight` | Search match context. | | `tags` | Hacker News tags attached to the item. | ### How much does it cost to scrape Hacker News search results? The actor uses pay-per-event pricing. - A small one-time start event is charged per run. - A result event is charged for each Hacker News item saved. - You control cost with `maxItems`. - The default prefill is intentionally small for a cheap first run. For example, set `maxItems` to 25 for a quick sample or 500+ for broader monitoring. ### Input options | Input | Description | | --- | --- | | `query` | Keyword, company, product, domain, or topic. | | `tags` | HN tags such as `story`, `comment`, `show_hn`, `ask_hn`, `poll`, or `author_pg`. | | `includeComments` | Adds matching comments to the export. | | `author` | Convenience filter for a Hacker News username. | | `dateFrom` | Earliest creation date in `YYYY-MM-DD` format. | | `dateTo` | Latest creation date in `YYYY-MM-DD` format. | | `sortBy` | Relevance, date, or points. | | `maxItems` | Maximum number of results to save. | ### Example input ```json { "query": "apify", "tags": ["story"], "includeComments": false, "sortBy": "relevance", "maxItems": 25 } ```` ### Search stories To collect story results for a company or product: 1. Enter your keyword in `query`. 2. Set `tags` to `["story"]`. 3. Choose `sortBy` as `relevance` or `date`. 4. Set `maxItems`. 5. Run the actor and export the dataset. ### Search comments To inspect user discussion and sentiment: 1. Enter your keyword. 2. Use `tags: ["comment"]` or enable `includeComments`. 3. Keep `maxItems` moderate for review workflows. 4. Export `commentText`, `author`, `createdAt`, and `hnUrl`. ### Search Ask HN and Show HN Use tags to focus on specific HN communities: - `ask_hn` for question threads. - `show_hn` for launches and demos. - `poll` for poll posts. - `story` for general submitted stories. - `comment` for comments. ### Search by author Use the `author` input or a tag such as `author_pg`. This is useful for: - Tracking activity by known founders or maintainers. - Reviewing public posts from a company team member. - Finding technical communities around an expert user. ### Date range filtering Use `dateFrom` and `dateTo` to narrow exports. Examples: - `dateFrom: 2026-01-01` for current-year monitoring. - `dateFrom: 2025-06-01`, `dateTo: 2025-06-30` for a monthly report. - Leave both empty to search all available public history. ### Sorting tips - `relevance` is best for keyword research. - `date` is best for monitoring new discussions. - `points` is useful when you want higher-scoring stories first. ### Output example ```json { "objectId": "123456", "type": "story", "title": "Example product launch", "url": "https://example.com", "hnUrl": "https://news.ycombinator.com/item?id=123456", "author": "example_user", "createdAt": "2026-06-01T12:00:00Z", "points": 42, "numComments": 17, "storyId": 123456, "parentId": null, "commentText": null, "highlight": "Example product launch", "tags": ["story"] } ``` ### Integrations Use the dataset in: - Google Sheets for lightweight research workflows. - Airtable for content and lead databases. - Notion for research libraries. - Slack alerts through Apify integrations. - BI dashboards for trend reporting. - CRM enrichment when HN discussions mention target accounts. ### API usage with Node.js ```js import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('fetch_cat/hacker-news-search-scraper').call({ query: 'open source database', tags: ['story'], sortBy: 'date', maxItems: 50 }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); console.log(items); ``` ### API usage with Python ```python from apify_client import ApifyClient client = ApifyClient('APIFY_TOKEN') run = client.actor('fetch_cat/hacker-news-search-scraper').call(run_input={ 'query': 'developer tools', 'tags': ['story'], 'maxItems': 50, }) items = client.dataset(run['defaultDatasetId']).list_items().items print(items[:3]) ``` ### API usage with cURL ```bash curl -X POST 'https://api.apify.com/v2/acts/fetch_cat~hacker-news-search-scraper/runs?token=APIFY_TOKEN' \ -H 'Content-Type: application/json' \ -d '{"query":"apify","tags":["story"],"maxItems":25}' ``` ### MCP usage Use this actor from AI tools through Apify MCP. MCP URL: ```text https://mcp.apify.com/?tools=fetch_cat/hacker-news-search-scraper ``` Claude Code setup: ```bash claude mcp add apify-hacker-news "https://mcp.apify.com/?tools=fetch_cat/hacker-news-search-scraper" ``` MCP JSON configuration example: ```json { "mcpServers": { "apify-hacker-news": { "url": "https://mcp.apify.com/?tools=fetch_cat/hacker-news-search-scraper" } } } ``` Example prompts: - "Search Hacker News for recent discussions about Apify and summarize the main themes." - "Find Show HN posts mentioning vector databases and extract titles, authors, and links." - "Collect comments about browser automation tools from the last year." ### How to get better results - Use specific product names rather than broad categories. - Add date filters for monitoring reports. - Use `comment` results when you need sentiment or pain points. - Use `show_hn` for launch discovery. - Use `ask_hn` for problem and recommendation research. - Increase `maxItems` gradually after validating a sample. ### Limits and notes - Only public Hacker News search data is returned. - Deleted or unavailable items may have fewer fields. - Some comment results have a parent story title instead of their own title. - External URLs are not present for every HN item. - Very broad queries may return many historical results; use `maxItems` and dates to control scope. ### Legality This actor collects public Hacker News search results. Use the data responsibly, respect applicable laws, and avoid using exported data for spam, harassment, or unwanted outreach. If your workflow involves personal data, make sure you have a lawful basis and follow your local privacy requirements. ### Troubleshooting #### Why did I get no results? Try removing restrictive tags or widening the date range. A query with `tags`, `author`, and narrow dates can be too specific. #### Why are points empty on comments? Hacker News points are generally attached to stories. Comment rows may not include points. #### Why is `url` empty? Some HN posts are text posts or comments and do not have an external URL. Use `hnUrl` to open the HN item. ### Related scrapers - [Google News Scraper](https://apify.com/fetch_cat/google-news-scraper) for broader news monitoring. - [Google Autocomplete Scraper](https://apify.com/fetch_cat/google-autocomplete-scraper) for keyword discovery. - [Y Combinator Companies Scraper](https://apify.com/fetch_cat/y-combinator-companies-scraper) for startup/company research. - Explore other data extraction actors from `fetch_cat` on Apify for web, search, social, and marketplace research workflows. ### FAQ #### Can I search only by tag? Yes. Leave `query` empty and provide tags such as `show_hn` or `ask_hn`. #### Can I export comments? Yes. Use `tags: ["comment"]` or enable `includeComments`. #### Can I track a username? Yes. Fill `author` with the HN username or add a tag like `author_pg`. #### Can I schedule this actor? Yes. Use Apify schedules to run daily, weekly, or monthly monitoring jobs. #### Can I export to CSV? Yes. Apify datasets support CSV, JSON, Excel, XML, RSS, and API access. #### Does this require my Hacker News account? No. The actor only uses public search data. # Actor input Schema ## `query` (type: `string`): Keyword, product, company, domain, person, or topic to find on Hacker News. Leave empty for tag/date/author-only searches. ## `tags` (type: `array`): Filter by Hacker News / Algolia tags. Use built-in values or custom tags such as `author_pg`. ## `includeComments` (type: `boolean`): Also include matching comments. Useful for sentiment, pain-point, and lead research. ## `author` (type: `string`): Optional Hacker News username. This is converted to an author tag automatically. ## `dateFrom` (type: `string`): Only include items created on or after this date. Format: YYYY-MM-DD. ## `dateTo` (type: `string`): Only include items created on or before this date. Format: YYYY-MM-DD. ## `sortBy` (type: `string`): Choose relevance, newest first, or client-side ordering by points. ## `maxItems` (type: `integer`): Maximum number of stories/comments to save. Keep the prefill low for a quick first run. ## Actor input object example ```json { "query": "apify", "tags": [ "story" ], "includeComments": false, "sortBy": "relevance", "maxItems": 20 } ``` # Actor output Schema ## `overview` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "query": "apify", "tags": [ "story" ], "maxItems": 20 }; // Run the Actor and wait for it to finish const run = await client.actor("fetch_cat/hacker-news-search-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "query": "apify", "tags": ["story"], "maxItems": 20, } # Run the Actor and wait for it to finish run = client.actor("fetch_cat/hacker-news-search-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "query": "apify", "tags": [ "story" ], "maxItems": 20 }' | apify call fetch_cat/hacker-news-search-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=fetch_cat/hacker-news-search-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Hacker News Search Scraper", "description": "Search Hacker News stories, comments, Ask HN, Show HN, polls, and authors. Export clean HN result data for trend, product, and market research.", "version": "0.1", "x-build-id": "oWGQPnf2NXPfSJQ2K" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/fetch_cat~hacker-news-search-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-fetch_cat-hacker-news-search-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/fetch_cat~hacker-news-search-scraper/runs": { "post": { "operationId": "runs-sync-fetch_cat-hacker-news-search-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/fetch_cat~hacker-news-search-scraper/run-sync": { "post": { "operationId": "run-sync-fetch_cat-hacker-news-search-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "maxItems" ], "properties": { "query": { "title": "Search query", "type": "string", "description": "Keyword, product, company, domain, person, or topic to find on Hacker News. Leave empty for tag/date/author-only searches." }, "tags": { "title": "HN result types / tags", "type": "array", "description": "Filter by Hacker News / Algolia tags. Use built-in values or custom tags such as `author_pg`.", "items": { "type": "string" } }, "includeComments": { "title": "Include comments", "type": "boolean", "description": "Also include matching comments. Useful for sentiment, pain-point, and lead research.", "default": false }, "author": { "title": "Author username", "type": "string", "description": "Optional Hacker News username. This is converted to an author tag automatically." }, "dateFrom": { "title": "Date from", "type": "string", "description": "Only include items created on or after this date. Format: YYYY-MM-DD." }, "dateTo": { "title": "Date to", "type": "string", "description": "Only include items created on or before this date. Format: YYYY-MM-DD." }, "sortBy": { "title": "Sort by", "enum": [ "relevance", "date", "points" ], "type": "string", "description": "Choose relevance, newest first, or client-side ordering by points.", "default": "relevance" }, "maxItems": { "title": "Maximum HN items", "minimum": 1, "maximum": 5000, "type": "integer", "description": "Maximum number of stories/comments to save. Keep the prefill low for a quick first run.", "default": 50 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```