# Hackaday Articles Scraper (`scrapyx/hackaday-articles-scraper`) Actor

Searches Hackaday, the leading hardware/electronics/maker publication, and returns full article records — title, author, categories, tags, image, publish date — from its public WordPress API, with optional comment threads.

- **URL**: https://apify.com/scrapyx/hackaday-articles-scraper.md
- **Developed by:** [Ibnu Adzim](https://apify.com/scrapyx) (community)
- **Categories:** News, Developer tools, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.40 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Hackaday Articles Scraper

Searches **[Hackaday](https://hackaday.com)** — the leading hardware,
electronics and maker publication (58,000+ articles back to 2004) — and
returns full article records: title, author, categories, tags, image and
publish date, straight from its public WordPress API, with optional
comment threads.

Public data only. No login, no cookies, no browser, **no bot challenge of
any kind** — 4/4 TLS profiles tested return 200, cold, on both the HTML
blog index and the JSON API.

### What you get

Four record types share one dataset, told apart by `recordType`.

#### `POST` — one row per article

Raw passthrough of Hackaday's own WordPress REST object (`id`, `date`,
`slug`, `link`, `title`, `excerpt`, `content`, `author`, `categories`,
`tags`, `featured_media`, `jetpack_featured_media_url`, …) plus flattened
`article*` convenience fields (plain-text title/excerpt, resolved category
and author **names** — not just numeric ids — publish/modified timestamps,
image URL).

#### `SEARCH_SUMMARY` — one row per query

Upstream's own `X-WP-Total` count for that exact filtered query, how many
rows this run pulled, how many pages it fetched, and every filter that was
actually applied (in both slug and resolved-id form) — so a row can be
audited without rerunning the query.

#### `COMMENT` — one row per comment (opt-in)

Only emitted when **Include comments** is on: comment author, text, date
and the parent comment id for threading, batched across articles rather
than one request per article.

#### `ERROR` — one row per input that failed

So every query/URL maps to at least one row, even when it fails.

### Input

| Field | What it does |
| --- | --- |
| **Search queries** | one full-text search per entry — e.g. `raspberry pi`, `esp32` |
| **Direct article URLs** | fetch specific articles by exact URL instead of searching |
| **Filter by category / tag / author (slugs)** | narrow every query — the slug from the term's own URL on hackaday.com, not the display name |
| **Published after / before** | date window, `YYYY-MM-DD` or `YYYY-MM-DDTHH:MM:SS` (UTC) |
| **Max articles per query** | `0` = unlimited, up to `maxPagesPerQuery` |
| **Max pages per query** | caps how deep an unlimited crawl goes |
| **Include full article HTML** | adds the full rendered article body (~10x payload); off by default |
| **Include comments** | adds `COMMENT` rows, batched; off by default |

Leave every query field empty to fetch the **latest articles** instead —
a first-class mode, not an edge case, optionally still narrowed by the
category/tag/author/date filters.

#### Example

```json
{
  "searchQueries": ["raspberry pi", "esp32"],
  "categorySlugs": ["arduino-hacks"],
  "maxItemsPerQuery": 30,
  "includeComments": true,
  "maxCommentsPerArticle": 20
}
```

### Filters are verified, not guessed — and refuse rather than lie

Every filter this actor exposes was A/B tested against Hackaday's live API
before being offered, per this portfolio's trap checklist:

- **Category/tag/author slugs must resolve to a real term.** `?slug=` on
  Hackaday's taxonomy routes answers HTTP 200 with an **empty list** for a
  bogus slug — not an error. Silently dropping an unresolvable slug would
  serve the whole unfiltered 58,000-article archive under a filtered
  label, so this actor raises instead and the run ends with a named
  `ERROR` row (`unresolved_term`) telling you exactly which slug failed.
- **`sortBy` is a verified 5-value enum**, each checked to produce a
  genuinely different ordering (not two values silently aliasing to the
  same one). `relevance` is deliberately *not* offered as a free-standing
  choice — upstream only allows it alongside a search query (HTTP 400
  otherwise) — so it is applied automatically whenever `searchQueries` is
  set, instead of handing you an input combination that always fails.
- **Pagination ends honestly.** Walked to the literal end of the archive
  during recon (page 581 of 100/page returns exactly the last 99 rows,
  matching `X-WP-Total` to the row) — asking past the last page answers a
  plain HTTP 400, never a silent re-serve of page 1.
- **`X-WP-Total` is trustworthy**, cross-checked during recon against an
  independent source (the taxonomy term's own `count` field matched
  exactly). No display-ceiling caveat to carry here, unlike some other
  actors in this portfolio.

### Notes on reliability

- **No WAF.** `chrome124`, `firefox133`, `safari17_0` and
  `chrome99_android` all return byte-identical 200s, cold, on the blog
  index and every JSON endpoint used.
- **`robots.txt`** disallows only `/wp-admin/` (with `admin-ajax.php`
  explicitly allowed back) — no ClaudeBot/anthropic-ai/CCBot/GPTBot group
  of any kind, and `/wp-json/` is untouched. Policy-open.
- **Comment fetches are batched**, not one request per article — Hackaday's
  `/comments?post=<id1,id2,…>` genuinely spans every id in the batch in one
  call, verified during recon against a bogus post id answering an honest
  zero (not the site-wide 1.47M-comment firehose).
- **Residential proxy, no country pin** is the cloud default — no geo-gate
  observed anywhere in recon.

### Known limits

- **Full-text search is relevance-based, not exact-phrase.** A
  narrow-looking multi-word query can still match on individual words. A
  genuinely nonsense query does return a real, verified zero (checked
  during recon).
- **`resultsPerRequest` is hard-capped at 100** by WordPress core — this is
  an upstream limit, not a choice this actor makes.

# Actor input Schema

## `searchQueries` (type: `array`):

One search per entry, each with its own SEARCH\_SUMMARY row — e.g. `raspberry pi`, `esp32`. This is WordPress full-text search across title/content, not exact-phrase matching. Leave empty (with 'Post URLs' also empty) to fetch the latest articles instead, optionally narrowed by the category/tag/author/date filters below.

## `postUrls` (type: `array`):

Full hackaday.com article URLs (e.g. https://hackaday.com/2026/08/17/some-slug/) to fetch by exact match instead of searching. Each becomes its own POST row (plus COMMENT rows if enabled below). Combine freely with 'Search queries'.

## `categorySlugs` (type: `array`):

Restrict every query to these categories, e.g. `raspberry-pi-2`, `arduino-hacks`, `3d-printer-hacks` — the slug from the category's own URL on hackaday.com, not the display name. A slug that does not exist on the site fails the run with an ERROR row naming the bad slug, rather than silently returning the unfiltered archive.

## `tagSlugs` (type: `array`):

Restrict every query to these tags, e.g. `esp32`, `linux`, `3d-printing` — the slug from the tag's own URL. Same fail-loud behaviour as category slugs on an unresolvable value.

## `authorSlugs` (type: `array`):

Restrict every query to these authors, e.g. `mayaposch` — the slug from /author/<slug>/ on hackaday.com. Same fail-loud behaviour as category/tag slugs.

## `publishedAfter` (type: `string`):

Only articles published on or after this date/time. Format YYYY-MM-DD or YYYY-MM-DDTHH:MM:SS (UTC). Leave empty for no lower bound.

## `publishedBefore` (type: `string`):

Only articles published before this date/time. Format YYYY-MM-DD or YYYY-MM-DDTHH:MM:SS (UTC). Leave empty for no upper bound.

## `sortBy` (type: `string`):

How results are ordered. Ignored (upstream forces relevance ordering) whenever a search query is non-empty — set here it only affects filter-only / latest-articles runs.

## `sortOrder` (type: `string`):

Ascending or descending for the 'Sort by' field above.

## `maxItemsPerQuery` (type: `integer`):

Stop after this many articles per search query / URL / latest-articles run. Set to 0 for unlimited (walks every page 'Max pages per query' allows).

## `maxPagesPerQuery` (type: `integer`):

Caps how deep the crawl walks per query, even when 'Max articles per query' is unlimited. Each page holds up to 'Results per request' articles.

## `resultsPerRequest` (type: `integer`):

Articles fetched per HTTP request (upstream's own per\_page parameter). Hard-capped at 100 by the API — any higher value fails the run.

## `includeArticleContent` (type: `boolean`):

The full rendered article body HTML, in addition to title/excerpt/author/categories/tags/image. On by default -- full article content is the main reason to use this actor. Set to false only to save payload weight when you genuinely just need headlines/metadata.

## `includeComments` (type: `boolean`):

Fetch each article's comments as separate COMMENT rows (batched, not one request per article). Off by default since most articles carry zero-to-dozens of comments and many use cases only need the articles themselves.

## `maxCommentsPerArticle` (type: `integer`):

Only used when 'Include comments' is on. Set to 0 for unlimited (all comments on the article).

## `maxConcurrency` (type: `integer`):

Upper bound on requests in flight at once, across all queries, taxonomy lookups and comment fetches.

## `minRequestInterval` (type: `number`):

Shared pacing floor across all requests this run makes, independent of 'Max concurrent requests'. robots.txt sets no Crawl-delay for this host; the default is this portfolio's own politeness baseline, not a site requirement.

## `proxyConfiguration` (type: `object`):

Apify Proxy on the shared datacenter pool. This is the default because it is included in your plan at no extra cost and this target works through it. If you start seeing blocks, challenges or empty results, switch the group here to Residential -- it uses real consumer IPs and gets through more, but Apify bills residential traffic per gigabyte, so leave it off unless you need it.

## Actor input object example

```json
{
  "searchQueries": [
    "raspberry pi"
  ],
  "postUrls": [],
  "categorySlugs": [],
  "tagSlugs": [],
  "authorSlugs": [],
  "publishedAfter": "",
  "publishedBefore": "",
  "sortBy": "date",
  "sortOrder": "desc",
  "maxItemsPerQuery": 25,
  "maxPagesPerQuery": 10,
  "resultsPerRequest": 25,
  "includeArticleContent": true,
  "includeComments": false,
  "maxCommentsPerArticle": 50,
  "maxConcurrency": 5,
  "minRequestInterval": 0.25,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `items` (type: `string`):

One row per scraped record. See the dataset's default view for field definitions.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "searchQueries": [
        "raspberry pi"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("scrapyx/hackaday-articles-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "searchQueries": ["raspberry pi"] }

# Run the Actor and wait for it to finish
run = client.actor("scrapyx/hackaday-articles-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "searchQueries": [
    "raspberry pi"
  ]
}' |
apify call scrapyx/hackaday-articles-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,scrapyx/hackaday-articles-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/uPhafgFOQOL1Y1OMR/builds/9Wf4i5XhREbOclFR7/openapi.json