Hackaday Articles Scraper

Searches Hackaday — the leading hardware, electronics and maker publication (58,000+ articles back to 2004) — and returns full article records: title, author, categories, tags, image and publish date, straight from its public WordPress API, with optional comment threads.

Public data only. No login, no cookies, no browser, no bot challenge of any kind — 4/4 TLS profiles tested return 200, cold, on both the HTML blog index and the JSON API.

What you get

Four record types share one dataset, told apart by recordType.

POST — one row per article

Raw passthrough of Hackaday's own WordPress REST object (id, date, slug, link, title, excerpt, content, author, categories, tags, featured_media, jetpack_featured_media_url, …) plus flattened article* convenience fields (plain-text title/excerpt, resolved category and author names — not just numeric ids — publish/modified timestamps, image URL).

SEARCH_SUMMARY — one row per query

Upstream's own X-WP-Total count for that exact filtered query, how many rows this run pulled, how many pages it fetched, and every filter that was actually applied (in both slug and resolved-id form) — so a row can be audited without rerunning the query.

COMMENT — one row per comment (opt-in)

Only emitted when Include comments is on: comment author, text, date and the parent comment id for threading, batched across articles rather than one request per article.

ERROR — one row per input that failed

So every query/URL maps to at least one row, even when it fails.

Input

Field	What it does
Search queries	one full-text search per entry — e.g. raspberry pi, esp32
Direct article URLs	fetch specific articles by exact URL instead of searching
Filter by category / tag / author (slugs)	narrow every query — the slug from the term's own URL on hackaday.com, not the display name
Published after / before	date window, YYYY-MM-DD or YYYY-MM-DDTHH:MM:SS (UTC)
Max articles per query	0 = unlimited, up to maxPagesPerQuery
Max pages per query	caps how deep an unlimited crawl goes
Include full article HTML	adds the full rendered article body (~10x payload); off by default
Include comments	adds COMMENT rows, batched; off by default

Leave every query field empty to fetch the latest articles instead — a first-class mode, not an edge case, optionally still narrowed by the category/tag/author/date filters.

Example

{
  "searchQueries": ["raspberry pi", "esp32"],
  "categorySlugs": ["arduino-hacks"],
  "maxItemsPerQuery": 30,
  "includeComments": true,
  "maxCommentsPerArticle": 20
}

Filters are verified, not guessed — and refuse rather than lie

Every filter this actor exposes was A/B tested against Hackaday's live API before being offered, per this portfolio's trap checklist:

• Category/tag/author slugs must resolve to a real term. ?slug= on Hackaday's taxonomy routes answers HTTP 200 with an empty list for a bogus slug — not an error. Silently dropping an unresolvable slug would serve the whole unfiltered 58,000-article archive under a filtered label, so this actor raises instead and the run ends with a named ERROR row (unresolved_term) telling you exactly which slug failed.
• sortBy is a verified 5-value enum, each checked to produce a genuinely different ordering (not two values silently aliasing to the same one). relevance is deliberately not offered as a free-standing choice — upstream only allows it alongside a search query (HTTP 400 otherwise) — so it is applied automatically whenever searchQueries is set, instead of handing you an input combination that always fails.
• Pagination ends honestly. Walked to the literal end of the archive during recon (page 581 of 100/page returns exactly the last 99 rows, matching X-WP-Total to the row) — asking past the last page answers a plain HTTP 400, never a silent re-serve of page 1.
• X-WP-Total is trustworthy, cross-checked during recon against an independent source (the taxonomy term's own count field matched exactly). No display-ceiling caveat to carry here, unlike some other actors in this portfolio.

Notes on reliability

• No WAF. chrome124, firefox133, safari17_0 and chrome99_android all return byte-identical 200s, cold, on the blog index and every JSON endpoint used.
• robots.txt disallows only /wp-admin/ (with admin-ajax.php explicitly allowed back) — no ClaudeBot/anthropic-ai/CCBot/GPTBot group of any kind, and /wp-json/ is untouched. Policy-open.
• Comment fetches are batched, not one request per article — Hackaday's /comments?post=<id1,id2,…> genuinely spans every id in the batch in one call, verified during recon against a bogus post id answering an honest zero (not the site-wide 1.47M-comment firehose).
• Residential proxy, no country pin is the cloud default — no geo-gate observed anywhere in recon.

Known limits

• Full-text search is relevance-based, not exact-phrase. A narrow-looking multi-word query can still match on individual words. A genuinely nonsense query does return a real, verified zero (checked during recon).
• resultsPerRequest is hard-capped at 100 by WordPress core — this is an upstream limit, not a choice this actor makes.