Techmeme Tech News Scraper

Fetches Techmeme's curated tech-news river: the headline, author, source publication, original article link and Techmeme's own social-post permalinks (Twitter/X, Mastodon, Threads, Bluesky) for every story currently on the page.

What you get

One SEARCH_SUMMARY row plus one STORY row per story, most-recent-first:

• storyHeadline, storySourceName, storySourceUrl, storyAuthorName
• storyArticleUrl -- the original publisher's article
• techmemeUrl -- Techmeme's own permalink for the story
• storyDate, storyTimeText -- as Techmeme itself labels the story
• techmemeTwitterUrl / techmemeMastodonUrl / techmemeThreadsUrl / techmemeBlueskyUrl -- Techmeme's own social posts about the story

Coverage: current river, not a historical archive

Techmeme's /river page is a rolling snapshot -- at any given time it holds roughly the trailing several days of stories (~100-160 on a normal day). There is no working way to ask for an older window over plain HTTP: the day/date/ts query parameters are silently ignored by upstream (proven during recon: identical response regardless of value), and the dated archive pages that do exist (techmeme.com/YYMMDD/) sit behind Cloudflare on every TLS profile tested. This actor does not offer a date-range input because it could not honestly honor one -- run it on a schedule (e.g. daily) if you want an accumulating history; each run's storyDate/techmemePml make de-duplication straightforward downstream.

Filtering

keywords is applied client-side, after the fetch (case-insensitive substring match against headline/source/author, OR-combined). Techmeme exposes no server-side search or filter on this page, so this is this actor's own post-filter, not something upstream evaluates -- documented here so it isn't mistaken for a real query parameter.

No WAF, no auth

/river answered clean on every TLS profile tested (chrome124, firefox133, safari17_0, chrome99_android), cold, no warmup needed. robots.txt disallows only a handful of tracking/redirect paths (/r2/, /goto/, /search/, etc.) that this actor never touches -- see CRAWLING_METHOD.md.

Known limits

• Sponsor posts and "Featured Podcasts" are excluded -- they use unrelated markup routed through /r2/, which robots.txt disallows.
• No per-story "related coverage" cluster (Techmeme's own site groups a story with the other outlets covering it on the story's own permalink page) -- this actor reads the /river list view only, one row per listed citation, not a second fetch per story.