Techmeme News Scraper

Turn Techmeme's ranked technology news into structured data for monitoring, analysis, alerts, and newsletters.

Techmeme News Scraper collects each primary story as one clean dataset record. It preserves the editorial context that makes Techmeme useful: the headline, publication, author, excerpt, story cluster, related coverage, thumbnail, and Techmeme's social-post links.

No login or API key is required. The Actor uses lightweight HTTP extraction and works with the current Top News page as well as public Techmeme archive or permalink pages.

What does Techmeme News Scraper do?

The scraper converts Techmeme pages into export-ready records.

It extracts:

• 🥇 Current rank on the requested Techmeme page
• 🧩 Stable Techmeme cluster ID and permalink
• ✍️ Author and publication as separate fields
• 📰 Headline and canonical external article URL
• 📝 Techmeme's editorial excerpt
• 🖼️ Story thumbnail URL when available
• 🔗 Related coverage outlet names and article URLs
• 📣 Techmeme post URLs on X, Threads, Bluesky, and Mastodon
• 🕒 Source-page provenance and scrape timestamp

Each result is charged only after it is saved to the dataset.

Who is it for?

Technology PR and communications teams

Track which announcements reach Techmeme, where they rank, and which publications join the story cluster.

Investors and market analysts

Schedule frequent runs to identify technology narratives, companies, products, and competitive events gaining editorial attention.

Competitive-intelligence teams

Feed ranked headlines and related sources into a warehouse, Slack alert, or internal dashboard.

Newsletter and data-pipeline builders

Use a curated source instead of processing an unranked firehose of individual feeds.

Media researchers

Study how multiple publications cover the same technology story using structured related-coverage links.

Why use this Techmeme scraper?

Techmeme is more than a list of links. It groups reporting into editorial story clusters and highlights the primary narrative.

This Actor keeps that structure intact.

• Fast HTTP-only execution
• No browser overhead
• No account or target-site credentials
• Structured nested related coverage
• Cluster-ID deduplication across multiple pages
• Strict URL validation
• Bounded retries and clear failures
• CSV, JSON, Excel, XML, and RSS-compatible dataset exports
• Scheduling, webhooks, API, and MCP support through Apify

What data can I extract?

Field	Type	Description
rank	number	Story position in the extracted results
clusterId	string	Techmeme story cluster identifier
techmemeUrl	URL	Techmeme cluster permalink
author	string	Primary article author or byline
publication	string	Primary publication name
headline	string	Techmeme primary headline
articleUrl	URL	Canonical primary article link
excerpt	string	Techmeme editorial summary text
thumbnailUrl	URL	Optional Techmeme thumbnail
relatedCoverage	array	Related outlet and article URL objects
relatedCoverageCount	number	Number of included related links
socialPosts	object	X, Threads, Bluesky, and Mastodon URLs
sourcePageUrl	URL	Techmeme page that produced the record
scrapedAt	date	UTC extraction timestamp

Example output

{
  "rank": 1,
  "clusterId": "260713p31",
  "techmemeUrl": "https://www.techmeme.com/260713/p31#a260713p31",
  "author": "Will Shanklin",
  "publication": "Engadget",
  "headline": "Apple releases the first public betas of iOS 27...",
  "articleUrl": "https://www.engadget.com/example-article/",
  "excerpt": "Try the new Siri AI and system-wide performance improvements.",
  "thumbnailUrl": "https://www.techmeme.com/260713/i31.jpg",
  "relatedCoverage": [
    { "outlet": "The Verge", "url": "https://www.theverge.com/example" }
  ],
  "relatedCoverageCount": 1,
  "socialPosts": {
    "x": "https://twitter.com/Techmeme/status/123",
    "threads": "https://www.threads.com/@techmeme/post/example"
  },
  "sourcePageUrl": "https://www.techmeme.com/",
  "scrapedAt": "2026-07-14T00:00:00.000Z"
}

Values change as Techmeme updates its page.

How to scrape Techmeme news

1. Open Techmeme News Scraper on Apify.
2. Keep the prefilled https://www.techmeme.com/ URL for current Top News.
3. Set the maximum number of stories.
4. Choose whether to include related coverage.
5. Click Start.
6. Preview results in the Dataset tab.
7. Export them or connect a webhook, schedule, API client, or integration.

The prefilled input is intentionally small and suitable for a first run.

Input options

Techmeme pages

startUrls accepts public HTTPS URLs on techmeme.com or www.techmeme.com.

Use the homepage for current news or a public archive/permalink page for historical research.

Other hosts are rejected to prevent accidental crawling outside the product scope.

Maximum stories

maxItems accepts 1 to 500.

The Actor stops after saving that many unique story clusters across all requested pages.

Include related coverage

Set includeRelatedCoverage to false when you only need primary stories and want smaller records.

Proxy configuration

Direct HTTP works by default.

A proxy is optional and intended as a fallback if frequent scheduled requests become rate-limited.

Example input

{
  "startUrls": [
    { "url": "https://www.techmeme.com/" }
  ],
  "maxItems": 25,
  "includeRelatedCoverage": true
}

For a lightweight headline feed, use five or ten items and disable related coverage.

How much does it cost to scrape Techmeme?

This Actor uses pay-per-event pricing.

• A small one-time start event covers run initialization.
• An item event is charged for each story successfully stored.
• Item pricing includes automatic Apify subscription-tier discounts.
• Failed pages are not charged as story results.

The live pricing table in the Apify Console is authoritative. Before starting a larger job, use the input preview and maximum-story limit to estimate the result count.

HTTP-only extraction keeps compute usage low compared with browser automation.

Ranking and deduplication

rank reflects the order in which primary stories appear on the requested pages.

When the same cluster appears on multiple input pages, clusterId is used to save it only once during the run.

This makes it safe to combine overlapping Techmeme pages without creating obvious duplicate rows.

The Actor extracts primary .ourh story records. Related articles remain nested under the primary cluster instead of becoming misleading top-level stories.

Related coverage clusters

Techmeme's More section links coverage from other publications.

The Actor represents each related link as:

{
  "outlet": "The Verge",
  "url": "https://www.theverge.com/example"
}

Repeated publication names are preserved because one outlet can publish several distinct articles in the same cluster.

Use relatedCoverageCount for quick filtering and relatedCoverage for deeper source analysis.

Social post URLs

When Techmeme embeds social syndication attributes, the Actor returns URLs for:

• X / Twitter
• Threads
• Bluesky
• Mastodon

Networks with no URL on a story are omitted from the socialPosts object.

These fields are useful for social listening, link verification, and notification workflows.

Scheduling a Techmeme monitor

Apify schedules can run the Actor hourly, daily, or on a custom cron interval.

A common monitoring workflow is:

1. Run every 30 or 60 minutes.
2. Store the output in a named dataset or forward it by webhook.
3. Deduplicate downstream on clusterId.
4. Notify a team when target companies or keywords appear.

Keep the interval respectful and collect only the number of stories your workflow needs.

Integrations

Slack or Microsoft Teams alerts

Trigger a webhook after each successful run and format new clusters as channel messages.

Google Sheets and Excel

Export the dataset as CSV or Excel for editorial planning and media tracking.

Airtable and Notion

Send records through Make or Zapier to maintain a technology-news research database.

Data warehouses

Load JSON results into BigQuery, Snowflake, PostgreSQL, or another warehouse keyed by clusterId.

LLM enrichment

Pass headlines, excerpts, and related publication names to a classification or summarization workflow.

API usage with JavaScript

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/techmeme-news-scraper').call({
  startUrls: [{ url: 'https://www.techmeme.com/' }],
  maxItems: 25,
  includeRelatedCoverage: true,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Python API example

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ['APIFY_TOKEN'])
run = client.actor('automation-lab/techmeme-news-scraper').call(run_input={
    'startUrls': [{'url': 'https://www.techmeme.com/'}],
    'maxItems': 25,
    'includeRelatedCoverage': True,
})
items = client.dataset(run['defaultDatasetId']).list_items().items
print(items)

cURL API example

curl -X POST \
  "https://api.apify.com/v2/acts/automation-lab~techmeme-news-scraper/runs?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://www.techmeme.com/"}],"maxItems":10,"includeRelatedCoverage":true}'

Poll the returned run ID or use the synchronous Apify endpoint when your client can wait for completion.

Use with Apify MCP

Connect the Actor to Claude and other MCP clients through Apify MCP Server:

https://mcp.apify.com?tools=automation-lab/techmeme-news-scraper

Claude Code setup

$claude mcp add --transport http apify "https://mcp.apify.com?tools=automation-lab/techmeme-news-scraper"

Claude Desktop setup

Add an HTTP MCP server in Claude Desktop. If your client uses JSON configuration, use the shared configuration below.

Cursor setup

Open Cursor MCP settings and add the Apify server URL with your Apify authentication.

VS Code setup

Add the same HTTP server in VS Code MCP settings and enable it for your chat workspace.

Desktop, Cursor, and VS Code MCP JSON

{
  "mcpServers": {
    "apify-techmeme": {
      "url": "https://mcp.apify.com?tools=automation-lab/techmeme-news-scraper"
    }
  }
}

Example prompts:

• “Run the Techmeme scraper and list the ten highest-ranked technology stories.”
• “Find story clusters with coverage from at least five publications.”
• “Compare the publications covering today’s AI stories.”
• “Return Techmeme clusters mentioning cybersecurity companies.”

Tips for reliable results

• Use the homepage for current ranked stories.
• Keep maxItems close to the amount you consume.
• Deduplicate scheduled exports with clusterId.
• Preserve sourcePageUrl for auditability.
• Enable related coverage for media analysis.
• Disable it for a compact headline-only feed.
• Use optional proxy settings only after observing rate limits.
• Treat scrapedAt as collection time, not article publication time.
• Expect rankings and excerpts to change as Techmeme updates stories.

Error handling

The Actor retries a page up to three times with short backoff.

It fails clearly when:

• The input URL is not HTTPS Techmeme.
• Techmeme returns an error or unexpected page.
• No primary story records can be parsed.

A loud failure is safer than an apparently successful empty dataset for monitoring workflows.

Malformed or unsupported URLs are not silently ignored.

Data freshness

The homepage reflects Techmeme's current editorial ranking at request time.

The Actor does not cache pages between runs.

scrapedAt records the UTC extraction time for every row.

For change tracking, schedule repeated runs and compare clusterId, rank, headline, and related coverage downstream.

Legality and responsible use

This Actor extracts publicly accessible pages without bypassing login controls.

You are responsible for your use case, request frequency, applicable laws, Techmeme's terms, source-publication rights, and personal-data obligations.

Do not republish copyrighted article text or use the tool to overload the source.

The output contains links and short page-presented metadata; linked publications have their own terms and rights.

FAQ

Does it scrape full article text?

No. It extracts Techmeme's headline, editorial excerpt, and links. It does not visit external publications or bypass paywalls.

Can it scrape historical Techmeme pages?

Yes, when you provide a public Techmeme archive or permalink URL that uses the same story markup.

Why is a publication repeated in related coverage?

One outlet may have several distinct articles in a cluster. Each link is preserved.

Why is thumbnailUrl missing?

Some Techmeme stories do not display an image. The field is optional.

Why did my run return fewer items than maxItems?

maxItems is a ceiling. A page may contain fewer primary stories, and duplicate cluster IDs are removed.

What if a scheduled run is rate-limited?

Reduce frequency first. If needed, configure an Apify proxy in the advanced connection section.

Does scrapedAt equal article publication time?

No. It is the Actor's extraction timestamp. Techmeme does not reliably expose a primary publication timestamp for every story.

Troubleshooting

The run says the page markup changed

Open the input URL in a browser and confirm it is a public Techmeme story page. Retry once. If Techmeme changed its HTML structure, share the failed run with the Actor developer.

My archive URL has no results

Confirm the URL resolves to a Techmeme page containing primary story headlines, not an external article or an empty date.

I see no related links

Make sure includeRelatedCoverage is true. Some stories naturally have no More links.

Related scrapers

Build a broader monitoring stack with other automation-lab Actors:

• Google News Scraper for keyword-driven news discovery
• Hacker News Scraper for community-ranked technology discussions
• Reddit Scraper for subreddit and post monitoring
• RSS Feed Scraper for direct publisher feeds

Choose Techmeme News Scraper when editorial technology-story clustering and related-source context are central to the workflow.

Support

If you encounter an unexpected result, share the Apify run ID and the Techmeme input URL through the Actor issue form.

Do not include private tokens or unrelated sensitive information.

A reproducible run helps diagnose source markup changes quickly.