Techmeme Scraper

Scrape public Techmeme pages into structured technology news stories for monitoring, daily briefings, trend analysis, and research workflows.

Extracted fields

storyId, section, title, sourceName, sourceUrl, articleUrl, summary, imageUrl, relatedLinks, sourcePageUrl, and scrapedAt.

Each dataset item represents one Techmeme story cluster. Publisher names and URLs come from the primary citation; related links are grouped as Techmeme exposes them, such as More, X, LinkedIn, Bluesky, and Forums.

Input

{ "pageUrl": "https://www.techmeme.com/", "maxResults": 50, "includeRelatedLinks": true }

The Actor accepts public Techmeme URLs such as the homepage or /river. Set maxResults from 1 to 200 and disable includeRelatedLinks when you only need primary story fields. It does not scrape article bodies or bypass paywalls.

Pricing

Pay per event: $0.00005 for the Actor start event plus $0.001 per schema-valid Techmeme story. A 50-story run costs approximately $0.05005 before any platform usage outside the event price.

Output

Each dataset row represents a story cluster, not a full article. A page with no parseable clusters produces an empty dataset and a SUMMARY diagnostic rather than fabricated records. Techmeme's public HTML and terms of use control whether and how the data may be reused; respect publisher rights and rate limits.

Use cases

• Monitor public articles and build research or alerting datasets.
• Run a one-off research job and export the structured result as JSON, CSV, Excel, XML, or RSS from Apify.
• Schedule the same input to monitor changes over time and send completed datasets to a webhook or integration.
• Feed schema-shaped records into a database, spreadsheet, BI tool, or AI workflow with the source URL retained for verification.

Run Techmeme Scraper with the Apify API

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('muhammadafzal/techmeme-scraper').call({
  "pageUrl": "https://www.techmeme.com/",
  "maxResults": 50,
  "includeRelatedLinks": true
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

You can also run the Actor from Apify Console, schedules, webhooks, the REST API, Make, Zapier, n8n, or the hosted Apify MCP server.

Reliability and limitations

Public websites and upstream APIs change over time. Start with a small result limit, inspect the dataset and run log, and keep a known-good input for scheduled canary runs. A valid query can return no records when the source has no matches. If a run is blocked, rate-limited, or missing an expected field, reduce concurrency or scope where the input supports it and include the run ID in a support report.

The Actor does not guarantee that every optional field is present on every record. Treat absent values as unavailable from that source response, not as proof that the real-world value does not exist.

Responsible use

Use this Actor only for data you are authorized to access. Follow the target website's terms, robots and access policies, and applicable privacy, database, copyright, anti-spam, and data-protection laws. Do not use it to bypass authentication or other access controls, collect private data, harass people, or make high-impact decisions without independent verification.

Frequently asked questions

Can I schedule Techmeme Scraper?

Yes. Use an Apify schedule to run the same saved input at a chosen interval, then connect a webhook or integration to process the dataset when the run finishes.

How should I test a new input?

Begin with the prefilled example or a small limit. Confirm that the output fields, source coverage, runtime, and live charges match your workflow before increasing the scope.

How do I export the results?

Open the run's default dataset in Apify Console and export JSON, CSV, Excel, XML, or RSS. Applications can retrieve the same records through the Apify API client or REST dataset endpoint.

Can an AI agent call this Actor?

Yes. Add muhammadafzal/techmeme-scraper through the hosted Apify MCP server or call it through the API. The Actor's input and dataset schemas help agents construct valid requests and interpret returned records.

Recommended workflow

1. Define the smallest useful scope. Choose a representative public URL, query, identifier, or filter and keep the first result limit low.
2. Run and inspect. Check the run log, dataset item count, field coverage, source URLs, and live event or usage charges.
3. Validate downstream assumptions. Confirm nullable fields, deduplication keys, timestamps, and any locale-specific formats before importing records into another system.
4. Scale gradually. Increase limits or scheduling frequency only after the small run behaves as expected. Use Apify's maximum-cost and timeout controls to bound large jobs.
5. Monitor changes. Keep a small known-good input as a canary. If the source layout or API changes, compare the new dataset with a previously validated run and report the run ID when requesting support.

For recurring workflows, store the exact Actor input with your pipeline configuration. This makes runs reproducible and helps distinguish a source-data change from an input change.