TechCrunch News Scraper for Startup Headlines and Articles

TechCrunch News Scraper collects recent stories from TechCrunch's public RSS feed and supported WordPress posts source. Each dataset row includes a headline, article link, author, publication date, categories, story identifier, and short summary.

Export results to JSON, CSV, or Excel for startup monitoring, company research, keyword tracking, and scheduled news collection. The RSS feed provides recent items, while the posts source supports pagination for collecting a larger set of stories.

Use Cases

• Startup monitoring: Track recent TechCrunch headlines on a schedule.
• Keyword filtering: Find stories mentioning a company, product, or funding round.
• Research notes: Export headlines, links, publication dates, and authors.
• Historical sampling: Collect more stories using the paginated posts source.
• News pipelines: Pass structured headline data to downstream workflows.

Input Parameters

Parameter	Type	Default	Description
feedUrls	Array	RSS feed and posts source	Supported TechCrunch source URLs. Other hosts or unsupported paths produce error rows.
keyword	String	Empty	Optional filter applied to headlines and short summaries.
maxItems	Integer	100	Maximum number of stories to save per run, from 1 to 1000.
timeoutSecs	Integer	300	Overall run time limit in seconds.
requestTimeoutSecs	Integer	30	Timeout for each download in seconds.

Supported Sources

The actor supports these two TechCrunch sources:

• RSS feed: Provides recent stories, without backward pagination.
• WordPress posts source: Supports page-based collection, allowing more stories to be collected up to the configured item limit.

The posts source processes up to 100 items per page. It continues through subsequent pages until the item cap is reached, a page is empty, or a short page is returned.

Example Input

Replace the placeholders with the supported TechCrunch source URLs.

{
  "feedUrls": [
    "TECHCRUNCH_RSS_FEED_URL",
    "TECHCRUNCH_WORDPRESS_POSTS_URL"
  ],
  "keyword": "",
  "maxItems": 100,
  "timeoutSecs": 300,
  "requestTimeoutSecs": 30
}

What Data Does This Actor Extract?

Each story is saved as a separate dataset row. The headline is stored in itemTitle, and the short blurb is stored in summary. HTML is removed from the summary, which is then trimmed to a maximum of 400 characters.

The actor does not collect full article bodies. RSS category labels are preserved as labels, while category values from the posts source remain in the form supplied by that source.

Example Output

{
  "itemTitle": "Example TechCrunch Story Headline",
  "link": "ARTICLE_URL",
  "author": "Example Author",
  "pubDate": "Sun, 11 Oct 2026 21:14:40 +0000",
  "categories": [
    "AI",
    "Gadgets"
  ],
  "guid": "STORY_IDENTIFIER",
  "summary": "A short summary of the story.",
  "scrapedAt": "2026-10-12T00:00:00+00:00"
}

Output Fields

Field	Type	Description
itemTitle	String	Story headline.
link	String or null	Article URL.
author	String or null	Author name, when available.
pubDate	String or null	Publication date provided by the source.
categories	Array	Category labels or category identifiers supplied by the source.
guid	String	Story identifier used for deduplication.
summary	String or null	Short, plain-text story summary.
scrapedAt	String	UTC timestamp when the row was collected.
error	String	Error description for a failed input.
input	String	URL or value associated with the error.

Error Handling and Deduplication

• Invalid URLs, failed downloads, and HTML error pages produce error rows instead of automatically failing the entire run.
• Valid sources can still be processed when another source fails.
• Duplicate guid values are skipped.
• Missing publication dates or summaries are stored as null when the source does not provide them.

How It Works

1. Reads the source list, optional keyword, item limit, and timeout settings.
2. Validates source URLs against the supported TechCrunch paths.
3. Downloads and parses RSS XML or posts data in JSON format.
4. Extracts story metadata and removes HTML from summary text.
5. Applies the optional keyword filter and skips duplicate story identifiers.
6. Continues through posts pages when applicable, stopping at the configured item cap or the end of available results.
7. Saves story rows and error details to the dataset.

Retries and Request Limits

• Schedule requests to the same source approximately 5–15 minutes apart.
• The actor waits about one second between requests and limits concurrent downloads.
• HTTP 429 and 5xx responses are retried up to three times, with delays of approximately 2, 4, and 8 seconds.
• A suitable Retry-After value is respected when provided, subject to the configured retry behavior.
• The overall run time and individual download timeout are controlled by timeoutSecs and requestTimeoutSecs.

Limits and Terms

This actor collects headlines, article links, publication dates, authors, categories, and short summaries. It does not store full article bodies.

The RSS feed only contains recent stories. Use the paginated posts source when you need a larger collection, subject to the maximum item limit and the content available from the source.

Review TechCrunch's applicable terms before reusing exported content. Copyright, attribution, commercial-use, and other restrictions may apply. You are responsible for ensuring that your use of the data complies with relevant requirements.

FAQ

Can I export results to CSV or JSON?

Yes. Apify dataset results can be downloaded as JSON, CSV, or Excel.

Why does the RSS feed only return recent stories?

The RSS feed provides the latest items and does not support backward pagination. Use the supported posts source for a larger collection, or schedule recurring RSS runs and compare guid values.

What happens if a source URL is incorrect?

The actor saves an error row containing the failed input. Other valid sources can still be processed.

Does the actor download full articles?

No. It saves headline metadata and short summaries rather than full article text.

How many stories can one run collect?

The maxItems setting defaults to 100 and supports a maximum of 1000 stories per run.

How can I avoid duplicate stories across scheduled runs?

Store the guid values from previous exports and compare them with new results before processing stories.

Trademark Notice

TechCrunch and related names, logos, and trademarks belong to their respective owners. Their use in this documentation is solely to identify the source of the data and does not imply affiliation, sponsorship, endorsement, or partnership.

Disclaimer

This actor is an independent tool and is not affiliated with, endorsed by, sponsored by, or officially associated with TechCrunch or its owners. Users are responsible for complying with applicable source terms, rate limits, attribution requirements, and licensing restrictions.

Is Web Scraping Legal?

Collecting information that is already public is generally legal. Scraping is not the same thing as hacking, and there is no single law that bans it. The limits are the usual ones: do not break into accounts, do not take private or confidential data, and be careful with personal data and copyrighted works. A site's terms can also restrict how you access it. Facts such as a public name, price, or rating are not the same thing as republishing someone's article or photo.

This actor collects information from public pages. It does not log in and it does not bypass access controls.

For a longer explanation, see Apify's article Is web scraping legal?.