RSS Feed Aggregator — Full Article Content Extraction

Aggregate any number of RSS/Atom feeds in one run and get the full article text, not just the feed summary. Every item is fetched, extracted with trafilatura (high-precision main-content extraction), deduplicated, and written as clean, structured JSON.

Ideal for news monitoring, content curation, AI/RAG pipelines, media intelligence, and SEO research — one scheduled run replaces a self-hosted feed reader plus a scraping stack.

What it does

1. Parses every RSS/Atom feed you give it (news sites, blogs, Substack, Google News queries, podcast feeds — anything with a valid feed).
2. Optionally visits each article URL and extracts the complete readable text, author, date, and site name — stripping navigation, ads, and boilerplate.
3. Deduplicates across feeds by URL or title, so overlapping sources don't produce duplicate rows.
4. Filters by publish date (ISO 8601) and/or feed categories.
5. Pushes one structured item per article to the dataset, ready for export as JSON, CSV, or Excel.

Input

{
  "feedUrls": [
    { "url": "https://hnrss.org/frontpage" },
    { "url": "https://techcrunch.com/feed/" }
  ],
  "maxArticles": 100,
  "extractFullContent": true,
  "deduplicateBy": "link",
  "dateFilter": "2026-07-01"
}

Field	Type	Default	Description
feedUrls	array	required	RSS/Atom feed URLs to aggregate
maxArticles	integer	100	Maximum articles per run (0 = unlimited)
extractFullContent	boolean	true	Visit each article and extract the full text
deduplicateBy	string	"link"	link, title, or none
dateFilter	string	""	Only keep articles published after this ISO 8601 date
includeCategories	string	""	Comma-separated category filter (matches feed categories)
proxyConfiguration	object	Apify proxy	Proxy settings used for article fetching

Output

One dataset item per article:

{
  "title": "Show HN: I built a local-first RSS reader",
  "link": "https://example.com/local-first-rss",
  "guid": "https://example.com/local-first-rss",
  "pubDate": "2026-07-16T08:30:00Z",
  "source": "Hacker News: Front Page",
  "sourceUrl": "https://hnrss.org/frontpage",
  "author": "jane_doe",
  "summary": "A local-first RSS reader with offline sync…",
  "fullText": "For the last six months I've been building…",
  "wordCount": 1240,
  "extractionMethod": "trafilatura",
  "categories": ["programming"],
  "tags": ["rss", "local-first"],
  "enclosures": []
}

All fields are null-tolerant: if an article page can't be fetched you still get the title, link, date, and RSS summary.

Use cases

• News monitoring — merge dozens of sources into one deduplicated stream; schedule hourly and pipe new items to Slack, email, or a webhook with an Apify integration.
• AI training / RAG pipelines — fullText + wordCount give you clean, boilerplate-free text chunks with provenance metadata.
• Content curation — feed newsletters or "best of" pages from niche blogs without opening a feed reader.
• Media intelligence — track competitor blogs and industry publications; filter with dateFilter to process only what's new since the last run.
• SEO research — monitor publications in your niche for content gaps and backlink opportunities.

Run it on a schedule

Create an Apify Schedule (e.g. every hour) pointing at this actor with your feed list, and set dateFilter to roll forward — each run then returns only fresh articles. Combine with the dataset API to pull new rows into your own system:

curl "https://api.apify.com/v2/acts/darknezz~rss-feed-aggregator/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -X POST -H "Content-Type: application/json" \
  -d '{"feedUrls": [{"url": "https://hnrss.org/frontpage"}], "maxArticles": 50}'

Or from Python with the Apify SDK:

from apify_client import ApifyClient

client = ApifyClient("YOUR-APIFY-TOKEN")

run = client.actor("darknezz/rss-feed-aggregator").call(
    run_input={"feedUrls": [{"url": "https://hnrss.org/frontpage"}], "maxArticles": 50}
)
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["title"], "->", item["link"])

Pricing

Free to use on the Apify Store — no pay-per-event charges. You only pay Apify's standard platform compute costs for running it, not per extracted article. No subscription, no monthly minimum.

Companion tools

Need to watch feeds instead of just dumping them? Change & Alert Engine (pay-per-event) watches the same RSS/Atom/JSON URLs on a schedule, diffs each poll against the last seen state, and emits only added / modified / removed rows to the dataset or your webhook — so a one-off aggregation can become a living alert stream.

FAQ

Does it work with Atom feeds / podcast feeds? Yes — anything feedparser understands: RSS 0.9x/1.0/2.0, Atom, RDF. Podcast enclosures land in enclosures.

What if a site blocks full-text extraction? You still get the feed-level data (title, link, summary, date). The item's fullText is null and extractionMethod reflects the fallback — the run does not fail.

Can I aggregate Google News search results? Yes — use a Google News RSS URL (https://news.google.com/rss/search?q=your+query) as a feed.

How is this different from a feed reader? A reader shows you summaries in an app. This actor gives you the complete article text as structured data you can pipe into a database, LLM, or BI tool.

How many feeds can I pass? No hard limit — runs process feeds sequentially; use maxArticles to cap cost on large lists.

Do I need a proxy? No — for most public feeds the default Apify proxy is enough. If a site blocks datacenter IPs, supply your own proxy via proxyConfiguration.

How do I get just the new articles each run? Set dateFilter to the previous run's timestamp (or a rolling window) and deduplicateBy: "link" — the run then returns only items not seen in earlier batches.

Limitations

• Full-text extraction depends on the target site being reachable and parseable; heavily JS-rendered article bodies may yield only the feed summary.
• Feed-level metadata comes from what the publisher exposes in the feed — missing author/date fields stay null.
• Aggregating hundreds of feeds in one run is sequential; for very large pipelines split feeds across runs and merge datasets.

Enjoying the aggregator? A quick review on the Apify Store helps others find it.