RSS Feed & XML Sitemap Scraper | Extract Articles, URLs, and Full-Text Content

Parse any RSS or Atom feed and crawl any XML sitemap — including compressed .xml.gz archives — without a browser. Get structured article data, sitemap URLs with SEO metadata, and optionally full article text, all via a fast, memory-efficient scraper built on curl_cffi, feedparser, and trafilatura.

✅ Scrapes RSS, Atom, and XML sitemaps from a single tool

✅ Auto-detects input type — just paste any URL

✅ Extracts full article text via trafilatura

✅ Handles .xml.gz sitemaps and deeply nested sitemap indexes

✅ Exports to JSON, CSV, Excel, XML via Apify Dataset

✅ Fully accessible via the Apify API, Zapier, Make, n8n

---

What Data Can Be Extracted?

RSS / Atom Mode

Field	Description
title	Article headline
link	Direct URL to the article
author	Author name (from <author>or dc:creator)
published_at	Publication date and time
summary	Short excerpt or HTML snippet from the feed
content	Full in-feed content (when provided by the publisher)
full_content	Clean article text extracted from the article page (requires expandArticles: true)
image	Main article image — from media:thumbnail,media:content, inline <img>, or og:image
guid	Unique identifier for the entry
enclosures	Podcast episode audio files and other attachments
feed_title	Name of the RSS feed
source_url	Original feed URL

Sitemap Mode (URL collection only)

Field	Description
url	Page URL
lastmod	Date of last modification
changefreq	How often the page changes
priority	SEO priority hint (0.0–1.0)

Sitemap Mode (with full crawl)

All URL collection fields above, plus:

Field	Description
title	Page title from <title>tag
full_content	Clean article text extracted by trafilatura
image	Main image from og:imagemeta tag
error	Error message if the page could not be crawled

---

Features

• Three scraping modes — rss, sitemap, or auto (detects the right mode from your URLs)
• RSS auto-discovery — provide any website URL; the scraper finds the RSS/Atom feed via <link rel="alternate">
• Recursive sitemap traversal — follows sitemap index files to collect every nested sitemap automatically
• Gzip decompression — handles .xml.gz archives on the fly without loading them entirely into memory
• Stream XML parsing — uses iterparse so even sitemaps with millions of entries stay within RAM limits
• Full-text extraction — visits individual article pages and extracts clean readable text via trafilatura
• OG image extraction — retrieves the main article image from og:image meta tags
• Browser fingerprint rotation — rotates between Chrome and Safari user-agent profiles to reduce detection
• WAF-aware retry logic — automatically retries on 403, 429, 502, 503, and 504 responses with jitter delays
• Concurrency control — limits simultaneous requests (3 for sitemap fetching, 5 for page crawling)
• Proxy support — integrates with Apify Proxy (datacenter or residential) for Cloudflare-protected sites
• Budget protection — set maxItems to cap results; use 0 for unlimited
• Export anywhere — download results as JSON, CSV, Excel, or XML directly from Apify

---

How to Scrape RSS Feeds and XML Sitemaps

1. Open the Actor on Apify and click "Try for free"
2. Paste your URLs into the startUrls field — these can be RSS feed URLs, website homepages (the feed will be discovered automatically), sitemap.xml URLs, or .xml.gz sitemap archives
3. Choose a mode — leave it as auto to let the Actor detect the right mode, or set it to rss or sitemap explicitly
4. Configure options — enable expandArticles to get full article text from RSS items, or crawlSitemapUrls to crawl pages discovered in a sitemap; set maxItems to limit results
5. Run the Actor and wait for it to complete
6. Download your results from the Dataset tab in JSON, CSV, Excel, or XML format, or access them via the Apify API

---

Input Examples

RSS feed — basic

{
  "startUrls": ["https://feeds.bbci.co.uk/news/rss.xml"],
  "mode": "rss",
  "expandArticles": false,
  "maxItems": 50
}

RSS feed — with full article text

{
  "startUrls": [
    "https://techcrunch.com/feed/",
    "https://www.theverge.com/rss/index.xml"
  ],
  "mode": "rss",
  "expandArticles": true,
  "maxItems": 200,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}

Sitemap — collect URLs only (fast, low cost)

{
  "startUrls": ["https://www.example.com/sitemap.xml"],
  "mode": "sitemap",
  "crawlSitemapUrls": false,
  "maxItems": 10000
}

Sitemap — with full page crawl

{
  "startUrls": ["https://www.nytimes.com/sitemaps/new/news.xml.gz"],
  "mode": "sitemap",
  "crawlSitemapUrls": true,
  "maxItems": 500,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"]
  }
}

Auto-detect mode

{
  "startUrls": ["https://news.ycombinator.com/"],
  "mode": "auto",
  "expandArticles": true,
  "maxItems": 30
}

---

Output Examples

RSS mode (without expandArticles)

{
  "title": "OpenAI Releases GPT-5 with Advanced Reasoning",
  "link": "https://techcrunch.com/2025/06/01/openai-gpt5/",
  "author": "Devin Coldewey",
  "published_at": "Mon, 01 Jun 2025 14:32:00 +0000",
  "summary": "<p>OpenAI has announced GPT-5, claiming significant improvements...</p>",
  "content": "",
  "image": "https://techcrunch.com/wp-content/uploads/2025/06/openai-gpt5.jpg",
  "guid": "https://techcrunch.com/?p=2987431",
  "enclosures": [],
  "feed_title": "TechCrunch",
  "source_url": "https://techcrunch.com/feed/"
}

RSS mode (with expandArticles: true)

{
  "title": "OpenAI Releases GPT-5 with Advanced Reasoning",
  "link": "https://techcrunch.com/2025/06/01/openai-gpt5/",
  "author": "Devin Coldewey",
  "published_at": "Mon, 01 Jun 2025 14:32:00 +0000",
  "summary": "<p>OpenAI has announced GPT-5...</p>",
  "content": "",
  "full_content": "OpenAI officially launched GPT-5 on Monday, describing it as the most capable model the company has ever released. The new model shows improvements in multi-step reasoning, coding, and following complex instructions...",
  "image": "https://techcrunch.com/wp-content/uploads/2025/06/openai-gpt5.jpg",
  "guid": "https://techcrunch.com/?p=2987431",
  "enclosures": [],
  "feed_title": "TechCrunch",
  "source_url": "https://techcrunch.com/feed/"
}

Sitemap mode — URL collection only

{
  "url": "https://www.example.com/blog/how-to-scrape-rss-feeds/",
  "lastmod": "2025-05-15",
  "changefreq": "weekly",
  "priority": "0.8"
}

Sitemap mode — with full crawl

{
  "url": "https://www.example.com/blog/how-to-scrape-rss-feeds/",
  "lastmod": "2025-05-15",
  "changefreq": "weekly",
  "priority": "0.8",
  "title": "How to Scrape RSS Feeds with Python",
  "full_content": "RSS (Really Simple Syndication) is one of the oldest and most reliable ways to track updates from websites. In this guide, we walk you through scraping RSS feeds programmatically...",
  "image": "https://www.example.com/images/rss-scraping-guide.png"
}

---

Input Parameters

Parameter	Type	Default	Description
startUrls	string[]	—	URLs to scrape: websites, RSS/Atom feeds, sitemap.xml, or sitemap.xml.gz
mode	string	auto	Processing mode:auto,rss, or sitemap
crawlSitemapUrls	boolean	false	Sitemap mode only — visit each collected URL to extract full text and images
expandArticles	boolean	false	RSS mode only — visit each article URL to extract full text not in the feed
maxItems	integer	100	Maximum results to collect;0= unlimited
proxyConfiguration	object	—	Apify Proxy settings (recommended for Cloudflare-protected sites)

---

Use Cases

Content Aggregation and News Monitoring

Pull articles from dozens of RSS or Atom feeds into one structured dataset. Schedule the Actor to run hourly or daily to keep up with new publications across news sites, industry blogs, and niche media.

LLM and RAG Pipeline Data Preparation

Use sitemap + full-text extraction to build a clean text corpus from any website. The full_content field produced by trafilatura strips ads, navigation, and boilerplate — leaving only the article body, which is exactly what embedding models and vector stores need.

SEO Auditing and URL Discovery

Extract every page URL from a website's sitemap along with lastmod, changefreq, and priority metadata. Use the output to spot stale pages, audit crawl priorities, or feed a list of URLs to another scraper for deeper analysis.

Podcast Metadata Collection

RSS feeds carry structured episode data including enclosures (audio file URLs), descriptions, and publication dates. Scrape multiple podcast feeds to build a searchable episode catalog or power a podcast aggregator.

Competitor Content Research

Monitor a competitor's publishing cadence, topics, and output volume by scraping their RSS feed on a schedule. Track new articles and identify content gaps or trending subjects in your niche.

Full-Site Content Backup and Archiving

Combine sitemap URL collection with full-text crawling to create a structured text snapshot of an entire website — useful for content migrations, audits, or offline archives.

Pre-Building Crawl Queues for Product Scrapers

Use sitemap mode (URL collection only) to gather all product or category page URLs from an e-commerce site's sitemap, then pass that list to a dedicated product scraper. This is faster and cheaper than discovering URLs by crawling.

---

API Access

All results are stored in an Apify Dataset and accessible immediately via the Apify API:

# Run the Actor via API
curl -X POST \
  "https://api.apify.com/v2/acts/YOUR_USERNAME~rss-sitemap-xml-scraper-pro/runs?token=YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "startUrls": ["https://feeds.bbci.co.uk/news/rss.xml"],
    "mode": "rss",
    "expandArticles": true,
    "maxItems": 100
  }'

# Retrieve results from the dataset
curl "https://api.apify.com/v2/datasets/DATASET_ID/items?token=YOUR_API_TOKEN&format=json"

Python:

from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")

run = client.actor("YOUR_USERNAME/rss-sitemap-xml-scraper-pro").call(run_input={
    "startUrls": ["https://www.example.com/sitemap.xml"],
    "mode": "sitemap",
    "crawlSitemapUrls": False,
    "maxItems": 5000,
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

JavaScript:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_API_TOKEN' });

const run = await client.actor('YOUR_USERNAME/rss-sitemap-xml-scraper-pro').call({
  startUrls: ['https://techcrunch.com/feed/'],
  mode: 'rss',
  expandArticles: true,
  maxItems: 200,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

You can also schedule runs from the Apify Console, connect to Zapier or Make without any code, or trigger the Actor via webhooks on completion.

---

Pricing

This Actor uses pay-per-result billing with three charge tiers:

Billing event	When it fires	Relative cost
rss-result	Each RSS/Atom item saved (with or without full text)	Low
sitemap-url	Each URL collected from a sitemap (no crawling)	Low
sitemap-crawl	Each page fully crawled from a sitemap	Higher

Typical scenarios:

• Scraping 100 RSS items without expandArticles — lowest cost, only rss-result events
• Scraping 1,000 sitemap URLs without crawlSitemapUrls — very low cost, only sitemap-url events
• Scraping 500 articles via sitemap with crawlSitemapUrls: true — moderate cost, one sitemap-crawl event per article

To minimize costs: use crawlSitemapUrls: false when you only need a URL list. Set maxItems while testing to avoid unexpectedly large runs.

---

Why Use This Instead of Building Your Own Parser?

Feature	This Actor	DIY feedparser script	Generic web scraper
RSS + Sitemap in one tool	✅	❌	❌
Auto-detect input type	✅	❌	❌
RSS feed auto-discovery from homepage	✅	❌	❌
.xml.gzsitemap support	✅	❌	Rarely
Recursive sitemap index traversal	✅	❌	Rarely
Stream XML parsing (low RAM)	✅	❌	❌
Browser fingerprint rotation	✅	❌	Sometimes
WAF retry with jitter	✅	❌	Sometimes
Full-text extraction (trafilatura)	✅	❌	Sometimes
OG image extraction	✅	❌	Rarely
Proxy integration	✅	Manual	Sometimes
Hosted, scheduled, no infra needed	✅	❌	❌
API + Zapier/Make integrations	✅	❌	❌

---

FAQ

Is it legal to scrape RSS feeds and sitemaps?

RSS feeds and XML sitemaps are publicly published by website owners specifically for machine consumption — RSS for content syndication, sitemaps for search engine crawlers. Scraping them is generally considered acceptable. Always review a site's Terms of Service for specific restrictions and avoid storing personal data in violation of applicable laws.

Do I need proxies?

For most RSS feeds and sitemaps, no proxy is needed. For sites protected by Cloudflare or aggressive rate limiting, enabling Apify Proxy (especially residential IPs) significantly improves reliability.

Can I schedule this Actor to run automatically?

Yes. Apify Schedules let you run the Actor on a cron schedule — hourly, daily, weekly, or any custom interval — without any additional code.

Can I export results to CSV or Excel?

Yes. Apify Datasets support one-click export to JSON, CSV, Excel (XLSX), and XML from the Console, or via the API with a format query parameter.

How many items can I scrape in one run?

There is no hard limit. Set maxItems: 0 for unlimited. For very large sitemaps (100k+ URLs), consider running in batches or using crawlSitemapUrls: false first to inspect the URL list.

Does it work on Cloudflare-protected websites?

Often yes. The Actor rotates browser fingerprints (Chrome 116–120, Safari 15/17) and uses realistic HTTP headers to reduce detection. For stubborn sites, adding residential proxies via proxyConfiguration provides much better results.

Does full-text extraction work on JavaScript-rendered pages?

No. This Actor is browserless — it fetches raw HTML without executing JavaScript. Pages that require JS rendering (single-page apps, dynamically loaded content) may return empty or incomplete full_content. For those, a browser-based actor is a better fit.

Can I scrape multiple RSS feeds or sitemaps in one run?

Yes. Add as many URLs as you need to startUrls. The Actor processes them concurrently.

What happens if a feed or sitemap changes format?

The parser uses feedparser for RSS/Atom (which handles most format variations and quirks) and iterparse for sitemaps. Minor format changes are generally handled automatically. If a site substantially changes its structure, open an issue in the Actor's support section.

Can I use this without a coding background?

Yes. The Actor can be configured and run entirely through the Apify Console UI without writing any code. You can also connect it to Zapier or Make to route results to Google Sheets, Airtable, Slack, or any other app without code.

What is the concurrency limit?

The Actor fetches up to 3 sitemaps simultaneously and crawls up to 5 article pages at the same time. These limits are intentional to avoid triggering rate limits on target websites.

---

SEO Reference

How to Scrape RSS Feed Data

RSS feeds expose structured article data in XML format. This Actor fetches the raw feed, parses every <item> or <entry> element, and maps standard fields (title, link, author, pubDate, description, enclosures) to a normalized output schema. When expandArticles is enabled, it also visits each article URL and extracts clean body text via trafilatura.

XML Sitemap Scraper — API Alternative

Website owners publish sitemaps to help search engines discover all their pages. This Actor reads those same files — including nested sitemap indexes and gzip-compressed archives — and exports every URL with its associated metadata. It's a practical alternative to maintaining custom XML parsing code or paying for an SEO platform subscription when all you need is the URL list.

How Much Does It Cost to Scrape RSS Feeds or Sitemaps?

Cost depends on the number of results and whether full-text crawling is enabled. Collecting URLs from a sitemap (sitemap-url events) is the cheapest operation. Saving RSS items (rss-result events) is similarly low cost. Full page crawling (sitemap-crawl) costs more per item because it involves an additional HTTP request and content extraction step. Setting a maxItems limit is the easiest way to control spend during testing.

Export RSS or Sitemap Data to CSV, Excel, or JSON

All scraped data is saved to an Apify Dataset. From the Console, click the "Export" button and choose CSV, JSON, XLSX, or XML. Via the API, append ?format=csv (or json, xlsx, xml) to the dataset items endpoint. Results can also be piped directly into Google Sheets via the Apify Google Sheets integration.

Automate RSS and Sitemap Data Collection

Use Apify Schedules to run this Actor on a recurring basis — for example, every morning at 6 AM to pull the latest articles from your tracked feeds. Combine with webhooks to trigger downstream workflows (send to a Slack channel, insert into a database, pass to an LLM pipeline) as soon as a run completes.

---

Support

If you encounter an issue or have a feature request, use the Issues tab on the Actor's Apify Store page. Please include the input configuration you used and a sample of any error messages from the run log. For general questions about the Apify platform, visit the Apify Help Center.