RSS / XML Scraper

What does RSS XML Feed Scraper do?

RSS XML Feed Scraper collects feed entries from RSS, Atom, RDF, JSON Feed, OPML, and sitemap URLs and writes them to an Apify dataset. Add one or more feed URLs, set an entry limit, and run the Actor. Each run returns feed metadata plus article records with titles, links, dates, authors, tags, media links, and optional full article text.

The Actor also accepts ordinary website URLs. It scans the page for feed links, checks each candidate, and collects the valid feeds. If a previously working feed path starts serving an HTML page, the Actor looks for the publisher's current feed instead of stopping at the first failed request.

Why use RSS XML Feed Scraper?

• One dataset for many publishers. Collect BBC, Guardian, TechCrunch, Reddit, WordPress, and personal blogs in a single run.
• Format detection without setup. RSS, Atom, RDF, JSON Feed, OPML, and sitemaps are recognized from the response, so you do not need to sort URLs by format.
• Recovery when a publisher blocks a request. The Actor retries temporary failures, switches request profiles when a desktop request is rejected, and falls back to a direct request when a proxy fails.
• Article text when you need it. Keep short feed summaries for fast monitoring, or fetch the linked page for full text and clean HTML.
• Scheduled monitoring. Run hourly, daily, or weekly and compare new entries against your stored dataset.

What data can you extract?

Field	Description
item_type	feed_meta for the feed record, entry for an article record
feed_url	Feed that produced the record
title	Feed title or article headline
link	Feed homepage or article URL
description	Teaser or description from the feed
content	Entry content as plain text
content_html	Entry content as HTML when the feed provides it
author / authors	Primary author and author list
published	Publication time in ISO 8601 when parseable
updated	Last update time in ISO 8601 when parseable
tags	Categories and tags from the entry
image_url	Best available image from media fields
enclosure_url	Podcast, video, or attachment URL
full_text	Full article text from the linked page
full_html	Cleaned article HTML
meta_description	Article meta description
top_image	Primary image found on the article page
content_source	feed_payload or article_page
error	Failure reason when a URL cannot be read

Feed records also include language, generator, feed_image, managing_editor, webmaster, entries_count, feed_type, and parsed_at.

Supported feed formats

• RSS 2.0, including content:encoded, dc:creator, media, and enclosure fields.
• Atom, including alternate links, summaries, content, and author lists.
• RDF and RSS 1.0 feeds.
• JSON Feed, including content_html, content_text, authors, and attachments.
• WordPress REST post collections at /wp-json/wp/v2/posts.
• OPML subscription lists.
• XML sitemaps and sitemap indexes.
• Microformats h-feed pages that expose h-entry items in HTML.

How to use RSS XML Feed Scraper

1. Open the Actor on Apify.
2. Add feed URLs or website URLs under Feed or Website URLs.
3. Set Maximum Entries per Feed. Use a small number for a first run.
4. Leave Extract Full Article Content off for fast metadata collection.
5. Run the Actor and download the dataset as JSON, CSV, Excel, or XML.

Input Parameters

Parameter	Type	Required	Default	Description
urls	Array	No	["https://feeds.bbci.co.uk/news/rss.xml"]	Feed URLs or website URLs. One URL per line.
extractContent	Boolean	No	false	Fetch every linked article and add full text and article metadata.
autoExpandSnippets	Boolean	No	true	Fetch the full article when an entry contains only a short summary.
discoverFeeds	Boolean	No	false	Treat website URLs as discovery targets before collecting entries.
maxEntries	Integer	No	20	Maximum entries per feed. Set to 0 to keep every entry.
proxyConfiguration	Object	No	{"useApifyProxy": false}	Optional proxy settings. Apify Proxy stays off by default.

Output Data

Field	Type	Description
item_type	String	Record type: feed_meta or entry
feed_url	String	Source feed URL
title	String	Feed or article title
link	String	Feed homepage or article URL
id	String	Entry ID or GUID from the feed
description	String	Short description from the feed
summary	String	Summary field when the feed provides one
content	String	Entry content as plain text
content_html	String	Entry content as HTML
description_html	String	Description HTML when available
summary_html	String	Summary HTML when available
author	String	Primary author
authors	Array	Author list
creator	String	Creator field from the feed
published	String	Publication date in ISO 8601 when parseable
updated	String	Update date in ISO 8601 when parseable
raw_pub_date	String	Original date string from the feed
raw_iso_date	String	Original ISO date string from the feed
tags	Array	Categories and tags
comments	String	Comment URL when the feed provides one
source_title	String	Source channel title
source_url	String	Source channel URL
enclosure_url	String	Attachment URL
enclosure_type	String	Attachment MIME type
enclosure_length	String	Attachment size
media_content_url	String	Media content URL
media_thumbnail_url	String	Media thumbnail URL
image_url	String	Best available image URL
full_text	String	Full article text
full_html	String	Cleaned article HTML
meta_description	String	Article meta description
keywords	Array	Keywords found by the article extractor
top_image	String	Primary article image
publish_date	String	Article publish date from the linked page
content_source	String	Where full content came from
content_error	String	Full content failure reason, if any
error	String	Feed or URL failure reason, if any
language	String	Feed language
generator	String	Feed generator
feed_image	String	Feed image or logo
managing_editor	String	Managing editor field
webmaster	String	Webmaster field
entries_count	Integer	Total entries available in the feed
feed_type	String	rss, atom, rdf, json, sitemap, or hfeed
parsed_at	String	Feed parse timestamp
collected_at	String	Record collection timestamp

Usage Examples

Collect one feed

{
  "urls": [
    "https://feeds.bbci.co.uk/news/rss.xml"
  ],
  "maxEntries": 20
}

Collect several publishers in one run

{
  "urls": [
    "https://feeds.bbci.co.uk/news/rss.xml",
    "https://www.theguardian.com/world/rss",
    "https://techcrunch.com/feed/",
    "https://hnrss.org/frontpage"
  ],
  "maxEntries": 50
}

Fetch full article text

{
  "urls": [
    "https://www.theguardian.com/world/rss"
  ],
  "extractContent": true,
  "maxEntries": 10
}

Find the feed for a website

{
  "urls": [
    "https://www.nasa.gov"
  ],
  "discoverFeeds": true,
  "maxEntries": 10
}

Keep only new entries for a daily job

Run the same input every day and compare the dataset against your stored records. The Actor removes duplicate entries inside a run, so repeated feed URLs and repeated discovery candidates do not inflate the dataset.

Sample Output

{
  "item_type": "entry",
  "feed_url": "https://feeds.bbci.co.uk/news/rss.xml",
  "title": "Example Article Title",
  "link": "https://www.bbc.co.uk/news/articles/example",
  "id": "https://www.bbc.co.uk/news/articles/example",
  "description": "Short teaser from the feed.",
  "summary": "Short teaser from the feed.",
  "content": "Short teaser from the feed.",
  "author": "Reporter Name",
  "authors": [
    "Reporter Name"
  ],
  "published": "2026-10-22T12:17:44.000Z",
  "raw_pub_date": "Thu, 22 Oct 2026 12:17:44 GMT",
  "tags": [],
  "image_url": "https://example.com/image.jpg",
  "full_text": "Expanded article text from the linked page.",
  "full_html": "<p>Expanded article text from the linked page.</p>",
  "content_source": "article_page",
  "collected_at": "2026-10-22T13:41:48.910Z"
}

Tips for Best Results

• Start with maxEntries at 10 or 20 to check titles, links, and dates.
• Large podcast and archive feeds stay fast because the Actor reads only enough entries to satisfy maxEntries. Set maxEntries to 0 when you need the entire feed.
• Keep extractContent off for monitoring jobs that only need headlines and summaries.
• Turn autoExpandSnippets off when you want feed-only collection with the fastest runtime.
• Use direct feed URLs when you know them. Use discoverFeeds when you only have a homepage.
• Add a proxy only when a publisher rate-limits your runs. Most feeds work without one.
• A missing field usually means the publisher does not include it in the feed. Check several entries before treating it as a failure.

Integrations

• Google Sheets - Send entries to a spreadsheet for editorial or research review.
• Webhooks - Notify internal services when a run finishes.
• Make or Zapier - Route new entries into publishing, alerting, or archiving workflows.
• Apify API - Read the dataset from your own application or scheduler.
• Slack - Post new headlines to a channel through a webhook workflow.

Export formats

• JSON for applications and data pipelines.
• CSV for spreadsheets and quick review.
• Excel for reporting.
• XML for systems that already consume feed-style data.

Frequently Asked Questions

Does this Actor work with Atom and JSON Feed?

Yes. The format is detected from the response, so RSS, Atom, RDF, JSON Feed, OPML, and sitemaps can be mixed in one input list.

Can I collect a feed without knowing its URL?

Yes. Add the website URL and enable discoverFeeds. The Actor scans the page for feed links, checks each candidate, and collects the valid feeds.

What happens if a feed URL returns an HTML page?

The Actor scans that page for feed links. If it finds a valid feed, it processes it. If not, the dataset receives an error record with the URL and reason.

Why are some entries missing full text?

Full text depends on the article page. If the page blocks the request or provides no readable article body, the Actor keeps the feed summary and records the reason in content_error.

Does the Actor remove duplicate entries?

Yes. Entries are deduplicated by feed and article ID or URL within a run. Duplicate feed URLs and discovery candidates are skipped.

Can I run this on a schedule?

Yes. Schedule the Actor in Apify Console and connect the dataset to your downstream workflow. Daily or hourly runs work well for news and blog monitoring.

How many entries can I collect?

There is no fixed cap. maxEntries limits entries per feed, and 0 keeps every entry in the feed. Large sitemap indexes follow the first 25 child sitemaps to keep a run bounded.

Is this Actor legal to use?

The Actor reads publicly available feeds and pages. You are responsible for complying with publisher terms, robots policies, copyright, and applicable law.

Support

For issues or feature requests, use the Issues tab on the Actor page or contact the developer through Apify.

Legal Notice

This Actor is intended for legitimate collection of publicly available feed data. Users are responsible for how the collected content is stored, republished, or redistributed.