# RSS Feed Scraper & Keyword Monitor (`sturdydata/rss-feed-monitor`) Actor

Turn any list of RSS/Atom feeds — or just website homepages, feeds are auto-discovered — into a clean, deduplicated stream of news items, optionally filtered by keywords. Normalized output across RSS 2.0 and Atom: title, link, published date, author, categories, summary and full content as plain...

- **URL**: https://apify.com/sturdydata/rss-feed-monitor.md
- **Developed by:** [Sturdy Data](https://apify.com/sturdydata) (community)
- **Categories:** News
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage

## What's an Apify Actor?

Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md):

```bash
npm install apify-client
```

In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md):

```bash
pip install apify-client
```

In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md):

````bash
# MacOS / Linux
curl -fsSL https://apify.com/install-cli.sh | bash
# Windows
irm https://apify.com/install-cli.ps1 | iex
```bash

In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md).

If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).


# README

## RSS Feed Scraper & Keyword Monitor

Turn any list of RSS/Atom feeds — or just website homepages — into a **clean, deduplicated, keyword-filtered stream of news items** with one normalized schema.

Built for: brand monitoring, competitor tracking, niche news aggregation, content pipelines, and feeding fresh articles to LLM/RAG systems.

### Why this monitor

News scrapers on the marketplace are mostly $20+/month rentals that wrap a single source and break when it changes. Feeds don't break — they are *designed* for programmatic consumption. This actor makes them effortless:

- **Feed auto-discovery.** Don't know the feed URL? Give the homepage; feeds are found via `<link rel="alternate">` tags, with fallback probing of common paths (`/feed`, `/rss.xml`, `/atom.xml`...).
- **One schema for RSS 2.0 and Atom.** Title, link, GUID, ISO-8601 published date, author, categories, summary and full content as plain text — no XML wrangling on your side.
- **Keyword filtering server-side.** Only pay for items that mention your keywords; matches are recorded per item.
- **Cross-feed deduplication.** Overlapping feeds (aggregators, category feeds) don't produce duplicate items.
- **Pay per item, no rental.** Run it hourly on a schedule for pennies.

### Input

| Field | Type | Default | Notes |
|---|---|---|---|
| `feedUrls` | array | `[]` | Direct RSS/Atom URLs. |
| `siteUrls` | array | `[]` | Homepages — feeds auto-discovered. |
| `keywords` | array | `[]` | Keep only items matching ≥1 keyword (case-insensitive). Empty = keep all. |
| `maxItemsPerFeed` | integer | `50` | Cap per feed. |
| `includeContent` | boolean | `true` | Full content text when the feed provides it. |
| `deduplicate` | boolean | `true` | Drop repeated links/GUIDs within the run. |

```json
{
    "feedUrls": ["https://techcrunch.com/feed/"],
    "siteUrls": ["https://www.theverge.com"],
    "keywords": ["insurance", "climate"],
    "maxItemsPerFeed": 50
}
````

### Output schema

One dataset item per feed entry:

```json
{
    "feedUrl": "https://techcrunch.com/feed/",
    "feedTitle": "TechCrunch",
    "siteUrl": null,
    "title": "Insurtech startup raises $40M Series B",
    "url": "https://techcrunch.com/2026/07/03/insurtech-startup...",
    "guid": "https://techcrunch.com/?p=2751...",
    "publishedAt": "2026-07-03T14:02:11.000Z",
    "author": "Jane Reporter",
    "categories": ["Fintech", "Insurance"],
    "summaryText": "The round was led by...",
    "contentText": "Full article text when the feed publishes it...",
    "matchedKeywords": ["insurance"],
    "fetchedAt": "2026-07-04T10:00:00.000Z"
}
```

### Pricing

**Pay per result. 1 result = 1 feed item kept** (after keyword filtering and deduplication — filtered-out items cost nothing).

| Items | Cost at $0.50 / 1,000 |
|---|---|
| 1,000 | $0.50 |
| 20,000 (50 feeds, hourly, a day) | $10.00 |

Compare: single-source news actors at $20/month rental *before* usage.

### Run summary

Every run writes a `RUN_SUMMARY` record to the key-value store: per-feed item counts and errors, per-site discovery counts and `robotsBlocked` status, total items pushed, and feed failure count. No silent zero-result runs — if a feed 404s, times out, or is blocked by robots.txt, it shows up here with the reason.

### Scheduling

Pair with an Apify Schedule (e.g. every 30 min) and a webhook/integration (Slack, Google Sheets, Make, Zapier) for a complete monitoring pipeline. `deduplicate` works within a run; for cross-run dedup, key on `url` in your destination.

### Compliance & fair use

RSS/Atom feeds exist to be fetched programmatically. One polite request per feed per run, standard retries only. No paywall circumvention — the actor returns what the publisher chose to publish in the feed. For `siteUrls`, the site's `robots.txt` is checked first — the homepage fetch and every auto-discovery probe path (`/feed`, `/rss.xml`, etc.) is skipped if disallowed, and this is recorded per site in the run summary.

### Using from API / MCP / AI agents

```bash
curl -X POST "https://api.apify.com/v2/acts/<ACTOR_ID>/run-sync-get-dataset-items?token=<TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{ "feedUrls": ["https://hnrss.org/frontpage"], "keywords": ["postgres"] }'
```

Works with the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp) — agents can ask "what's new about X across these sources" in one tool call.

### FAQ

**Full article text or just the feed content?** What the feed publishes. Many feeds carry full `content:encoded`; truncated feeds carry summaries. Fetching and parsing article pages beyond the feed is a different (heavier) job — pipe `url` into the **Website to Markdown** actor if you need it.

**Google News?** Its robots.txt disallows automated fetching of its RSS endpoints, so this actor does not special-case it. Use publisher feeds directly — they are richer anyway.

### Changelog

- **0.1.0** (2026-07) — Initial release: RSS 2.0 + Atom, feed auto-discovery with path probing, keyword filter with per-item match records, cross-feed dedup, plain-text content extraction, robots.txt-gated homepage discovery, per-run `RUN_SUMMARY`.

***

Maintained by **Sturdy Data** — boring, reliable scrapers. Report issues in the Issues tab; we respond within 24 hours.

# Actor input Schema

## `feedUrls` (type: `array`):

Direct RSS or Atom feed URLs.

## `siteUrls` (type: `array`):

Don't know the feed URL? Give the site homepage — feeds are discovered from <link rel="alternate"> tags and common paths (/feed, /rss.xml, /atom.xml).

## `keywords` (type: `array`):

Only keep items mentioning at least one keyword (case-insensitive, matched in title, summary and content). Leave empty to keep everything. Matched keywords are recorded per item.

## `maxItemsPerFeed` (type: `integer`):

Cap per feed (most feeds publish 10-100 items).

## `includeContent` (type: `boolean`):

Include the item's full content (content:encoded / atom content) as plain text when the feed provides it.

## `deduplicate` (type: `boolean`):

Drop items whose link/GUID was already seen in this run (useful when monitoring overlapping feeds).

## Actor input object example

```json
{
  "feedUrls": [
    "https://techcrunch.com/feed/",
    "https://hnrss.org/frontpage"
  ],
  "siteUrls": [],
  "keywords": [],
  "maxItemsPerFeed": 50,
  "includeContent": true,
  "deduplicate": true
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "feedUrls": [
        "https://techcrunch.com/feed/",
        "https://hnrss.org/frontpage"
    ],
    "siteUrls": [],
    "keywords": []
};

// Run the Actor and wait for it to finish
const run = await client.actor("sturdydata/rss-feed-monitor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "feedUrls": [
        "https://techcrunch.com/feed/",
        "https://hnrss.org/frontpage",
    ],
    "siteUrls": [],
    "keywords": [],
}

# Run the Actor and wait for it to finish
run = client.actor("sturdydata/rss-feed-monitor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "feedUrls": [
    "https://techcrunch.com/feed/",
    "https://hnrss.org/frontpage"
  ],
  "siteUrls": [],
  "keywords": []
}' |
apify call sturdydata/rss-feed-monitor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=sturdydata/rss-feed-monitor",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

```json
{
    "openapi": "3.0.1",
    "info": {
        "title": "RSS Feed Scraper & Keyword Monitor",
        "description": "Turn any list of RSS/Atom feeds — or just website homepages, feeds are auto-discovered — into a clean, deduplicated stream of news items, optionally filtered by keywords. Normalized output across RSS 2.0 and Atom: title, link, published date, author, categories, summary and full content as plain...",
        "version": "0.1",
        "x-build-id": "aaFYTqyIgBzbIUQjS"
    },
    "servers": [
        {
            "url": "https://api.apify.com/v2"
        }
    ],
    "paths": {
        "/acts/sturdydata~rss-feed-monitor/run-sync-get-dataset-items": {
            "post": {
                "operationId": "run-sync-get-dataset-items-sturdydata-rss-feed-monitor",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK"
                    }
                }
            }
        },
        "/acts/sturdydata~rss-feed-monitor/runs": {
            "post": {
                "operationId": "runs-sync-sturdydata-rss-feed-monitor",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor and returns information about the initiated run in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK",
                        "content": {
                            "application/json": {
                                "schema": {
                                    "$ref": "#/components/schemas/runsResponseSchema"
                                }
                            }
                        }
                    }
                }
            }
        },
        "/acts/sturdydata~rss-feed-monitor/run-sync": {
            "post": {
                "operationId": "run-sync-sturdydata-rss-feed-monitor",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK"
                    }
                }
            }
        }
    },
    "components": {
        "schemas": {
            "inputSchema": {
                "type": "object",
                "properties": {
                    "feedUrls": {
                        "title": "Feed URLs",
                        "type": "array",
                        "description": "Direct RSS or Atom feed URLs.",
                        "items": {
                            "type": "string"
                        }
                    },
                    "siteUrls": {
                        "title": "Website URLs (feeds auto-discovered)",
                        "type": "array",
                        "description": "Don't know the feed URL? Give the site homepage — feeds are discovered from <link rel=\"alternate\"> tags and common paths (/feed, /rss.xml, /atom.xml).",
                        "items": {
                            "type": "string"
                        }
                    },
                    "keywords": {
                        "title": "Keywords (optional filter)",
                        "type": "array",
                        "description": "Only keep items mentioning at least one keyword (case-insensitive, matched in title, summary and content). Leave empty to keep everything. Matched keywords are recorded per item.",
                        "items": {
                            "type": "string"
                        }
                    },
                    "maxItemsPerFeed": {
                        "title": "Max items per feed",
                        "minimum": 1,
                        "maximum": 1000,
                        "type": "integer",
                        "description": "Cap per feed (most feeds publish 10-100 items).",
                        "default": 50
                    },
                    "includeContent": {
                        "title": "Include full content text",
                        "type": "boolean",
                        "description": "Include the item's full content (content:encoded / atom content) as plain text when the feed provides it.",
                        "default": true
                    },
                    "deduplicate": {
                        "title": "Deduplicate across feeds",
                        "type": "boolean",
                        "description": "Drop items whose link/GUID was already seen in this run (useful when monitoring overlapping feeds).",
                        "default": true
                    }
                }
            },
            "runsResponseSchema": {
                "type": "object",
                "properties": {
                    "data": {
                        "type": "object",
                        "properties": {
                            "id": {
                                "type": "string"
                            },
                            "actId": {
                                "type": "string"
                            },
                            "userId": {
                                "type": "string"
                            },
                            "startedAt": {
                                "type": "string",
                                "format": "date-time",
                                "example": "2025-01-08T00:00:00.000Z"
                            },
                            "finishedAt": {
                                "type": "string",
                                "format": "date-time",
                                "example": "2025-01-08T00:00:00.000Z"
                            },
                            "status": {
                                "type": "string",
                                "example": "READY"
                            },
                            "meta": {
                                "type": "object",
                                "properties": {
                                    "origin": {
                                        "type": "string",
                                        "example": "API"
                                    },
                                    "userAgent": {
                                        "type": "string"
                                    }
                                }
                            },
                            "stats": {
                                "type": "object",
                                "properties": {
                                    "inputBodyLen": {
                                        "type": "integer",
                                        "example": 2000
                                    },
                                    "rebootCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "restartCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "resurrectCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "computeUnits": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            },
                            "options": {
                                "type": "object",
                                "properties": {
                                    "build": {
                                        "type": "string",
                                        "example": "latest"
                                    },
                                    "timeoutSecs": {
                                        "type": "integer",
                                        "example": 300
                                    },
                                    "memoryMbytes": {
                                        "type": "integer",
                                        "example": 1024
                                    },
                                    "diskMbytes": {
                                        "type": "integer",
                                        "example": 2048
                                    }
                                }
                            },
                            "buildId": {
                                "type": "string"
                            },
                            "defaultKeyValueStoreId": {
                                "type": "string"
                            },
                            "defaultDatasetId": {
                                "type": "string"
                            },
                            "defaultRequestQueueId": {
                                "type": "string"
                            },
                            "buildNumber": {
                                "type": "string",
                                "example": "1.0.0"
                            },
                            "containerUrl": {
                                "type": "string"
                            },
                            "usage": {
                                "type": "object",
                                "properties": {
                                    "ACTOR_COMPUTE_UNITS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_WRITES": {
                                        "type": "integer",
                                        "example": 1
                                    },
                                    "KEY_VALUE_STORE_LISTS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_INTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_EXTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_SERPS": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            },
                            "usageTotalUsd": {
                                "type": "number",
                                "example": 0.00005
                            },
                            "usageUsd": {
                                "type": "object",
                                "properties": {
                                    "ACTOR_COMPUTE_UNITS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_WRITES": {
                                        "type": "number",
                                        "example": 0.00005
                                    },
                                    "KEY_VALUE_STORE_LISTS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_INTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_EXTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_SERPS": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}
```