# RSS & News Full-Text for LLM & RAG (Multilingual) (`henrycrawl/rss-news-fulltext-rag`) Actor

Turn any RSS, Atom or JSON feed (or just a website URL) into clean full-text articles with language detection, de-duplication and ready-made RAG chunks. Respects robots.txt. Pay only for articles delivered.

- **URL**: https://apify.com/henrycrawl/rss-news-fulltext-rag.md
- **Developed by:** [Henry Crawl](https://apify.com/henrycrawl) (community)
- **Categories:** AI, News, Developer tools
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.50 / 1,000 full-text articles

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## RSS & News Full-Text for LLM & RAG (Multilingual)

Turn any **RSS, Atom, RDF or JSON Feed** - or just a website URL - into **clean full-text news articles** that are ready for LLMs, AI agents and RAG pipelines: boilerplate removed, **language detected**, **duplicates removed across feeds**, and **articles pre-split into overlapping chunks** with stable IDs for your vector database.

Works with news sites, blogs, Substack, WordPress, Medium-style publications, release notes and podcasts' show notes in **any language** - tested on English, German, French, Spanish, Portuguese, Italian, Dutch, Polish, Ukrainian, Japanese, Chinese, Korean, Vietnamese, Thai, Arabic, Hindi, Turkish and Indonesian sources.

**You pay only for articles delivered.** Articles that a site forbids in robots.txt, that are behind a bot check or a paywall teaser, and feeds that are broken come back with a clear status code and are **free**.

### What you get

| | Feed readers | This Actor |
|---|---|---|
| Title, link, date, categories, image | Yes | Yes |
| Full article text (not just the feed teaser) | Rarely | Yes, main content only (no menus, ads, comments) |
| Language of the text | Feed claim only | Detected from the text (ja/zh/ko/th script-aware) |
| Same story from several feeds | Repeated | Returned once (canonical URL + title fingerprint) |
| RAG chunks | No | Paragraph / sentence-aware, token-sized, overlapping, with IDs |
| One row per chunk for vector DBs | No | Yes (`outputFormat: chunks`) |
| Feed auto-discovery from a website URL | Some | Yes |
| Failed articles | Often charged or silently missing | Row with status code, not charged |

### Use cases

- **News monitoring for AI agents** - schedule a run every hour with `maxAgeHours: 2` and feed the new articles to your agent or LLM summarizer.
- **RAG knowledge base** - keep a vector index of industry news, competitor blogs or regulatory updates fresh; `id` and `contentHash` make upserts idempotent.
- **Multilingual market and media research** - mix feeds from many countries and filter by detected `languages`.
- **Newsletters and digests** - collect the full text of the day's posts from 50 blogs in one run.
- **Brand / topic tracking** - `includeKeywords` searches title, summary and full body.

### Input

```json
{
  "feeds": [
    "https://feeds.bbci.co.uk/news/world/rss.xml",
    "https://www.spiegel.de/schlagzeilen/index.rss",
    "https://techcrunch.com"
  ],
  "fullText": true,
  "maxItemsPerFeed": 20,
  "maxAgeHours": 24,
  "includeKeywords": ["AI", "chip"],
  "languages": [],
  "outputFormat": "articles",
  "chunkTokens": 500,
  "chunkOverlapTokens": 50
}
```

- **feeds** - feed URLs or plain website / section URLs (the feed is discovered from the page).
- **fullText** - open each article and extract the main text. Off = only what the feed contains (cheaper).
- **maxItemsPerFeed / maxItems / maxAgeHours** - limits.
- **includeKeywords / excludeKeywords / languages** - filters; filtered articles are neither output nor charged.
- **outputFormat** - `articles` (one row per article, chunks nested) or `chunks` (one flat row per chunk).
- **chunkTokens / chunkOverlapTokens** - chunk sizing. Chunks never cut a sentence in half unless one sentence alone is longer than the chunk.
- **includeMarkdown** - also return Markdown with headings, lists and tables.

### Output (one row per article)

```json
{
  "status": "OK",
  "id": "ce7f8d696ae13ec5",
  "url": "https://www.bbc.co.uk/news/articles/...",
  "canonicalUrl": "https://bbc.co.uk/news/articles/...",
  "title": "What we know about the global Microsoft outage",
  "description": "Outlook, Teams and Copilot users report problems...",
  "publishedAt": "2026-11-13T10:12:00Z",
  "language": "en",
  "languageSource": "detected",
  "sourceDomain": "bbc.co.uk",
  "feedUrl": "https://feeds.bbci.co.uk/news/world/rss.xml",
  "textSource": "article",
  "text": "A major global Microsoft outage has caused problems ...",
  "charCount": 4893,
  "tokenEstimate": 1228,
  "chunkCount": 4,
  "chunks": [
    { "chunkIndex": 0, "id": "ce7f8d696ae13ec5-0", "text": "...", "tokenEstimate": 480 }
  ],
  "contentHash": "5b1f0c2e9a7d4c11",
  "fetchedAt": "2026-11-13T14:20:05Z"
}
```

With `outputFormat: "chunks"` each row is one chunk: `id`, `articleId`, `chunkIndex`, `chunkCount`, `text`, `tokenEstimate` plus the article's `title`, `url`, `publishedAt`, `language`, `sourceDomain`, `author`, `categories`.

#### Status codes (only `OK` is charged)

| Status | Meaning |
|---|---|
| `OK` | Article delivered. `textSource` = `article` (page extracted) or `feed` (the feed itself carries the full post). |
| `ROBOTS_DISALLOWED` | The site's robots.txt forbids automated access to this URL. We respect it. |
| `BLOCKED` / `BOT_CHECK` | The site refused access or answered with a bot challenge. Not bypassed by design. |
| `TEXT_TOO_SHORT` | Page has almost no article text (paywall teaser, video or gallery page). |
| `NOT_FOUND`, `HTTP_ERROR`, `NETWORK_ERROR` | The page could not be loaded. |
| `FEED_NOT_FOUND`, `NOT_A_FEED` | No feed at / linked from the URL you gave. |

A run summary (counts per status, duplicates removed, items filtered) is saved as `SUMMARY` in the key-value store.

### Pricing

Pay per event:

- **Full-text article: $2.50 per 1,000 articles** (`fullText: true`, status `OK`).
- **Feed item: $1.00 per 1,000 items** (`fullText: false`).
- A tiny start fee of $0.00005 per run.

Error rows, duplicates and filtered articles are free. Set **Max total charge** on the run to cap spend.

### Responsible use

- Only public pages are read; no login, no paywall circumvention, no CAPTCHA solving.
- robots.txt (including `Crawl-delay`) is honoured for every feed and article; requests to one website are sequential and spaced out (default 1 s).
- Article text is copyrighted by its publishers. Use it in line with each publisher's terms and applicable law (e.g. internal analysis, search, summarisation with attribution). Google News RSS is not supported because its terms forbid non-personal use.

### FAQ

**Why is a site returning `ROBOTS_DISALLOWED`?** Many publishers block crawlers in robots.txt. Use `fullText: false` for those feeds to still get the feed's own title and summary.

**How accurate is `tokenEstimate`?** Within ~10-15 % of OpenAI/Anthropic tokenizers for most languages; it is for chunk sizing only and never used for billing.

**Can I run it on a schedule?** Yes - create a Task with `maxAgeHours` slightly above the schedule interval and de-duplicate on `id` in your database.

# Actor input Schema

## `feeds` (type: `array`):

RSS, Atom, RDF or JSON Feed URLs - or just a website / section URL (e.g. https://techcrunch.com) and the feed is discovered automatically. Mix languages and countries freely; the same story appearing in several feeds is returned once.

## `fullText` (type: `boolean`):

Open each article and extract the clean main text (no menus, ads, comments). If off, only what the feed itself contains is returned (cheaper, faster). Articles whose site forbids it in robots.txt, blocks automated access, or shows only a paywall teaser come back with an error status and are NOT charged - unless the feed itself carries the full post.

## `maxItemsPerFeed` (type: `integer`):

Newest items first, as ordered by the feed.

## `maxItems` (type: `integer`):

Hard cap for the whole run (0 = no cap). Also limited by your max total charge.

## `maxAgeHours` (type: `integer`):

Skip items published longer ago than this. 0 = no age limit. Ideal for scheduled monitoring (e.g. 24 for a daily run).

## `includeKeywords` (type: `array`):

Keep only articles mentioning at least one keyword (whole-word, case-insensitive; substring match for Chinese/Japanese/Korean/Thai). With full text on, the article body is searched too. Filtered articles are not output and not charged.

## `excludeKeywords` (type: `array`):

Drop items whose title, summary or categories contain any of these.

## `languages` (type: `array`):

ISO 639-1 codes such as en, de, ja, vi. Language is detected from the article text, not trusted from the feed. Empty = all languages.

## `outputFormat` (type: `string`):

"One row per article" keeps text, metadata and chunks together. "One row per chunk" gives flat rows (chunk text + article metadata) ready to embed and upsert into a vector database.

## `includeChunks` (type: `boolean`):

Split each article into overlapping chunks along paragraph and sentence boundaries.

## `chunkTokens` (type: `integer`):

Approximate tokens per chunk (Chinese/Japanese/Korean/Thai characters count as ~1 token each, other scripts ~4 characters per token).

## `chunkOverlapTokens` (type: `integer`):

Context repeated from the end of the previous chunk (whole sentences/paragraphs only).

## `includeMarkdown` (type: `boolean`):

Adds a `markdown` field with headings, lists and tables preserved.

## `maxConcurrency` (type: `integer`):

Different websites fetched in parallel. Requests to the same website are always sequential and spaced out.

## `perSiteDelaySecs` (type: `number`):

Politeness delay. A longer Crawl-delay in the site's robots.txt is honoured automatically.

## Actor input object example

```json
{
  "feeds": [
    "https://feeds.bbci.co.uk/news/world/rss.xml",
    "https://www.spiegel.de/schlagzeilen/index.rss",
    "https://techcrunch.com"
  ],
  "fullText": true,
  "maxItemsPerFeed": 20,
  "maxItems": 0,
  "maxAgeHours": 0,
  "includeKeywords": [],
  "excludeKeywords": [],
  "languages": [],
  "outputFormat": "articles",
  "includeChunks": true,
  "chunkTokens": 500,
  "chunkOverlapTokens": 50,
  "includeMarkdown": false,
  "maxConcurrency": 8,
  "perSiteDelaySecs": 1
}
```

# Actor output Schema

## `results` (type: `string`):

Full-text articles or RAG chunks, plus error rows.

## `summary` (type: `string`):

Counts per status for this run (SUMMARY record).

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "feeds": [
        "https://feeds.bbci.co.uk/news/world/rss.xml",
        "https://www.spiegel.de/schlagzeilen/index.rss",
        "https://techcrunch.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("henrycrawl/rss-news-fulltext-rag").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "feeds": [
        "https://feeds.bbci.co.uk/news/world/rss.xml",
        "https://www.spiegel.de/schlagzeilen/index.rss",
        "https://techcrunch.com",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("henrycrawl/rss-news-fulltext-rag").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "feeds": [
    "https://feeds.bbci.co.uk/news/world/rss.xml",
    "https://www.spiegel.de/schlagzeilen/index.rss",
    "https://techcrunch.com"
  ]
}' |
apify call henrycrawl/rss-news-fulltext-rag --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,henrycrawl/rss-news-fulltext-rag"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/wcStWrQt4PvjiwVdB/builds/n1cNTEHur4YJJw5uO/openapi.json
