RSS & News Full-Text for LLM & RAG (Multilingual) avatar

RSS & News Full-Text for LLM & RAG (Multilingual)

Pricing

from $2.50 / 1,000 full-text articles

Go to Apify Store
RSS & News Full-Text for LLM & RAG (Multilingual)

RSS & News Full-Text for LLM & RAG (Multilingual)

Turn any RSS, Atom or JSON feed (or just a website URL) into clean full-text articles with language detection, de-duplication and ready-made RAG chunks. Respects robots.txt. Pay only for articles delivered.

Pricing

from $2.50 / 1,000 full-text articles

Rating

0.0

(0)

Developer

Henry Crawl

Henry Crawl

Maintained by Community

Actor stats

0

Bookmarked

2

Total users

1

Monthly active users

5 hours ago

Last modified

Features

Limited permissions

Share

Turn any RSS, Atom, RDF or JSON Feed - or just a website URL - into clean full-text news articles that are ready for LLMs, AI agents and RAG pipelines: boilerplate removed, language detected, duplicates removed across feeds, and articles pre-split into overlapping chunks with stable IDs for your vector database.

Works with news sites, blogs, Substack, WordPress, Medium-style publications, release notes and podcasts' show notes in any language - tested on English, German, French, Spanish, Portuguese, Italian, Dutch, Polish, Ukrainian, Japanese, Chinese, Korean, Vietnamese, Thai, Arabic, Hindi, Turkish and Indonesian sources.

You pay only for articles delivered. Articles that a site forbids in robots.txt, that are behind a bot check or a paywall teaser, and feeds that are broken come back with a clear status code and are free.

What you get

Feed readersThis Actor
Title, link, date, categories, imageYesYes
Full article text (not just the feed teaser)RarelyYes, main content only (no menus, ads, comments)
Language of the textFeed claim onlyDetected from the text (ja/zh/ko/th script-aware)
Same story from several feedsRepeatedReturned once (canonical URL + title fingerprint)
RAG chunksNoParagraph / sentence-aware, token-sized, overlapping, with IDs
One row per chunk for vector DBsNoYes (outputFormat: chunks)
Feed auto-discovery from a website URLSomeYes
Failed articlesOften charged or silently missingRow with status code, not charged

Use cases

  • News monitoring for AI agents - schedule a run every hour with maxAgeHours: 2 and feed the new articles to your agent or LLM summarizer.
  • RAG knowledge base - keep a vector index of industry news, competitor blogs or regulatory updates fresh; id and contentHash make upserts idempotent.
  • Multilingual market and media research - mix feeds from many countries and filter by detected languages.
  • Newsletters and digests - collect the full text of the day's posts from 50 blogs in one run.
  • Brand / topic tracking - includeKeywords searches title, summary and full body.

Input

{
"feeds": [
"https://feeds.bbci.co.uk/news/world/rss.xml",
"https://www.spiegel.de/schlagzeilen/index.rss",
"https://techcrunch.com"
],
"fullText": true,
"maxItemsPerFeed": 20,
"maxAgeHours": 24,
"includeKeywords": ["AI", "chip"],
"languages": [],
"outputFormat": "articles",
"chunkTokens": 500,
"chunkOverlapTokens": 50
}
  • feeds - feed URLs or plain website / section URLs (the feed is discovered from the page).
  • fullText - open each article and extract the main text. Off = only what the feed contains (cheaper).
  • maxItemsPerFeed / maxItems / maxAgeHours - limits.
  • includeKeywords / excludeKeywords / languages - filters; filtered articles are neither output nor charged.
  • outputFormat - articles (one row per article, chunks nested) or chunks (one flat row per chunk).
  • chunkTokens / chunkOverlapTokens - chunk sizing. Chunks never cut a sentence in half unless one sentence alone is longer than the chunk.
  • includeMarkdown - also return Markdown with headings, lists and tables.

Output (one row per article)

{
"status": "OK",
"id": "ce7f8d696ae13ec5",
"url": "https://www.bbc.co.uk/news/articles/...",
"canonicalUrl": "https://bbc.co.uk/news/articles/...",
"title": "What we know about the global Microsoft outage",
"description": "Outlook, Teams and Copilot users report problems...",
"publishedAt": "2026-11-13T10:12:00Z",
"language": "en",
"languageSource": "detected",
"sourceDomain": "bbc.co.uk",
"feedUrl": "https://feeds.bbci.co.uk/news/world/rss.xml",
"textSource": "article",
"text": "A major global Microsoft outage has caused problems ...",
"charCount": 4893,
"tokenEstimate": 1228,
"chunkCount": 4,
"chunks": [
{ "chunkIndex": 0, "id": "ce7f8d696ae13ec5-0", "text": "...", "tokenEstimate": 480 }
],
"contentHash": "5b1f0c2e9a7d4c11",
"fetchedAt": "2026-11-13T14:20:05Z"
}

With outputFormat: "chunks" each row is one chunk: id, articleId, chunkIndex, chunkCount, text, tokenEstimate plus the article's title, url, publishedAt, language, sourceDomain, author, categories.

Status codes (only OK is charged)

StatusMeaning
OKArticle delivered. textSource = article (page extracted) or feed (the feed itself carries the full post).
ROBOTS_DISALLOWEDThe site's robots.txt forbids automated access to this URL. We respect it.
BLOCKED / BOT_CHECKThe site refused access or answered with a bot challenge. Not bypassed by design.
TEXT_TOO_SHORTPage has almost no article text (paywall teaser, video or gallery page).
NOT_FOUND, HTTP_ERROR, NETWORK_ERRORThe page could not be loaded.
FEED_NOT_FOUND, NOT_A_FEEDNo feed at / linked from the URL you gave.

A run summary (counts per status, duplicates removed, items filtered) is saved as SUMMARY in the key-value store.

Pricing

Pay per event:

  • Full-text article: $2.50 per 1,000 articles (fullText: true, status OK).
  • Feed item: $1.00 per 1,000 items (fullText: false).
  • A tiny start fee of $0.00005 per run.

Error rows, duplicates and filtered articles are free. Set Max total charge on the run to cap spend.

Responsible use

  • Only public pages are read; no login, no paywall circumvention, no CAPTCHA solving.
  • robots.txt (including Crawl-delay) is honoured for every feed and article; requests to one website are sequential and spaced out (default 1 s).
  • Article text is copyrighted by its publishers. Use it in line with each publisher's terms and applicable law (e.g. internal analysis, search, summarisation with attribution). Google News RSS is not supported because its terms forbid non-personal use.

FAQ

Why is a site returning ROBOTS_DISALLOWED? Many publishers block crawlers in robots.txt. Use fullText: false for those feeds to still get the feed's own title and summary.

How accurate is tokenEstimate? Within ~10-15 % of OpenAI/Anthropic tokenizers for most languages; it is for chunk sizing only and never used for billing.

Can I run it on a schedule? Yes - create a Task with maxAgeHours slightly above the schedule interval and de-duplicate on id in your database.