AI-Ready RSS Content Cleaner (Full-Text Article to Markdown)

AI-Ready RSS Content Cleaner is an enterprise-grade, ultra-lightweight content ingestion actor designed for AI engineers, RAG system architects, newsletter curators, and market intelligence teams.

Most RSS feeds only provide short, truncated summaries or messy HTML snippets stuffed with ad banners, tracking pixels, and navigation clutter. This Actor reads any standard RSS or Atom feed, fetches the full target article, strips all boilerplate DOM elements using fast C-based parsing (selectolax.lexbor), and formats the article into clean, structured Markdown ready for immediate LLM processing.

---

⚡ Why Use AI-Ready RSS Content Cleaner?

• 🚀 Full-Text Ingestion (Not Just Snippets): Bypasses feed truncation by following article links and extracting the complete article text.
• 🧹 Zero Boilerplate & Ad Removal: Automatically strips <script>, <style>, <nav>, <footer>, <header>, <aside>, cookie notices, social share widgets, and ads.
• 🤖 Structured Markdown for LLMs & RAG: Formats headings (#, ##, ###), blockquotes (>), bullet lists (-), and code blocks (```) cleanly so your LLM embeddings preserve semantic context.
• 💰 Runs on Under 128 MB RAM: Built with asynchronous HTTP/2 (httpx) and fast C-based parsing (selectolax.lexbor), eliminating heavy headless browsers (Puppeteer/Playwright) and slashing execution cost.
• 🏷️ Pay-Per-Event (PPE) Pricing: Pay only for the actual articles cleaned and saved ($0.0015 per article). No wasted credits on empty or duplicate feeds.

---

🔌 High-Impact Use Cases & Integrations

1. Vector Search & RAG Ingestion (LangChain, LlamaIndex, Pinecone)

Feed real-time news and industry blogs into your knowledge base:

1. Schedule this Actor to poll feeds hourly via Apify Schedules or webhooks.
2. Pipe output records straight into Pinecone, Qdrant, Weaviate, or Chroma.
3. Pass markdown_content directly as the page_content chunk, with author, published_date, word_count, and source_feed_url as metadata.

2. Autonomous Newsletter Automation (n8n, Make.com)

Automate your weekly curation newsletter without manual copy-pasting:

1. Connect this Actor to your target publication feeds (Substack, TechCrunch, arXiv, Hacker News).
2. Filter articles in n8n or Make.com by word_count or date.
3. Use OpenAI or Claude to summarize the clean Markdown and draft issue briefs straight into Mailchimp, Beehiiv, or Ghost.

3. Executive Briefing & Market Intelligence

Track competitors and market shifts across 50+ industry blogs daily without getting bogged down by ad trackers or bloated webpage scripts.

---

⚙️ Input Configuration

The Actor provides a clean Standard Mode for quick extraction and an Advanced / Nerd Mode for fine-tuning network parameters.

Standard Mode

Field	Type	Default	Description
rss_urls	Array	["https://techcrunch.com/feed/"]	List of RSS or Atom feed URLs to process.
max_articles_per_feed	Integer	20	Maximum number of recent articles to clean per feed (1 to 500).

Advanced / Nerd Mode (Engine & Network Tuning)

Field	Type	Default	Description
concurrency	Integer	5	Number of simultaneous HTTP workers (1 to 25).
request_timeout_sec	Integer	20	Timeout in seconds for HTTP requests.
custom_user_agent	String	None	Optional custom HTTP User-Agent string.
proxy_configuration	Object	{"useApifyProxy": true}	Apify Proxy settings to bypass anti-scraping firewalls.

---

📊 Normalized Output Schema

Each processed item is emitted into the default Apify Dataset:

{
  "source_feed_url": "https://techcrunch.com/feed/",
  "article_url": "https://techcrunch.com/2026/10/17/the-rise-of-local-ai-models",
  "title": "The Rise of Local AI Models",
  "author": "Dr. Jane Doe",
  "published_date": "Sat, 17 Oct 2026 12:00:00 GMT",
  "word_count": 482,
  "reading_time_minutes": 2.4,
  "markdown_content": "# The Rise of Local AI Models\n\nOver the last year, small language models running locally on edge hardware have grown exponentially.\n\n## Key Benefits\n\n- Zero latency network bottlenecks\n- Complete data privacy and sovereignty\n- Drastically reduced operational cost\n\n> Privacy is not an afterthought; it is a fundamental architecture requirement.\n\n```\ncurl -X POST http://localhost:11434/api/generate\n```\n\nIn conclusion, local AI transforms consumer and enterprise software forever.",
  "scraped_at": "2026-10-17T20:25:30.123456+00:00"
}

---

💰 Pay-Per-Event (PPE) Pricing Transparency

This Actor operates under Apify's Pay-Per-Event (PPE) model:

Event	Price (USD)	Description
apify-actor-start	$0.00005	Charged once at container startup.
article-cleaned	$0.0015	Charged once per full article cleaned and converted to Markdown.

Why PPE Saves Budget:

• Cleaning 100 full-text articles costs only $0.15.
• If a feed has no new posts or fails to resolve, you are not billed for idle browser compute.
• Automatic budget protection terminates the run gracefully when your user spending limit is hit.

---

❓ Frequently Asked Questions (FAQ)

What happens if an article is behind a hard paywall?

If the full HTML page cannot be reached or returns an access restriction, the Actor automatically falls back to the RSS item's content or summary payload, cleaning and formatting whatever text is available.

Can it handle Atom feeds as well as RSS?

Yes. The engine parses RSS 0.9x, RSS 2.0, Atom 1.0, and RDF feeds seamlessly.

How are word count and reading time calculated?

Word count counts whitespace-delimited tokens from the cleaned Markdown content. Reading time is calculated using the standard cognitive reading benchmark of 200 words per minute.

---

🛠️ Local Development & Testing

# Clone the repository
git clone https://github.com/your-org/ai-ready-rss-content-cleaner.git
cd ai-ready-rss-content-cleaner

# Install dependencies into your local virtual environment
.venv\Scripts\python.exe -m pip install -r requirements.txt

# Run the test suite with 100% mocked offline requests
.venv\Scripts\python.exe -m pytest -v

# Run locally using Apify CLI
apify run

---

📜 License & Compliance

Distributed under the Apache 2.0 License. Ensure scraping and feed consumption comply with feed publisher policies.