Pricing

from $2.00 / 1,000 html extractions

Refinery — HTML to LLM Text (Cut RAG Token Cost)

HTML to LLM text cleaner for RAG pipelines. Strip scripts, nav & layout junk after Firecrawl or your fetch. BeautifulSoup-alternative speed. $0.002/page · 3 README demos in Console.

Pricing

from $2.00 / 1,000 html extractions

Rating

0.0

(0)

Developer

Lare Labs

Actor stats

Bookmarked

Total users

Monthly active users

9 hours ago

Last modified

Refinery — HTML to LLM text cleaner for RAG pipelines

Apify Actor that cleans bloated HTML before chunking and embedding — strip scripts, nav, ads, and layout junk from pages you already fetched.
Pay $0.002/page · ~2–8ms per page (Rust core, after your crawler runs).

Refinery pipeline: raw HTML to clean JSON for RAG

Reduce LLM token cost — HTML cleaner for RAG

Token reduction: bloated HTML vs clean text after Refinery

News-style homepages and heavy DOM pages often waste tokens on chrome. Refinery returns main-body text plus word_count so you can budget embeddings — up to ~97% fewer estimated tokens on bloated HTML (your mileage varies).

Scraped timelines and comment threads ship messy DOM — scripts, sidebars, widgets. Refinery keeps post body text and normalizes @mentions / #hashtags for RAG chunking without paying for layout noise.

Social and feed HTML: mentions and hashtags preserved as clean text

Paste raw_payload from your scraper, or pass URLs if you already fetch HTML elsewhere.

Apify Console output — clean text and word count

Run Try actor with the prefilled example.com URL — each dataset row includes text, word_count, and timing:

Apify dataset output: clean text, word count, and timing

Bulk HTML cleaning for crawl batches

Send many URLs in one run — each row gets text, word_count, and processing_time_ms. Ideal after a sitemap pass, Firecrawl export, or Apify crawler dataset.

Bulk URL mode: many pages in, dataset rows out

{
  "urls": [
    "https://example.com",
    "https://www.bbc.com/news",
    "https://httpbin.org/html"
  ],
  "removeScripts": true,
  "removeStyles": true,
  "includeMetadata": true
}

Who uses this HTML text extractor

RAG and agent builders cutting OpenAI / Anthropic token spend on page HTML
Scrape pipelines that already fetch HTML (Firecrawl, Crawl4AI, Playwright, Apify Web Scraper)
Teams replacing per-worker BeautifulSoup with a fast HTML parser API on Apify

Refinery is not a web crawler. It is an HTML-to-text preprocessing step after fetch.

Your crawler → raw HTML → Refinery → clean text → chunk → embed → vector DB → LLM

Try the HTML to LLM cleaner (3 demos)

Demo 1 is prefilled in Console. Paste Demo 2 or Demo 3 to see different modes.

Demo 1 — Quick URL

{
  "urls": ["https://example.com"],
  "removeScripts": true,
  "removeStyles": true,
  "includeMetadata": true
}

Demo 2 — Bloated news homepage

{
  "urls": ["https://www.bbc.com/news"],
  "removeScripts": true,
  "removeStyles": true,
  "includeMetadata": true
}

Demo 3 — Paste HTML (middleware)

{
  "raw_payload": "<html><head><script>gtag('event')</script></head><body><nav>Home · Pricing</nav><article><h1>Update</h1><p>Clean before embedding.</p></article></body></html>",
  "removeScripts": true,
  "removeStyles": true,
  "includeMetadata": true
}

Output — text, word_count, language, timing

{
  "text": "Example Product Page\nEnterprise AI Infrastructure...",
  "language": "en",
  "word_count": 12,
  "content_type": "web",
  "processing_time_ms": 19.29,
  "success": true
}

Field	Use it for
`text`	Chunking, embeddings, LLM context
`word_count`	Cost estimates
`processing_time_ms`	Latency monitoring

Firecrawl, Crawl4AI, and BeautifulSoup alternative

Refinery in your stack: crawler, clean, vector DB, LLM

You already use…	Refinery's job
Firecrawl, Crawl4AI	Clean their HTML before chunking — fetch with them, clean with Refinery
Apify Web Scraper, Website Content Crawler	Clean the `html` field in your dataset
BeautifulSoup (self-hosted)	Same job, ~281× faster hot path in our benchmarks — pay per page on Apify instead of worker CPU

Pricing — HTML extraction on Apify

Event	Cost
HTML extraction	$0.002 / page
~1,000 pages	~$2.05

Integrate via Apify API (JavaScript and Python)

JavaScript

import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('larelabs/refinery-html-to-llm-cleaner').call({
  urls: ['https://example.com'],
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items[0].text, items[0].word_count);

Python

from apify_client import ApifyClient
client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("larelabs/refinery-html-to-llm-cleaner").call(
    run_input={"urls": ["https://example.com"]}
)
print(next(client.dataset(run["defaultDatasetId"]).iterate_items()))

FAQ — HTML cleaning for LLM and RAG

Is Refinery a replacement for Firecrawl or Crawl4AI?

No. Fetch with Firecrawl or Crawl4AI, then clean with Refinery. Refinery does not crawl URLs on its own schedule — it strips noise from HTML you already have (or fetches URLs you pass in this run).

How do I reduce RAG token cost from bloated HTML?

Run Refinery on raw HTML before chunking and embedding. Use word_count in the output to estimate savings. Remove scripts, styles, nav, and footer chrome so embeddings only see article body text.

Is this a BeautifulSoup alternative for HTML text extraction?

Yes — same preprocessing job (HTML → clean text), implemented in Rust for low latency. Use it when you want a managed Apify step instead of BeautifulSoup on every worker.

Can I clean HTML after Apify Web Scraper or Website Content Crawler?

Yes. Pass each page's HTML via raw_payload, or pipe URLs from your crawl. Refinery returns plain text ready for chunking.

Does Refinery handle JavaScript SPAs?

Only if you pass rendered HTML from a browser crawler (Playwright, Puppeteer, Firecrawl). Refinery cleans DOM; it does not execute JavaScript.

No login or feed scraping. Pass saved timeline HTML via raw_payload — Refinery extracts post text and normalizes @mentions / #hashtags.

Support — LareLabs

LareLabs · Apify Store listing · Console

Rust core · Apify Actor · Update WebPs in assets/store/, upload PNGs to Imgur, edit image_urls.json, then run embed + sync scripts.

Rent.com Scraper

haketa/rent-com-scraper

Scrape rental listings from Rent.com — apartments, houses, condos and townhouses across all US cities. Extract rent prices, floor plans, amenities, pet policies, parking, photos, phone numbers and availability. Filter by city, price range, bedrooms and property type.

Haketa

Website Content Crawler

mikolabs/website-content-crawler

Deep-crawl websites to extract clean text, Markdown, or HTML for AI/LLM apps, RAG pipelines, and vector databases. Supports adaptive crawling, HTML cleaning, file downloads, and structured dataset output. Easily integrates with LangChain, LlamaIndex, and other LLM tools.

mikolabs

5.0

(1)

RAG-Ready Web Scraper & Smart Chunker for AI Knowledge Bases

adinfosys-labs/rag-ready-web-scraper-smart-chunker-for-ai-knowledge-bases

RAG-ready web scraper that collects, cleans, deduplicates, filters, and chunks web content into structured datasets for AI pipelines. Generates high-quality knowledge-base data optimized for LLMs, embeddings, and vector databases

Artashes Arakelyan

RAG Spider - Web to Markdown Crawler for AI Training Data

lenient_grove/RAG-Spider

Enterprise-grade web crawler that converts messy websites into clean, chunked Markdown for AI systems. Uses Mozilla Readability for 95% cleaner extraction than competitors. Outputs RAG-ready data with metadata and token estimates. Perfect for building knowledge bases and training AI chatbots.

Tejas Rawool

5.0

(1)

🕷️ Website Crawler — Full-Site Scraping for AI

nexgendata/website-content-crawler

Crawl entire websites for clean text, markdown or HTML. Perfect for RAG pipelines, AI training & content analysis. Handles JS-rendered pages. Alternative to Firecrawl & Jina. Pay per page.

Stephan Corbeil

Website Content Crawler for AI — Clean Markdown, 4x Cheaper

joyouscam35875/website-content-crawler

Crawl any website and extract clean text/markdown for LLMs, RAG pipelines, vector databases. BFS crawl with depth control, robots.txt support, boilerplate removal. Perfect for feeding AI models. $0.001/page — 4x cheaper than the official Apify crawler.

Ken Digital

Webpage to Markdown Converter

automation-lab/webpage-to-markdown-converter

Convert webpages to clean Markdown for LLM/RAG pipelines. Uses @mozilla/readability to strip ads, navigation, and footers. Outputs structured JSON.

Stas Persiianenko

Website to Markdown — AI-Ready Content for RAG

ryanclinton/website-to-markdown

Website To Markdown. Available on the Apify Store with pay-per-event pricing.

Ryan Clinton

Gemini File Search Builder

yoloshii/gemini-file-search-builder

Build Gemini File Search RAG knowledge bases from any website with automatic citations.

Yoloshi

AI Blog Dataset Creator

datapilot/ai-blog-dataset-creator

Smart Article Scraper Actor extracts structured article data from URLs using, and Newspaper3k. It collects title, author, publish date, tags, full content, language, and word count. Supports proxy usage, JavaScript-rendered pages, and outputs clean JSON datasets.