Under maintenance

Pricing

from $0.005 / actor start

Try for free

Go to Apify Store

General Web Scraper & AI Data Extraction

Under maintenance

Try for free

Scrape any website and extract clean markdown/text content. Uses Playwright for JS rendering, Readability for article extraction, and outputs clean machine-readable data.

Pricing

from $0.005 / actor start

Rating

0.0

(0)

Developer

Ernest Gaigulo

Actor stats

Bookmarked

Total users

Monthly active users

6 days ago

Last modified

Categories

Developer tools

Automation

You can access the General Web Scraper & AI Data Extraction programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in Integrations settings in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1import { ApifyClient } from 'apify-client';
2
3// Initialize the ApifyClient with your Apify API token
4// Replace the '<YOUR_API_TOKEN>' with your token
5const client = new ApifyClient({
6    token: '<YOUR_API_TOKEN>',
7});
8
9// Prepare Actor input
10const input = {
11    "startUrls": [
12        {
13            "url": "https://apify.com"
14        }
15    ]
16};
17
18// Run the Actor and wait for it to finish
19const run = await client.actor("erng/general-web-scraper").call(input);
20
21// Fetch and print Actor results from the run's dataset (if any)
22console.log('Results from dataset');
23console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
24const { items } = await client.dataset(run.defaultDatasetId).listItems();
25items.forEach((item) => {
26    console.dir(item);
27});
28
29// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

General Web Scraper & AI Data Extraction API in JavaScript

The Apify API client for JavaScript is the official library that allows you to use General Web Scraper & AI Data Extraction API in JavaScript or TypeScript, providing convenience functions and automatic retries on errors.

Install the apify-client

$npm install apify-client

Other API clients include:

General Web Scraper & AI Data Extraction API in Python

General Web Scraper & AI Data Extraction API through CLI

General Web Scraper & AI Data Extraction OpenAPI definition

General Web Scraper & AI Data Extraction API

Website To Markdown

swarmgarden/website-to-markdown

Convert any webpage to clean, readable Markdown format. Perfect for content extraction and readability.

Swarm Garden

Web Content Extractor - Clean Markdown for AI

geekguymj/web-content-extractor

Extract clean, readable markdown content from any web page. Removes navigation, ads, footers, and boilerplate — outputs structured markdown optimized for LLM training, RAG pipelines, and AI agents. Pay-per-event pricing. $0.002/page.

Matthew Jenkins

Ai Data Extraction

vivid_astronaut/ai-data-extraction

Fabio Suizu

Web Scraper — Extract Data from Any Website

oneary/web-scraper

🕸️ Generic web scraper for any website with CSS selectors & JS rendering via Playwright. Static & dynamic pages. Free platform compute pricing.

Luan M.

Web Page to Clean Markdown

consistent_tradition/web-to-markdown

Extracts clean Markdown text from any web page. Perfect for AI/RAG datasets, research corpora, and content analysis.

Peter PANG

Web Article Extractor — Clean Reader Mode Text & Metadata

maged120/reader-mode

Extract clean, readable article content from any web page. Strips ads, navigation, and clutter — returns title, author, full body text, and publish date in structured JSON.

Maged

News & Article Extractor

automation-lab/news-article-extractor

Auto-discover news/blog articles and extract clean text plus Markdown for LLM/RAG corpora. Uses RSS, sitemaps, and Readability; outputs metadata, counts, and token estimates.

Stas Persiianenko

AI Web Crawler

hounderd/ai-web-crawler

Crawl websites and extract clean, LLM-ready markdown content with stealth browser rendering, anti-bot hardening, smart content filtering, and structured metadata extraction. Built for RAG pipelines, AI agents, and data workflows.

Hounderd

JS Content Crawler Lite for RAG

george.the.developer/js-content-crawler-lite-rag

Extract clean Markdown, text, metadata, links, and diagnostics from web pages. Static-first for low cost, Browserless rendering only when needed.

George Kioko

RAG Spider - Web to Markdown Crawler for AI Training Data

lenient_grove/RAG-Spider

Enterprise-grade web crawler that converts messy websites into clean, chunked Markdown for AI systems. Uses Mozilla Readability for 95% cleaner extraction than competitors. Outputs RAG-ready data with metadata and token estimates. Perfect for building knowledge bases and training AI chatbots.