Pricing

$5.00/month + usage

Try for free

Go to Apify Store

AI Website Content Extractor

Try for free

Crawl website pages, strip noise, and convert the main content to clean Markdown for RAG/LLM training.

Pricing

$5.00/month + usage

Rating

5.0

(2)

Developer

ScrapeAI

Actor stats

Bookmarked

Total users

Monthly active users

23 days ago

Last modified

Features

Crawl any public website page(s)
Automatically dismiss cookie / consent dialogs
Strip navigation bars, headers, footers, sidebars, ads, and modals
Detect the main content area using semantic HTML selectors (main, article, [role="main"], etc.)
Convert HTML to clean Markdown via turndown
Skip low-content pages (login walls, redirects) automatically
Outputs a structured dataset ready for AI use-cases

Input

Field	Type	Description	Default
startUrls	Array	List of `{url}` objects or plain URL strings to crawl	`[{url: "https://example.com"}]`
maxPages	Number	Maximum number of pages to process	`20`
proxyConfiguration	Object	Apify proxy settings (optional)	`{}`

Example Input

{
    "startUrls": [
        { "url": "https://en.wikipedia.org/wiki/Artificial_intelligence" },
        { "url": "https://openai.com/blog" }
    ],
    "maxPages": 10
}

Output

Each extracted page produces one dataset record:

Field	Type	Description
url	String	URL of the crawled page
title	String	Page `<title>`
markdown	String	Clean Markdown of the main content
text	String
wordCount	Number	Approximate word count of the Markdown
extractedAt	String	ISO 8601 timestamp

Example Output

{
    "url": "https://en.wikipedia.org/wiki/Artificial_intelligence",
    "title": "Artificial intelligence - Wikipedia",
    "markdown": "# Artificial intelligence\n\nArtificial intelligence (AI) is the simulation of human intelligence...",
    "text": "Example Domain\n\nThis domain is for use in documentation examples without needing permission. Avoid use in operations.\n\nLearn more",	
    "wordCount": 4312,
    "extractedAt": "2026-03-13T08:00:00.000Z"
}

Use Cases

RAG pipelines — ingest Markdown directly into your vector store
LLM fine-tuning — build clean text corpora from any website
AI chatbots — feed domain knowledge to your assistant
Research — extract and archive article content at scale

Tips

Supply multiple startUrls to crawl several pages in one run
Increase maxPages to crawl an entire site (combine with Apify's link-following features)
For authenticated pages, configure a proxy or session in proxyConfiguration

Google Hotels Scraper

scrapeai/google-hotels-scraper

A scraper that extracts hotel listings from Google Travel search pages. It collects hotel names, prices, ratings, locations, amenities, and booking links from dynamic Google Hotels results for travel analysis, price comparison, and market research.

ScrapeAI

5.0

Fssai Scraper

scrapeai/fssai-scraper

Validates FSSAI license numbers using the FoSCoS API with AES-128-ECB encrypted communication. Retrieves CSRF tokens via Playwright, decrypts responses, and returns structured JSON with license details like business name, address, status, and category.

ScrapeAI

5.0

Google Careers Scraper

scrapeai/google-careers-scraper

Google Careers Scraper extracts job listings from the Google Careers website. It collects details such as job title, location, department, job ID, description, and application links. Ideal for job market analysis, recruitment research, and building job databases from Google Careers pages.

ScrapeAI

5.0

FSSAI FoSCoS Scraper

codingfrontend/fssai-foscos-scraper

Verify FSSAI license numbers using FoSCoS portal

codingfrontend

Google Hotels Search Scraper

johnvc/google-hotels-search-scraper

Extract comprehensive hotel data from Google Hotels including prices, ratings, amenities, property details, and images. Supports advanced filtering, vacation rentals, localization, and multi-page pagination. Ideal for market research, price monitoring, and travel applications.

John

4.5

Website Content to Markdown for LLM Training

easyapi/website-content-to-markdown-for-llm-training

🚀 Transform web content into clean, LLM-ready Markdown! 📘 Scrape multiple pages, extract main content, and convert to Markdown format. Perfect for AI researchers, data scientists, and LLM developers. Fast, efficient, and customizable. Supercharge your AI training data today! 🌐📝🧠

EasyApi

259

5.0

Website To Markdown

smart_api/website-to-markdown

Convert any webpage into clean, LLM-ready Markdown in seconds — perfect for AI training data, RAG pipelines, and content archiving.

SmartApi

5.0

Website Main Content Extractor

sync-network/website-main-content-extractor

Alam

Expired Jobs API for: Career Site Job Listing API

fantastic-jobs/expired-jobs-api-for-career-site-job-listing-api

This Actor is a companion to the Career Site Job Listing API Actor. It provides expired jobs once per day.

Fantastic.jobs

Website to Markdown Crawler â€” AI/RAG Data Pipeline

sovereigntaylor/website-to-markdown

Crawl any website and convert every page to clean, structured Markdown. Perfect for RAG pipelines, LLM training data, vector database ingestion, knowledge base building, and AI-powered search. Extracts main content, strips boilerplate, handles metadata, and chunks output for embeddings. Works with L