Pricing

$3.00 / 1,000 results

Try for free

Go to Apify Store

AST Website Content Crawler for RAG

Try for free

Advanced web crawler extracting token-chunked Markdown for LLMs & RAG. Features dual-engine hybrid parsing (IBM Docling & Trafilatura) to preserve tables and layouts. Delivers clean text split by native tiktoken counts with rich metadata (headings, levels, tokens) for instant vector search.

Pricing

$3.00 / 1,000 results

Rating

5.0

(2)

Developer

lukas schmeck

Actor stats

Bookmarked

Total users

Monthly active users

14 days ago

Last modified

Categories

Developer tools

Automation

You can access the AST Website Content Crawler for RAG programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in Integrations settings in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1import { ApifyClient } from 'apify-client';
2
3// Initialize the ApifyClient with your Apify API token
4// Replace the '<YOUR_API_TOKEN>' with your token
5const client = new ApifyClient({
6    token: '<YOUR_API_TOKEN>',
7});
8
9// Prepare Actor input
10const input = {
11    "startUrls": [
12        {
13            "url": "https://docs.python.org/3/"
14        }
15    ]
16};
17
18// Run the Actor and wait for it to finish
19const run = await client.actor("lukas459/ai-web-to-markdown-crawler-llm-rag-optimized").call(input);
20
21// Fetch and print Actor results from the run's dataset (if any)
22console.log('Results from dataset');
23console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
24const { items } = await client.dataset(run.defaultDatasetId).listItems();
25items.forEach((item) => {
26    console.dir(item);
27});
28
29// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

AST Website Content Crawler for RAG API in JavaScript

The Apify API client for JavaScript is the official library that allows you to use AST Website Content Crawler for RAG API in JavaScript or TypeScript, providing convenience functions and automatic retries on errors.

Install the apify-client

$npm install apify-client

Other API clients include:

AST Website Content Crawler for RAG API in Python

AST Website Content Crawler for RAG API through CLI

AST Website Content Crawler for RAG OpenAPI definition

AST Website Content Crawler for RAG API

AI Web Scraper

apify/ai-web-scraper

AI-first web scraper that extracts structured data from any website using natural-language prompts. No programming knowledge required. No hard-coded logic that breaks when a website changes.

Apify

8.4K

4.3

RAG Web Browser

apify/rag-web-browser

Web search and fetch tool for AI agents and RAG pipelines. It queries Google Search, scrapes the top N pages using a full web browser, and returns their content as clean Markdown for further processing by an LLM. Can also fetch individual URLs.

Apify

146K

4.6

OpenRouter

apify/openrouter

You can use any AI LLM model without accounts in AI providers. Use this Actor as a proxy for all requests. Use pay-per-event pricing to pay only for the real credit used.

Apify

6.9K

4.8

Website Content Crawler

parseforge/website-content-crawler

Crawl any website and pull clean Markdown content ready for AI! Follow links across a whole domain and extract page text, titles, headings, images, and metadata. Perfect for building RAG pipelines, training datasets, knowledge bases, and vector databases. Start crawling content in minutes!

ParseForge

URL to markdown

apify/url-to-markdown

An Apify Actor that takes a URL as input and returns the content of the page in Markdown format.

Apify

Sitemap Extractor

apify/sitemap-extractor

This Apify Actor extracts all URLs from a website's sitemaps and checks their status codes via lightweight HTTP requests. It provides a clean list of valid links, acting as an ideal pre-processor to ensure your larger crawling projects target only active URLs.

Apify

250

3.2

Ollama Apify Mcp

lenticular_negative/ollama-apify-mcp

The Ollama MCP Actor brings together Apify’s web-scraping power with fast, private, on-device AI. No external APIs required. It lets you run local LLMs through Ollama using the Model Context Protocol, so you can analyze scraped data, extract insights, and generate responses with full control.

Anwesh Mishra

🔥 FireScrape AI Website Content Markdown Scraper

mohamedgb00714/fireScraper-AI-Website-Content-Markdown-Scraper

Advanced web scraper powered by Crawlee and Puppeteer — extracts website content, converts it to Markdown, and structures it for LLM training datasets.

mohamed el hadi msaid

303

1.9

Stealth Web Scraper

lentic_clockss/stealth-web-scraper

General-purpose stealth collection helper for public pages when a dedicated source Actor is not available. Prefer specialized Actors for jobs, ads, maps, and e-commerce.