Pricing

from $2.00 / 1,000 page crawleds

AI Web Crawler

Extract clean, chunked content from any website for LLMs and AI agents. Gets title, headings, main content, links, tables, and JSON-LD. Outputs token-estimated chunks ready for RAG pipelines.

Pricing

from $2.00 / 1,000 page crawleds

Rating

0.0

(0)

Developer

Oaida Adrian

Actor stats

Bookmarked

Total users

Monthly active users

14 days ago

Last modified

AI Web Content Crawler — LLM-Ready Text from Any Website

Crawl a website and get back clean, boilerplate-free content already chunked for LLMs — token-estimated segments, heading structure, tables, links, and JSON-LD — one dataset item per page. No API keys, no browser fleet, no post-processing script.

Perfect for:

🤖 RAG pipelines — feed contentChunks (~2,000-char segments with token estimates) straight into your embedder
📚 Knowledge-base ingestion — docs sites, blogs, wikis → vector store, without writing a scraper
🧠 LLM fine-tuning / evaluation corpora — clean mainContent with nav/footer/script noise removed
🕸️ Site mapping — internal/external link graphs and heading outlines per page
🔎 SEO & metadata audits — OpenGraph, Twitter Card, canonical URL, language, JSON-LD

How it works

Give it one or more start URLs. The crawler fetches each page, strips boilerplate (navigation, footers, scripts, ads), extracts the main content, and follows same-site links up to maxPages. Every page becomes one dataset item — ready to consume, no HTML parsing on your side.

Input

{
  "startUrls": [{ "url": "https://docs.example.com" }],
  "maxPages": 25
}

Field	Type	Default	Description
`startUrls`	array	required	Pages/sites to crawl
`maxPages`	integer	`5`	Max pages per start URL (`1` = just that page, up to 100)

Output

One item per crawled page:

{
  "url": "https://docs.example.com/getting-started",
  "metadata": {
    "title": "Getting Started — Example Docs",
    "description": "Install and configure Example in five minutes.",
    "canonicalUrl": "https://docs.example.com/getting-started",
    "language": "en",
    "ogTitle": "Getting Started",
    "ogType": "article"
  },
  "mainContent": "Getting started with Example. Install the CLI ...",
  "contentChunks": [
    { "id": "9f8e2c11ab04d7e3", "text": "Getting started with Example...", "estimatedTokens": 486 }
  ],
  "estimatedTokens": 1874,
  "headings": [
    { "level": 1, "text": "Getting Started" },
    { "level": 2, "text": "Installation" }
  ],
  "tables": [],
  "internalLinks": ["https://docs.example.com/configuration"],
  "externalLinks": ["https://github.com/example/cli"],
  "jsonld": [{ "@type": "TechArticle", "headline": "Getting Started" }]
}

contentChunks are split on paragraph boundaries at roughly 2,000 characters with a per-chunk token estimate — sized for embedding models, so most users index them with zero further processing.

Use it from the API

curl -s "https://api.apify.com/v2/acts/darknezz~ai-web-crawler/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"startUrls": [{"url": "https://docs.example.com"}], "maxPages": 10}'

Also available through the Apify SDK (Python/JS), scheduled tasks for recurring re-indexing, Zapier/Make, and MCP-enabled AI agents.

Pricing

Pay per event: $0.002 per page crawled, plus Apify's standard compute. Crawling a 500-page docs site costs about $1 in event fees — no subscription.

FAQ

Does it render JavaScript? No — it fetches server-rendered HTML, which keeps it fast and cheap. Docs sites, blogs, wikis, news, and marketing sites work great; pure client-side SPAs may return thin content.

How does it decide what's "main content"? Nav, header, footer, aside, script, style, and form elements are stripped, then the <main> or <article> element is preferred (falling back to <body>) and its paragraphs, list items, quotes, and code blocks are collected. Headings and tables are extracted separately so structure isn't lost.

Can I crawl just one page? Yes — set maxPages to 1.

Does it stay on the site? Yes — only same-host links are followed. External links are still reported in externalLinks for graph building.

What are token estimates based on? A ~4 characters/token heuristic — close enough for chunk budgeting with OpenAI/Anthropic-class tokenizers.

AI Web Extractor: URL → Clean Markdown + JSON for LLM/RAG

boxbox10/ai-web-extractor

Turn any URL into clean, LLM-ready Markdown + structured JSON (title, headings, main content, links, metadata, token count). Perfect for RAG pipelines, AI agents, and LLM context.

Marvin Eguilos

RAG-Ready Website Crawler — Clean Content for LLMs & Vector DBs

yourwingman/rag-ready-crawler

Crawl websites and output clean, chunked content optimized for RAG pipelines, LLM training data, and vector databases. Built for AI knowledge bases and semantic search.

Wingman

AI / RAG Web Crawler

groupoject/ai-rag-web-crawler

Crawl any website and extract clean, LLM-ready Markdown chunks to feed AI agents, chatbots, and RAG pipelines. One row per embeddable chunk.

Group Oject

AI Web Reader (RAG Ready)

viinaysonii/ai-web-reader-rag-ready

Convert any webpage into clean, structured, AI-ready Markdown. Removes ads, images, and UI noise, normalizes content, and outputs data optimized for LLMs, RAG pipelines, and AI agents. Fast, scalable, and built for real-world AI workflows.

Site to Agent Feed (URL to RAG-ready Markdown)

constant_quadruped/site-to-agent-feed

Turn any URL into clean, RAG-ready Markdown + structured JSON for LLMs and AI agents. Self-healing main-content extraction (survives redesigns), headings/links/tables, optional change-detection. No paid APIs.

Web Content Scraper - Clean Text, Metadata & AI Data

unrivaled_fortress/web-content-intelligence

Scrape public webpages and extract clean text, metadata, headings, links and structured data ready for AI and LLM consumption. Web scraping for RAG pipelines and content monitoring. Pay per result. Try free — no API key, instant results.

David Ahn

AI Web Content Crawler - Markdown for LLMs

intelscrape/ai-web-content-crawler

Crawl any website and extract clean Markdown optimized for LLM training, RAG pipelines, and AI knowledge bases - removes boilerplate and outputs structured JSON with URL, title, markdown, and metadata.

IntelScrape

Contextractor — clean web content extraction for LLMs

glueo/contextractor

Crawl any website and extract clean main-content text as Markdown, plain text, JSON, or HTML — ready for LLMs, RAG pipelines, and vector databases. Built on the rs-trafilatura engine and an adaptive Crawlee + Playwright crawler.

Glueo

Website Content Crawler

parseforge/website-content-crawler

Crawl any website and pull clean Markdown content ready for AI! Follow links across a whole domain and extract page text, titles, headings, images, and metadata. Perfect for building RAG pipelines, training datasets, knowledge bases, and vector databases. Start crawling content in minutes!

ParseForge

Web Content Extractor API — URL to JSON

george.the.developer/web-content-extractor-api

Extract structured JSON from any webpage. Articles, products, recipes, jobs. Auto-detects content type. Returns metadata, headings, images, links. For AI agents and RAG.