Pricing

from $2.00 / 1,000 page crawleds

AI Web Crawler

Web crawler for LLMs and RAG: extract clean, chunked content from any website — title, headings, main content, links, tables and JSON-LD. Outputs token-estimated chunks ready for AI pipelines.

Pricing

from $2.00 / 1,000 page crawleds

Rating

0.0

(0)

Developer

Oaida Adrian

Actor stats

Bookmarked

Total users

Monthly active users

9 days ago

Last modified

AI Web Content Crawler — LLM-Ready Text from Any Website

Crawl a website and get back clean, boilerplate-free content already chunked for LLMs — token-estimated segments, heading structure, tables, links, and JSON-LD — one dataset item per page. No API keys, no browser fleet, no post-processing script.

Perfect for:

🤖 RAG pipelines — feed contentChunks (~2,000-char segments with token estimates) straight into your embedder
📚 Knowledge-base ingestion — docs sites, blogs, wikis → vector store, without writing a scraper
🧠 LLM fine-tuning / evaluation corpora — clean mainContent with nav/footer/script noise removed
🕸️ Site mapping — internal/external link graphs and heading outlines per page
🔎 SEO & metadata audits — OpenGraph, Twitter Card, canonical URL, language, JSON-LD

How it works

Give it one or more start URLs. The crawler fetches each page, strips boilerplate (navigation, footers, scripts, ads), extracts the main content, and follows same-site links up to maxPages. Every page becomes one dataset item — ready to consume, no HTML parsing on your side.

The content pipeline is opinionated for LLM consumption:

Boilerplate removal — nav, header, footer, aside, script, style and form elements are dropped.
Main-content selection — the <main> or <article> element is preferred, falling back to <body>.
Structure preservation — headings and tables are extracted separately so hierarchy isn't lost.
Chunking — mainContent is split on paragraph boundaries at roughly 2,000 characters, each chunk carrying an estimated token count (~4 chars/token heuristic).

Input

{
  "startUrls": [{ "url": "https://docs.example.com" }],
  "maxPages": 25
}

Field	Type	Default	Description
`startUrls`	array	required	Pages/sites to crawl
`maxPages`	integer	`5`	Max pages per start URL (`1` = just that page, up to 100)

Output

One item per crawled page:

{
  "url": "https://docs.example.com/getting-started",
  "metadata": {
    "title": "Getting Started — Example Docs",
    "description": "Install and configure Example in five minutes.",
    "canonicalUrl": "https://docs.example.com/getting-started",
    "language": "en",
    "ogTitle": "Getting Started",
    "ogType": "article"
  },
  "mainContent": "Getting started with Example. Install the CLI ...",
  "contentChunks": [
    { "id": "9f8e2c11ab04d7e3", "text": "Getting started with Example...", "estimatedTokens": 486 }
  ],
  "estimatedTokens": 1874,
  "headings": [
    { "level": 1, "text": "Getting Started" },
    { "level": 2, "text": "Installation" }
  ],
  "tables": [],
  "internalLinks": ["https://docs.example.com/configuration"],
  "externalLinks": ["https://github.com/example/cli"],
  "jsonld": [{ "@type": "TechArticle", "headline": "Getting Started" }]
}

contentChunks are split on paragraph boundaries at roughly 2,000 characters with a per-chunk token estimate — sized for embedding models, so most users index them with zero further processing.

Real example — index a docs site into a vector store

Input:

{
  "startUrls": [{ "url": "https://docs.example.com" }],
  "maxPages": 50
}

Crawl the docs root, follow same-site links to capture up to 50 pages, and every page comes back as a dataset item with mainContent plus pre-split contentChunks. Pipe the chunks into an embedder (OpenAI, Cohere, local sentence-transformers…) and you have a searchable knowledge base — no scraper code, no HTML cleanup.

At $0.002/page, a 50-page docs site costs about $0.10 in event fees — a full 500-page site costs about $1.

Use it from the API

curl -s "https://api.apify.com/v2/acts/darknezz~ai-web-crawler/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"startUrls": [{"url": "https://docs.example.com"}], "maxPages": 10}'

Also available through the Apify SDK (Python/JS), scheduled tasks for recurring re-indexing, Zapier/Make, and MCP-enabled AI agents.

# Python SDK — crawl + dump chunks straight to your pipeline
from apify_client import ApifyClient
client = ApifyClient("YOUR_TOKEN")
result = client.actor("darknezz~ai-web-crawler").call(
    run_input={"startUrls": [{"url": "https://docs.example.com"}], "maxPages": 10})
items = client.dataset(result["defaultDatasetId"]).list_items().items
for page in items:
    for chunk in page["contentChunks"]:
        embed(chunk["text"])  # your embedding function

Pricing

Pay per event: $0.002 per page crawled, plus Apify's standard compute. Crawling a 500-page docs site costs about $1 in event fees — no subscription.

FAQ

Does it render JavaScript? No — it fetches server-rendered HTML, which keeps it fast and cheap. Docs sites, blogs, wikis, news, and marketing sites work great; pure client-side SPAs may return thin content.

How does it decide what's "main content"? Nav, header, footer, aside, script, style, and form elements are stripped, then the <main> or <article> element is preferred (falling back to <body>) and its paragraphs, list items, quotes, and code blocks are collected. Headings and tables are extracted separately so structure isn't lost.

Can I crawl just one page? Yes — set maxPages to 1.

Does it stay on the site? Yes — only same-host links are followed. External links are still reported in externalLinks for graph building.

What are token estimates based on? A ~4 characters/token heuristic — close enough for chunk budgeting with OpenAI/Anthropic-class tokenizers.

Why are chunks ~2,000 characters? That size fits comfortably inside most embedding model context windows (and even 8k-token contexts) with room for the query and system prompt, so each chunk can be embedded and retrieved directly.

Is it suitable for scheduled re-indexing? Yes — point a Schedule at it and diff on url + estimatedTokens to detect content changes over time.

Limits & reliability

Up to 100 pages per start URL per run
Same-host crawl boundary prevents runaway link traversal
Boilerplate removal is deterministic (tag-based), not ML — consistent, fast, cheap
Retries on transient network errors with backoff; failed pages are logged, not fatal

Website Content Crawler

rupom888/website-content-crawler

Syed Rupom

AI Web Extractor: URL → Clean Markdown + JSON for LLM/RAG

boxbox10/ai-web-extractor

Turn any URL into clean, LLM-ready Markdown + structured JSON (title, headings, main content, links, metadata, token count). Perfect for RAG pipelines, AI agents, and LLM context.

Marvin Eguilos

No-BS Content Crawler 🖕

successful_nonagon/no-bs-content-crawler

Fast web crawler that extracts clean text from websites. Returns readable content, headings, and links. Perfect for content aggregation, SEO research, and data collection.

hafsah nuzhat

5.0

AI / RAG Web Crawler

groupoject/ai-rag-web-crawler

Crawl any website and extract clean, LLM-ready Markdown chunks to feed AI agents, chatbots, and RAG pipelines. One row per embeddable chunk.

Group Oject

Website to Markdown Crawler – Clean RAG Content

signal_lab/website-to-markdown-crawler

Website-to-Markdown crawler for RAG and LLM datasets. Extract clean content, SEO metadata, links, JSON-LD, sitemaps, and robots.txt-compliant pages.

Signal Lab

RAG-Ready Website Crawler — Clean Content for LLMs & Vector DBs

yourwingman/rag-ready-crawler

Crawl websites and output clean, chunked content optimized for RAG pipelines, LLM training data, and vector databases. Built for AI knowledge bases and semantic search.

Wingman

Website Content Crawler — Extract Full Site Content

oneary/website-content-crawler

🌐 Full website crawler that extracts structured content (text, headings, metadata, links, images) from any domain. Free platform compute pricing.

Luan M.

AI Web Content Crawler - Markdown for LLMs

intelscrape/ai-web-content-crawler

Crawl any website and extract clean Markdown optimized for LLM training, RAG pipelines, and AI knowledge bases - removes boilerplate and outputs structured JSON with URL, title, markdown, and metadata.

IntelScrape

AST Website Content Crawler for RAG

lukas459/ai-web-to-markdown-crawler-llm-rag-optimized

Advanced web crawler extracting token-chunked Markdown for LLMs & RAG. Features dual-engine hybrid parsing (IBM Docling & Trafilatura) to preserve tables and layouts. Delivers clean text split by native tiktoken counts with rich metadata (headings, levels, tokens) for instant vector search.

lukas schmeck

5.0

RAG Spider - Web to Markdown Crawler for AI Training Data

lenient_grove/RAG-Spider

Enterprise-grade web crawler that converts messy websites into clean, chunked Markdown for AI systems. Uses Mozilla Readability for 95% cleaner extraction than competitors. Outputs RAG-ready data with metadata and token estimates. Perfect for building knowledge bases and training AI chatbots.