Pricing

from $0.60 / 1,000 page crawleds

Try for free

Go to Apify Store

Website Intelligence Crawler

Try for free

Crawl a public website and get every page as clean text. You also get Markdown, page metadata, same-site links and chunks ready for embedding. It follows robots.txt. No API key, no LLM call, no browser. Built for the boring part of a RAG pipeline. $0.60 per 1,000 pages.

Pricing

from $0.60 / 1,000 page crawleds

Rating

0.0

(0)

Developer

Dami's Studio

Actor stats

Bookmarked

Total users

Monthly active users

4 days ago

Last modified

Categories

Developer tools

SEO tools

You can access the Website Intelligence Crawler programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1from apify_client import ApifyClient
2
3# Initialize the ApifyClient with your Apify API token
4# Replace '<YOUR_API_TOKEN>' with your token.
5client = ApifyClient("<YOUR_API_TOKEN>")
6
7# Prepare the Actor input
8run_input = { "startUrls": [{ "url": "https://example.com/" }] }
9
10# Run the Actor and wait for it to finish
11run = client.actor("dami_studio/website-intelligence-crawler").call(run_input=run_input)
12
13# Fetch and print Actor results from the run's dataset (if there are any)
14print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
15for item in client.dataset(run.default_dataset_id).iterate_items():
16    print(item)
17
18# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

Website Crawler to Markdown & Text for RAG Pipelines API in Python

The Apify API client for Python is the official library that allows you to use Website Intelligence Crawler API in Python, providing convenience functions and automatic retries on errors.

Install the apify-client

$pip install apify-client

Other API clients include:

Website Intelligence Crawler API in JavaScript

Website Intelligence Crawler API through CLI

Website Intelligence Crawler OpenAPI definition

Website Intelligence Crawler API

Website to Markdown Crawler - Full-Site Text for LLMs & RAG

entranced_gelato/website-to-markdown-crawler

Crawl any website from a start URL and get every page as clean text + Markdown for LLMs, RAG, and AI agents. Follows internal links with depth and page limits, strips nav and ads, and returns one structured record per page. A fast, no-config site-to-Markdown crawler.

AIDevs

Website to Markdown Crawler for LLM & RAG (fast, no browser)

webdatatools/website-to-markdown

Website to Markdown Crawler turns any site into clean Markdown per page for LLMs, RAG pipelines and vector DBs — no headless browser, $1 per 1,000 pages.

Murat Uzun

Website Content Scraper: Clean Markdown for AI and RAG

scrapemint/website-content-scraper

Crawl any website and get clean markdown, text, or HTML per page, ready for RAG pipelines, chatbots, and LLM fine tuning. Plain HTTP, no browser, no API key. Pay per page.

Ken M

Website to Markdown API & Crawler – RAG/LLM

signal_lab/website-to-markdown-crawler

Website-to-Markdown API and crawler for RAG and LLM datasets. Extract clean Markdown, SEO metadata, links, JSON-LD, sitemaps, and robots.txt-compliant pages.

Signal Lab

Site Crawler: Website → Markdown Corpus for LLM/RAG

boxbox10/site-crawler

Crawl a whole website or docs site and get one clean, LLM-ready Markdown + JSON record per page (title, headings, content, links, token count). Built for RAG ingestion and AI knowledge bases.

Marvin Eguilos

Website Content Crawler — Text, Markdown & HTML for AI/LLM

hichemdev/website-content-crawler

Crawl any website and extract clean text, Markdown, and HTML from every page — ready for LLM, RAG, and AI ingestion.

Hichem Ben Moussa

Website Content Crawler & Markdown Extractor

victor187/website-content-crawler-lite

Crawls a website and extracts clean, boilerplate-free text/Markdown from every page -- ready for RAG pipelines, LLM context, or content research. Respects robots.txt.