Pricing

from $0.60 / 1,000 page crawleds

Website Intelligence Crawler

Crawl a public website and get each page as clean text, Markdown, metadata, same-site links and embedding-ready chunks. Robots-aware, no API key, no LLM call, no browser. Built for the boring part of a RAG pipeline. $0.60 per 1,000 pages plus a $0.001 start fee.

Pricing

from $0.60 / 1,000 page crawleds

Rating

0.0

(0)

Developer

Dami's Studio

Actor stats

Bookmarked

Total users

Monthly active users

3 days ago

Last modified

Website Crawler to Markdown & Text for RAG Pipelines API

Below, you can find a list of relevant HTTP API endpoints for calling the Website Intelligence Crawler Actor. For this, you’ll need an Apify account. Replace <YOUR_API_TOKEN> in the URLs with your Apify API token, which you can find under API & Integrations in Apify Console. For details, see the API reference.

Run Actor

POST
https://api.apify.com/v2/actors/dami_studio~website-intelligence-crawler/runs?token=<YOUR_API_TOKEN>

Note: By adding the method=POST query parameter, this API endpoint can be called using a GET request and thus used in third-party webhooks. Please refer to our Run Actor API documentation.

Run Actor synchronously and get dataset items

POST
https://api.apify.com/v2/actors/dami_studio~website-intelligence-crawler/run-sync-get-dataset-items?token=<YOUR_API_TOKEN>

Note: This endpoint supports both POST and GET request methods. However, only the POST method allows you to pass input data. For more information, please refer to our Run Actor synchronously and get dataset items API documentation.

Get Actor

GET
https://api.apify.com/v2/actors/dami_studio~website-intelligence-crawler?token=<YOUR_API_TOKEN>

For more information, please refer to our Get Actor API documentation.

Actors can be used to scrape web pages, extract data, or automate browser tasks. Use the Website Intelligence Crawler API programmatically via the Apify API.

You can choose from:

Website Intelligence Crawler API in Python

Website Intelligence Crawler API in JavaScript

Website Intelligence Crawler API through CLI

Website Intelligence Crawler OpenAPI definition

You can start Website Intelligence Crawler with the Apify API by sending an HTTP POST request to the Run Actorendpoint. An Actor’s input and its content type can be passed as a payload of the POST request, and additional options can be specified using URL query parameters. The Website Intelligence Crawler is identified within the API by its ID, which is the creator’s username and the name of the Actor.

When the Website Intelligence Crawler run finishes you can list the data from its default dataset(storage) via the API or you can preview the data directly on Apify Console.

Website to RAG Markdown Crawler

knotted_tussock/rag-markdown-crawler

Crawl any website or docs site and export clean Markdown plus JSONL-style chunks for RAG, LLM apps, and AI agents.

Ralph T

Website Content Scraper: Clean Markdown for AI and RAG

scrapemint/website-content-scraper

Crawl any website and get clean markdown, text, or HTML per page, ready for RAG pipelines, chatbots, and LLM fine tuning. Plain HTTP, no browser, no API key. Pay per page.

Ken M

Website to Markdown Crawler - Full-Site Text for LLMs & RAG

entranced_gelato/website-to-markdown-crawler

Crawl any website from a start URL and get every page as clean text + Markdown for LLMs, RAG, and AI agents. Follows internal links with depth and page limits, strips nav and ads, and returns one structured record per page. A fast, no-config site-to-Markdown crawler.

AIDevs

Site Crawler: Website → Markdown Corpus for LLM/RAG

boxbox10/site-crawler

Crawl a whole website or docs site and get one clean, LLM-ready Markdown + JSON record per page (title, headings, content, links, token count). Built for RAG ingestion and AI knowledge bases.

Marvin Eguilos

Website to Markdown for LLMs

agentictools/website-to-markdown-llm

Crawl a site and export clean Markdown with token counts and chunks, ready for RAG.

Ken Agland

Website to Markdown Crawler for LLM & RAG

logiover/website-text-markdown-crawler

Crawl any website to clean Markdown and plain text for LLM training and RAG. HTML to Markdown, no API or login. Export website text to CSV or JSON.

Logiover

Website to Markdown Crawler – Clean RAG Content

signal_lab/website-to-markdown-crawler

Website-to-Markdown crawler for RAG and LLM datasets. Extract clean content, SEO metadata, links, JSON-LD, sitemaps, and robots.txt-compliant pages.

Signal Lab

Website Content Crawler — Text, Markdown & HTML for AI/LLM

hichemdev/website-content-crawler

Crawl any website and extract clean text, Markdown, and HTML from every page — ready for LLM, RAG, and AI ingestion.

Hichem Ben Moussa

Website to Text & Markdown — AI / RAG Content Crawler

inexhaustible_glass/rag-website-crawler

Scrape any website into clean text & Markdown with RAG-ready chunks and token counts for LLMs, vector databases (Pinecone, Qdrant) and AI chatbots. Also extracts linked PDF/Word/Excel. Anti-block, robots.txt-aware. Website-to-text for beginners, full RAG pipeline for pros. CPU only, no API key.

Hitman studio

Website to Markdown – Clean LLM & RAG Content Extractor

dataquarry/website-to-markdown

Convert any public web page to clean, LLM-ready Markdown with metadata — by URL, a list of URLs, or a whole-site crawl. Strips nav/ads/boilerplate, keeps headings/lists/tables/code. Respects robots.txt. No API key.