Pricing

from $0.20 / 1,000 pages

Try for free

Go to Apify Store

Website to Markdown: Sitemap Crawler for RAG + llms.txt

Try for free

Website to Markdown crawler for RAG and LLMs: sitemap to Markdown, token chunks, llms.txt, incremental sync of only new or changed pages. No browser, no login. Export data, run via API, schedule and monitor runs, or integrate with other tools.

Pricing

from $0.20 / 1,000 pages

Rating

0.0

(0)

Developer

Mohamed Bahaa

Actor stats

Bookmarked

Total users

Monthly active users

13 days ago

Last modified

Categories

Developer tools

SEO tools

Security

You can access the Website to Markdown: Sitemap Crawler for RAG + llms.txt programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1from apify_client import ApifyClient
2
3# Initialize the ApifyClient with your Apify API token
4# Replace '<YOUR_API_TOKEN>' with your token.
5client = ApifyClient("<YOUR_API_TOKEN>")
6
7# Prepare the Actor input
8run_input = {
9    "startUrls": ["https://docs.apify.com/academy"],
10    "maxPages": 20,
11    "chunkSize": 0,
12}
13
14# Run the Actor and wait for it to finish
15run = client.actor("chimey_forearm/sitemap-to-markdown").call(run_input=run_input)
16
17# Fetch and print Actor results from the run's dataset (if there are any)
18print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
19for item in client.dataset(run.default_dataset_id).iterate_items():
20    print(item)
21
22# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

Website to Markdown Crawler for RAG and llms.txt API in Python

The Apify API client for Python is the official library that allows you to use Website to Markdown: Sitemap Crawler for RAG + llms.txt API in Python, providing convenience functions and automatic retries on errors.

Install the apify-client

$pip install apify-client

Other API clients include:

Website to Markdown: Sitemap Crawler for RAG + llms.txt API in JavaScript

Website to Markdown: Sitemap Crawler for RAG + llms.txt API through CLI

Website to Markdown: Sitemap Crawler for RAG + llms.txt OpenAPI definition

Website to Markdown: Sitemap Crawler for RAG + llms.txt API

🔥 FireScrape AI Website Content Markdown Scraper

mohamedgb00714/fireScraper-AI-Website-Content-Markdown-Scraper

Advanced web scraper powered by Crawlee and Puppeteer — extracts website content, converts it to Markdown, and structures it for LLM training datasets.

mohamed el hadi msaid

305

1.9

(2)

Website to Markdown Scraper for AI & LLMs

quaking_pail/ai-website-content-markdown-scraper

Crawl a website and get clean Markdown of each page's main content, with URL and title. Ready for LLMs, RAG, vector databases and AI agents. Headless Chrome, same-domain crawl, API access.

AI_Builder

946

3.5

(2)

Knowledge Intelligence Engine — Website to Markdown for RAG

ryanclinton/website-content-to-markdown

Turn any website, documentation site or help centre into a retrieval-ready knowledge corpus for RAG and AI search. Clean Markdown plus chunks, change detection, deduplication, retrieval scoring, version awareness and a full corpus audit, in one run.

Ryan Clinton

Web-to-Markdown Generator for AI & RAG Pipelines

profitstack/web-to-markdown-generator-for-ai-rag-pipelines

Convert any website into clean, heading-based chunking, LLM-ready Markdown for RAG and AI agents.

Manas Mantri

AI Web Crawler

hounderd/ai-web-crawler

Crawl websites and extract clean, LLM-ready markdown content with stealth browser rendering, anti-bot hardening, smart content filtering, and structured metadata extraction. Built for RAG pipelines, AI agents, and data workflows.

Hounderd

Greenhouse Jobs Scraper: Workday, Lever + Visa Flags

chimey_forearm/ats-jobs-scraper

Greenhouse jobs and Workday jobs scraper, plus Lever, Ashby and 4 more ATSs in one schema. Visa sponsorship, US citizenship, ITAR and clearance flags, salary, remote, job alerts. No login. Export data, run via API, schedule and monitor runs, or integrate with other tools.

Mohamed Bahaa

Universal Markdown Scraper for LLMs

botflowtech/universal-markdown-scraper-for-llms

Universal Markdown Scraper for LLMs

BotFlowTech

Fast Website to Markdown & RAG JSONL Crawler

orbiscribe/website-rag-dataset-builder

Paste a homepage or sitemap and get clean Markdown, metadata, JSONL chunks, and source URLs for RAG at a low per-page price.

Orbiscribe Labs

Website to Markdown API & Crawler – RAG/LLM

signal_lab/website-to-markdown-crawler

Website-to-Markdown API and crawler for RAG and LLM datasets. Extract clean Markdown, SEO metadata, links, JSON-LD, sitemaps, and robots.txt-compliant pages.

Signal Lab

Website Content Extractor — Pages to Markdown for LLM/RAG

taroyamada/website-content-extractor

Turn docs, help centers and pricing pages into clean Markdown for LLM and RAG: one row per URL with main text, title, word count and a content-quality score. No API key or login.