Pricing

$3.00 / 1,000 page records

Go to Apify Store

Website to Markdown and RAG Dataset Crawler

Try for free

Crawl public websites into clean Markdown, text, metadata, links, JSON-LD, and chunks for RAG and knowledge bases.

Pricing

$3.00 / 1,000 page records

Rating

0.0

(0)

Developer

Orbiscribe Labs

Actor stats

Bookmarked

Total users

Monthly active users

a day ago

Last modified

Categories

Developer tools

Automation

Start URLs

startUrls

Optional

Web pages or sites to crawl.

Type:array

Default:

[
  {
    "url": "https://example.com/"
  }
]

Max pages

maxPages

Optional

Maximum number of pages to crawl.

Type:integer

Minimum:1

Maximum:5000

Default:25

Max depth

maxDepth

Optional

Maximum link depth from each start URL.

Type:integer

Minimum:0

Maximum:10

Default:2

Same domain only

sameDomainOnly

Optional

Only enqueue links on the same registered host as a start URL.

Type:boolean

Default:true

Respect robots.txt

respectRobotsTxt

Optional

Fetch and apply simple User-agent * robots.txt disallow rules for start domains.

Type:boolean

Default:true

Include markdown

includeMarkdown

Optional

Include markdown converted from the main page content.

Type:boolean

Default:true

Include HTML

includeHtml

Optional

Include cleaned HTML in each dataset item.

Type:boolean

Default:false

Dataset output mode

datasetOutputMode

Optional

Choose whether dataset rows should be RAG chunks, page records, or both. Full page records are always stored in PAGE_RECORDS.

Type:string

Default:chunks

Options:

chunkspagesboth

Chunk size chars

chunkSizeChars

Optional

Approximate character length for RAG chunks.

Type:integer

Minimum:500

Maximum:12000

Default:2500

Chunk overlap chars

chunkOverlapChars

Optional

Character overlap between adjacent chunks.

Type:integer

Minimum:0

Maximum:2000

Default:250

Dry run

dryRun

Optional

Emit one deterministic example.com sample without crawling.

Type:boolean

Default:false

Docs Markdown Rag Ready Crawler

devwithbobby/docs-markdown-rag-ready-crawler

Turn any documentation site or website into clean, structured markdown—ready for RAG, embeddings, and AI agents.

Dev with Bobby

LLM Markdown Crawler

sleek_waveform/llm-markdown-crawler

Crawl any website and extract clean, boilerplate-free Markdown optimized for LLMs, RAG pipelines, and AI training datasets. Uses Mozilla Readability to strip navigation and ads, then converts to clean Markdown. No browser required — fast and cheap.

Daniel Dimitrov

Sitemap to URL Crawler

logiover/sitemap-to-url-crawler

nstantly extract all public URLs from any website's sitemap.xml recursively. Handles nested sitemap indexes automatically. The fastest & cheapest way to build URL lists for RAG pipelines, LLM training, and SEO audits. Zero-config & blazing fast.

Logiover

Website to Markdown

logiover/website-to-markdown

Convert any URL to clean Markdown for AI & RAG. Strips ads & junk for noise-free data. Perfect for OpenAI, Pinecone & LangChain. Advanced stealth browsing bypasses anti-bots. Blazing fast, token-efficient extraction for AI Agents and Vector Stores. Your essential AI Data Architect.

Logiover

🕷️ Website Crawler — Full-Site Scraping for AI

nexgendata/website-content-crawler

Crawl entire websites for clean text, markdown or HTML. Perfect for RAG pipelines, AI training & content analysis. Handles JS-rendered pages. Alternative to Firecrawl & Jina. Pay per page.

Stephan Corbeil

Website to Markdown — AI-Ready Content for RAG

ryanclinton/website-to-markdown

Website To Markdown. Available on the Apify Store with pay-per-event pricing.

Ryan Clinton

Website to Markdown Crawler â€” AI/RAG Data Pipeline

sovereigntaylor/website-to-markdown

Crawl any website and convert every page to clean, structured Markdown. Perfect for RAG pipelines, LLM training data, vector database ingestion, knowledge base building, and AI-powered search. Extracts main content, strips boilerplate, handles metadata, and chunks output for embeddings. Works with L

Ricardo Akiyoshi

Website Content Crawler

crawlerbros/website-content-crawler

Crawls websites and extracts clean text, markdown, or HTML content. Ideal for LLM training data, RAG pipelines, and knowledge base building.

Crawler Bros

5.0

(28)

Website Content Extractor

taroyamada/website-content-extractor

Extract clean text and markdown from docs, pricing, product, policy, and help-center URLs for RAG datasets and content operations.

太郎山田

Website Content Crawler

tugelbay/website-content-crawler

Website content crawler for RAG and LLM pipelines. Crawl entire websites and extract clean Markdown. HTTP-first: 10x faster than browser-based crawlers. Recursive crawling with depth control. Perfect for knowledge base ingestion, documentation scraping.