Pricing

Pay per event

Try for free

Go to Apify Store

Web Page to Markdown for LLMs (robots.txt-safe)

Try for free

Clean Markdown and RAG chunks from public pages or whole doc sites. Sitemap-first polite crawl: robots.txt, Crawl-delay, named bot, hard page/time budgets. Flat per-page price; cheap unchanged-page recrawls. No browser, no logins.

Pricing

Pay per event

Rating

0.0

(0)

Developer

Benjamin Cojocaru

Actor stats

Bookmarked

Total users

Monthly active users

6 days ago

Last modified

Categories

Developer tools

URLs (list mode) or start URLs (crawl mode)

urls

Required

List mode: the pages to convert (up to 500, duplicates removed). Crawl mode: up to 20 start URLs; the crawl stays on their host and inside their path (https://x.com/docs/intro stays under /docs/). Pages that robots.txt disallows are never fetched and never charged.

Type:array

Mode

mode

Optional

list: convert the URLs given. crawl: read the sites' sitemaps, then follow links inside the start path, up to maxPages and maxRunMinutes.

Type:string

Default:list

Options:

listcrawl

Max pages (crawl mode)

maxPages

Optional

Hard cap on pages fetched in crawl mode (sitemaps and robots.txt not counted). Discovered pages beyond it are counted as notFetched in SUMMARY.

Type:integer

Minimum:1

Maximum:1000

Default:100

Max link depth (crawl mode)

maxDepth

Optional

How many clicks away from a start URL the crawl may go (sitemap URLs count as depth 1). 0 = start URLs only.

Type:integer

Minimum:0

Maximum:20

Default:3

Use sitemaps first

useSitemaps

Optional

Read the sitemaps listed in robots.txt (or /sitemap.xml) before following links.

Type:boolean

Default:true

Follow links

followLinks

Optional

Follow links found on crawled pages (same host, inside the start path).

Type:boolean

Default:true

Only these paths

includePaths

Optional

Path prefixes such as /docs/ that replace the start-URL path scope. Empty = the start URLs' own directories.

Type:array

Default:

[]

Skip these paths

excludePaths

Optional

Path prefixes never fetched, e.g. /blog/tag/.

Type:array

Default:

[]

Remove blocks repeated across the site

removeRepeatedBlocks

Optional

Crawl mode: drop paragraphs that already appeared on 3 earlier pages of the same site (leftover banners, footers). Headings, code and tables are always kept.

Type:boolean

Default:true

Time budget (minutes)

maxRunMinutes

Optional

The run stops starting new pages after this time and always finishes cleanly with a SUMMARY (it also stops before the platform run timeout).

Type:integer

Minimum:1

Maximum:1440

Default:30

Main content only

mainContentOnly

Optional

Keep the main content only: drop navigation, headers, footers, sidebars, cookie and consent banners, share bars, newsletter boxes, related-post lists and comments. Tables, code blocks, headings and lists are kept. Turn off to convert the whole .

Type:boolean

Default:true

Structured tables

includeTables

Optional

Also return every HTML table as {caption, headers, rows} next to the Markdown.

Type:boolean

Default:true

Max Markdown characters per page

maxChars

Optional

Longer Markdown is cut and marked truncated: true.

Type:integer

Minimum:1000

Maximum:2000000

Default:200000

Max page size (MB)

maxPageMb

Optional

Pages larger than this are reported as errors (not charged).

Type:integer

Minimum:1

Maximum:10

Default:5

Incremental recrawl

incremental

Optional

Remember each page's ETag, Last-Modified and content hash in a named key-value store. Next time, unchanged pages are reported as status 'unchanged' and charged the much cheaper page-unchanged event.

Type:boolean

Default:false

State store name

stateStoreName

Optional

Named key-value store that keeps the recrawl state (letters, digits, '-'). Empty = one store per job, named from the mode, start URLs and paths. Set a name to share state between differently configured runs.

Type:string

Default:

Add RAG chunks

chunk

Optional

Also return token-budgeted chunks of each page's Markdown, in order, each with the URL, heading path and character offsets. Free.

Type:boolean

Default:false

Chunk size (approx. tokens)

chunkTokens

Optional

Approximate tokens per chunk (1 token ~ 4 characters).

Type:integer

Minimum:50

Maximum:8000

Default:512

Chunk overlap (approx. tokens)

chunkOverlapTokens

Optional

Repeated from the previous chunk; at most half the chunk size.

Type:integer

Minimum:0

Maximum:4000

Default:0

Notion Page Scraper | Extract Wiki, Tables & Markdown

forlex/notion-page-scraper

Scrape public Notion pages, databases, and wikis into clean JSON, Markdown, or HTML. Fully reliable Playwright engine that succeeds where others fail. Perfect for AI/RAG, backups, and data extraction.

Forlex Lab

5.0

(1)

AI-Powered Smart Web Scraper

cloud9_ai/ai-web-scraper

Intelligent content extraction from any website using Crawlee + AI. Auto-detects structure, adapts to layout changes, handles JavaScript rendering. No custom code needed. Extract articles, products, listings from 1000s of pages.

cloud9

PDF & DOCX to Markdown — Document Extractor for LLM/RAG

fetchbase/document-to-markdown

Convert PDF and Word (DOCX) documents into clean Markdown, text, or JSON. Smart PDF paragraph reflow, page markers for RAG citations, full DOCX structure (headings, lists, tables), custom auth headers. No browser — parses in seconds. Charged per page processed — no startup fee.

Fetchbase

Llm Ready Documentation Scraper

direct_duty/llm-ready-documentation-scraper

Developers and AI agents need to read documentation (e.g. Stripe Docs, Next.js Docs), but standard scrapers return noisy HTML that includes: navigation bars headers / footers ads / cookie banners This Actor must return pure content-only Markdown, suitable for vectorization and semantic search.

Sean

Web to LLM Spanish — RAG-ready content extractor

fanciful_geode/web-to-llm-es

Crawls Spanish-language websites and extracts clean markdown content optimized for LLM RAG pipelines. Handles Spanish character encoding, removes boilerplate, and outputs structured chunks ready for embedding.

Ernesto de Armas

Website Content Crawler - Clean Text, Markdown & RAG Chunks

topapi/website-content-crawler

Crawl any website or URL list for $1.20 per 1,000 pages: one row per page with clean text, Markdown, HTML, metadata, links, headings, tables and RAG chunks. Plain HTTP first, headless Chrome only for JavaScript pages. Sitemaps, llms.txt, PDF/DOCX text, Bing/DuckDuckGo search, change monitor.

Top Api

Doc2RAG AI - Documentation to LLM-Ready Markdown Converter

automation_studio/doc-to-rag-markdown-converter

Convert technical documentation websites (Stripe, Supabase, Next.js, Mintlify, Docusaurus) into clean, LLM-ready Markdown with preserved multi-language code blocks, tables, and RAG frontmatter. Exports JSON dataset, single llms-full.txt, and ZIP archive.

Automation Studio

Website to Markdown for RAG & LLMs

hereditary_model/website-to-markdown

Crawls a website and converts every page into clean, LLM-ready Markdown for RAG pipelines, vector databases, and AI agents. Removes nav, ads, and boilerplate. Predictable pricing: $0.004 per page converted.

Aaron Marxsen

Web Page to Single-Page PDF & HTML (Automation-Ready)

exciting_perfume/Web-Page-to-Single-Page-PDF-and-HTML

Convert webpages to single-page PDFs and extract raw HTML via API. Captures full scroll height (no A4 splits). Built for automation with n8n, Make, and Zapier. Ideal for archiving, AI workflows, compliance, and bulk processing.

Gavin Campbell

HTML to Markdown Converter - Bulk Web Content to MD

santamaria-automations/html-to-markdown

Extract main article content from any website and convert to clean Markdown including headings, links, images, tables, and code blocks. Perfect for LLM training, AI pipelines, and documentation. Export data, run via API, schedule and monitor runs, or integrate with other tools.