Pricing

from $1.00 / 1,000 pages

Try for free

Go to Apify Store

Website & Page to Markdown for RAG and LLMs

Try for free

Turn web pages or whole sites into clean Markdown for RAG and LLMs: main content only, with title, links and word count. Whole-site mode reads the sitemap first, can return only changed pages, and splits pages into heading-based chunks. Respects robots.txt and AI opt-outs. No browser.

Pricing

from $1.00 / 1,000 pages

Rating

0.0

(0)

Developer

Michael Costa

Actor stats

Bookmarked

Total users

Monthly active users

11 hours ago

Last modified

Categories

Developer tools

Web page URLs

urls

Required

The pages to convert, one per line, e.g. https://docs.apify.com/storage; a missing https:// is added. With crawlWholeSite, each line is a site (https://docs.example.com), a section of one (https://docs.example.com/guide: only pages under /guide) or a sitemap URL. Pages the site's robots.txt disallows, or whose robots.txt opts out of AI crawlers, are skipped and listed in the log.

Type:array

Default:

[
  "https://docs.apify.com/storage",
  "https://docs.apify.com/storage/key-value-store"
]

Whole site (sitemap first)

crawlWholeSite

Optional

Default false. When true, convert every page of each site in urls instead of only the page itself: the pages its sitemap lists (found from robots.txt, /sitemap.xml or /sitemap_index.xml), or, with no usable sitemap, the pages reached by following its same-site links. A URL with a path limits it to pages under that path. Capped by maxPagesPerSite; crawlDepth and maxPagesPerUrl are ignored in this mode.

Type:boolean

Default:false

Max pages per site (whole-site mode)

maxPagesPerSite

Optional

Whole-site mode only (crawlWholeSite true): return at most this many pages for each URL in urls. 1 to 10,000, default 100. Unchanged pages skipped by onlyChangedPages don't count. At most 5 pages are fetched per page returned (plus 100), so a site where almost nothing changed can't run for ever.

Type:integer

Minimum:1

Maximum:10000

Default:100

Follow links (crawl depth)

crawlDepth

Optional

How far to follow links from each URL: 0 to 2, default 0. 0 converts only the pages you list. 1 also converts the same-site pages they link to, 2 goes one more link further, up to maxPagesPerUrl pages. Links to other sites and to files (PDF, images, ...) are never followed. Ignored when crawlWholeSite is true.

Type:integer

Minimum:0

Maximum:2

Default:0

Max pages per URL when following links

maxPagesPerUrl

Optional

When following links (crawlDepth 1 or 2): fetch at most this many pages for each URL in urls, the URL itself included. 1 to 1,000, default 10. No effect at crawlDepth 0 or with crawlWholeSite.

Type:integer

Minimum:1

Maximum:1000

Default:10

Only pages matching

includeUrlPatterns

Optional

When crawling (crawlWholeSite, or crawlDepth 1 or 2), convert only pages whose URL matches one of these patterns, one per line. Without * a pattern matches anywhere in the URL (e.g. /docs/); with , it must match the whole URL (e.g. https://example.com/blog/). Case-insensitive. Pages that don't match are never fetched. The URLs you list are always converted. Default: empty (every page).

Type:array

Default:

[]

Skip pages matching

excludeUrlPatterns

Optional

When crawling (crawlWholeSite, or crawlDepth 1 or 2), never fetch pages whose URL matches one of these patterns, one per line, same syntax as includeUrlPatterns, e.g. /tag/ or ?page=. Default: empty.

Type:array

Default:

[]

Only changed pages

onlyChangedPages

Optional

Default false. When true, return only pages that are new or changed since the last run of this same input. A sitemap page whose lastmod date hasn't changed is skipped without being fetched; any other page is fetched and skipped (not charged) when its Markdown is identical to last time. The memory is kept in a key-value store named page-to-markdown-changes in your account, separately for each URL and set of options.

Type:boolean

Default:false

Include images

includeImages

Optional

Default false. When true, keep images in the Markdown as ![alt text](image URL). Off by default, since images are usually noise for text models.

Type:boolean

Default:false

Split into chunks for RAG

chunkMarkdown

Optional

Default false. When true, also split each page's Markdown into chunks: one per heading section, with sections larger than maxChunkChars split at paragraph breaks. Each chunk carries its heading path (the headings it sits under), the page URL, title and its index. Where chunks go: chunkOutput. Chunks are free: you still pay per page.

Type:boolean

Default:false

Max chunk size (characters)

maxChunkChars

Optional

Only with chunkMarkdown: no chunk is longer than this many characters. 200 to 50,000, default 2000. A longer section is split at paragraph breaks, then at line breaks.

Type:integer

Minimum:200

Maximum:50000

Default:2000

Chunk output

chunkOutput

Optional

Only with chunkMarkdown. page (the default): each page's chunks go in a chunks array on its row. chunk: every chunk is written as its own row to this run's second dataset, named chunks (the default dataset keeps one row per page; chunk rows are not charged).

Type:string

Default:page

Options:

pagechunk

Max pages per run

maxResults

Optional

Stop after this many converted pages in total, e.g. 20. Minimum 1; leave empty (the default) for no limit. The run also stops cleanly at the maximum cost per run you set in the run options, whichever comes first. Set it to keep a run small and quick.

Type:integer

Minimum:1

SEO Audit Crawler

humble-echidna/seo-audit

Audit your own website's on-page SEO: crawl it from a start URL or sitemap and get, per page, title and meta description length, H1s, canonical, robots, hreflang, Open Graph, JSON-LD types, image alt, broken links, redirects, status, speed, word count and issues.

Michael Costa

PDF, DOCX & XLSX to Text

humble-echidna/document-to-text

Convert PDF, Word (DOCX) and Excel (XLSX) files from URLs into plain text, Markdown for LLMs and RAG, tables as arrays, and metadata. Scanned PDF pages and images are read with OCR in 32 languages. Pay per document, plus per OCR page.

Michael Costa

Website Pages to Clean Markdown for LLM and RAG

usta/url-to-markdown

Website to markdown for rag: turn public pages or a sitemap into llm ready text, one row per page. Headings, links and tables stay; menus and scripts go. robots.txt is always obeyed.

US Tech Automations

Website RAG & Lead Intelligence Crawler

datavaultlabs/website-rag-lead-intelligence

Convert public websites to clean Markdown for RAG and AI. Crawl pages and extract emails, phones, social profiles, metadata, technologies, headings and links.

Samuel Huirau Atutahi

AI Website Content Extractor

scrapeai/ai-website-content-extractor

Crawl website pages, strip noise, and convert the main content to clean Markdown for RAG/LLM training.

ScrapeAI

5.0

(2)

Local SEO Audit for Small Businesses (AI Agent Ready)

hereditary_model/local-seo-audit

Runs a local SEO audit on any business website: on page SEO, technical health, NAP consistency, LocalBusiness schema. Returns 0 to 100 scores and a fix list an AI agent can act on. About $0.70 per 10 page audit.

Aaron Marxsen

Website to Markdown — Clean Content for AI & LLMs

hipersoft/web-to-markdown

Turn any web page into clean Markdown — stripped of nav, ads and boilerplate — ready to feed an LLM, RAG pipeline or n8n AI workflow. Article text, title and metadata from a URL.

hiper soft

Web Page to Clean Markdown for RAG & LLM — 77% Fewer Tokens

craftbox/fast-markdown-for-rag

Convert web pages into clean Markdown for LLM and RAG pipelines, using 77% fewer tokens than a raw crawl while keeping every content section. Works in any language.

Craftbox

Web Page to Clean Markdown | LLM & RAG Content Extractor

kira_netrunner/fixer-grip

FREE single-page web-to-Markdown converter/scraper for LLMs, RAG pipelines, and AI agents. Converts web pages into clean, structured Markdown, strips HTML clutter, preserves <pre>/<code> blocks, auto-scrolls dynamic SPAs, and extracts outbound links. Includes full proxy integration. See README!