Pricing

from $1.40 / 1,000 results

Try for free

Go to Apify Store

Website to AI / RAG Data Extractor

Try for free

Turns any website into a clean, embedding-ready corpus. Strips navigation, footers and cookie banners, converts the real content to markdown, and splits it into overlapping chunks that each carry their own URL, title and metadata. Crawl by URL list, sitemap or link graph.

Pricing

from $1.40 / 1,000 results

Rating

0.0

(0)

Developer

Ibnu Adzim

Actor stats

Bookmarked

Total users

Monthly active users

3 days ago

Last modified

Categories

Developer tools

Agents

Start URLs

startUrls

Required

Where to start. Must be absolute http(s) URLs — bare domains like 'example.com' are dropped during normalisation. In 'urls' mode these are the only pages fetched; in 'sitemap' mode their sitemaps are read; in 'crawl' mode they seed the link graph.

Type:array

Discovery mode

discoveryMode

Optional

How to decide which pages to fetch. 'urls' fetches exactly the start URLs and follows nothing. 'sitemap' reads the Sitemap: lines from robots.txt (falling back to /sitemap.xml) and follows nested sitemap indexes. 'crawl' walks internal links breadth-first up to Max crawl depth.

Type:string

Default:urls

Options:

urlssitemapcrawl

Output granularity

outputGranularity

Optional

What a dataset row is. 'chunk' emits RAG-ready chunks, each carrying its own URL, title and metadata so it can go straight into a vector store. 'page' emits one row per page with the full cleaned markdown. 'both' emits pages and chunks, which roughly doubles dataset size.

Type:string

Default:chunk

Options:

chunkpageboth

Max pages

maxPages

Optional

Hard cap on how many pages are fetched across the whole run. This is the main cost control — a sitemap or crawl can otherwise expand to tens of thousands of URLs.

Type:integer

Minimum:1

Maximum:20000

Default:50

Max crawl depth

maxDepth

Optional

How many link hops from a start URL to follow. 0 means the start URLs only, 1 adds the pages they link to, and so on. Only used in 'crawl' mode.

Type:integer

Minimum:0

Maximum:10

Default:2

Stay on the same domain

sameDomainOnly

Optional

In 'crawl' mode, only follow links whose host matches a start URL's host (ignoring a leading www.). Turning this off lets the crawl wander onto any site the pages link to, which is almost never what a RAG corpus wants.

Type:boolean

Default:true

Include URL patterns (regex)

includeUrlPatterns

Optional

If set, a URL is only fetched when at least one of these regular expressions matches it. Useful for pulling just the docs section out of a sitemap, e.g. '/docs/' or '^https://example\.com/blog/'.

Type:array

Exclude URL patterns (regex)

excludeUrlPatterns

Optional

URLs matching any of these regular expressions are skipped. Applied before the include patterns. Good for dropping tag pages, paginated archives or login flows, e.g. '/tag/' or '?page='.

Type:array

Chunk size (characters)

chunkSize

Optional

Target chunk length in CHARACTERS, not tokens — a token count would depend on your tokenizer and be wrong for non-English text. Each chunk also reports an estimatedTokens figure. Chunks break at a paragraph, line or sentence boundary near the target rather than cutting mid-sentence.

Type:integer

Minimum:200

Maximum:20000

Default:2000

Chunk overlap (characters)

chunkOverlap

Optional

How many characters each chunk repeats from the end of the previous one, so a fact spanning a boundary is retrievable from either side. Must be smaller than the chunk size. Set to 0 to disable.

Type:integer

Minimum:0

Maximum:5000

Default:200

Include raw HTML

includeRawHtml

Optional

Attach each page's raw HTML to its PAGE row. Off by default because it multiplies dataset size several times over and a RAG pipeline consumes the cleaned text, not the markup.

Type:boolean

Default:false

Respect robots.txt

respectRobotsTxt

Optional

Check each host's robots.txt before fetching and skip anything it disallows, reporting the skip in the summary row. Leave this on unless you own the site or have written authorisation to crawl it. An unreachable robots.txt is treated as 'no policy published', which allows the fetch.

Type:boolean

Default:true

Max concurrent requests

maxConcurrency

Optional

Upper bound on pages fetched at the same time. Lower it for small or slow sites — this actor points at hosts nobody profiled, and hammering one is both rude and a fast route to a block.

Type:integer

Minimum:1

Maximum:30

Default:5

Min seconds between requests

minRequestInterval

Optional

Minimum delay between the START of consecutive requests. This, not concurrency, is the honest speed control: once the rate cap binds, extra concurrency buys nothing. Raise it if a site starts returning 429s.

Type:string

Default:0.2

Proxy configuration

proxyConfiguration

Optional

Proxy used for every request. Datacenter Apify Proxy is the default here rather than Residential: most documentation sites, blogs and company websites have no bot mitigation, so residential rates would be wasted. Switch to Residential for hosts that block datacenter traffic.

Type:object

Default:

{
  "useApifyProxy": true
}

Website to Markdown — Content Extractor for LLMs & RAG

insight.solutions/website-content-extractor

Turn any website into clean Markdown and plain text for RAG pipelines, vector stores and LLM agents. Reads the sitemap or crawls links, strips navigation, cookie bars and footers, and returns one row per page with metadata, headings, links and optional chunks. Static HTML, no browser.

Insight Solutions

Agent Memory Feeder - Web to Clean Markdown Chunks

apricot_blackberry/agent-memory-feeder

Feed your AI agent knowledge. Fetches any web page, strips nav and boilerplate with Mozilla Readability, converts to clean markdown, and splits it into overlapping, deduplicated, token-sized chunks ready for embeddings and RAG. One call turns a URL list into agent memory.

Creator Fusion

Website Content Crawler

myagizm/website-content-crawler

Crawl any website and extract clean, LLM-ready Markdown for RAG pipelines, vector databases and AI agents. Boilerplate, navigation and cookie banners removed. No browser, no login.

MYM

Web to Markdown Converter: AI-Ready Scraper for RAG & LLMs

raional/web-to-markdown-converter

Convert any webpage into clean Markdown or JSON for AI, RAG, and LLM pipelines. Strips ads, navigation, and cookie banners. Optionally follows links to convert an entire site. Powered by the open-source Crawl4AI library.

Raion Al

URL to Clean Markdown

brunofin/url-to-markdown

Extract clean, readable Markdown content from any URL. Strips ads, navigation, and clutter. Perfect for feeding web content into AI agents, RAG pipelines, or LLM training data.

Bruno Finger

Web to Markdown — AI-Ready Text from Any URL

wsgcjj/web-to-markdown

Convert any web page URL to clean Markdown format. Perfect for LLM training data, RAG pipelines, and AI content processing. Extracts main content, strips ads/nav/footers.

陈俊杰

Website to Markdown for LLMs

agentictools/website-to-markdown-llm

Crawl a site and export clean Markdown with token counts and chunks, ready for RAG.

Ken Agland

RAG Content Crawler — Any Site to Clean Markdown

eszetael_lab/rag-content-crawler

Turn any website into clean Markdown for your LLM or RAG pipeline. Navigation, cookie banners and boilerplate removed; every page says whether the text was truncated. Pay per delivered page — pages that returned nothing are free.

Radosław Szal

Website Content Crawler - Markdown & Text for LLM / RAG

pear_fight/website-content-crawler-markdown-text-for-llm-rag

Crawl any website and extract clean article text and Markdown, ready to feed into LLMs, ChatGPT, vector databases and RAG pipelines. Removes navigation, ads and boilerplate. Configurable crawl depth and page limits. Export to JSON, CSV, Excel.

Harald

Website Intelligence API

ladra/Website-Intelligence-API

Crawl any public website and turn it into AI-ready intelligence. Extract screenshots, Markdown, HTML, metadata, links, PDFs, compliance evidence, RAG chunks, and structured JSON for sales research, audits, website snapshots, and automation.