Website to Markdown Crawler for AI & RAG

Give it one or more website addresses and get clean Markdown for every page — title, description, language, headings, word count and a content hash — ready for an AI agent, a RAG index or a knowledge base. It crawls over plain HTTP with no browser, reads the sitemap first, and respects robots.txt. $1 per 1,000 pages, no login, no API key.

Run it again — or put it on a daily schedule — and every page says whether it is new, changed or unchanged since the last run. Switch on Return only new and changed pages and you get (and pay for) only the pages whose main text changed, so keeping a RAG index fresh costs cents. The first run returns every page.

What you get

One page row per crawled page:

Field	What it is
url, canonicalUrl	the address the page was read from (after redirects) and the address its <link rel="canonical"> names (the same as url when the page names none)
title, description, language	<title>, the meta description and the lang of the page — null when the page does not state them
markdown	the main content as Markdown: headings, paragraphs, lists, links (absolute), bold/italic, code blocks, quotes and tables as GitHub tables
textLength, wordCount	characters and words of the main text (for Chinese, Japanese and Korean, about one word per two characters)
headings	the h1–h3 headings of the main content as {level, text}
linkCount	how many different links the main content holds
lastModified	<lastmod> from the sitemap, else the Last-Modified header, else null
contentHash	SHA-256 of the Markdown (first 32 characters) — the same text gives the same hash
status, previousContentHash, previousRunAt	new, changed or unchanged compared with the previous run for this start URL; previousContentHash is null the first time a page is seen, previousRunAt (when the start URL was last crawled) is null on its first run
depth, startUrl	clicks from the start page (pages taken from the sitemap count as 1) and the start URL that reached the page
html	the cleaned main content as HTML — only with Also return cleaned HTML
truncated, observedAt	true when the Markdown was cut at 1,000,000 characters or the page was too large to read to the end (see Limits); when the run started

Free rows next to the pages:

• summary — one per start URL: pagesFound, pagesCrawled, new, changed, unchanged, skippedByRobots, needsJavascript, tooShort, failed, notCrawled (found but beyond Maximum pages per start URL) and skippedOther (not HTML, duplicates, redirected away), plus firstRun, previousRunAt and changedOnly.
• note — a page that was not returned and why (reason): needs-javascript (the HTML holds almost no text because the page builds it with JavaScript — textLength is tiny), too-little-text, blocked-by-robots (the start URL itself is disallowed) or not-html.
• error — a page that could not be fetched (errorKind: not-found, blocked, rate-limited, network; statusCode).

Run it again and again

• On a schedule — save your input as a task and add a Schedule: the same site every day or week. Send each run to Google Sheets, Slack or email with an integration on the task.
• Watch for changes — Turn on Return only new and changed pages: from the second run, pages whose main text is the same as last time are not returned and not charged.
• In bulk — Put several sites in Start URLs in one run.

Use it for

• RAG and chatbots — turn a documentation site or help center into Markdown chunks, then re-run daily with only new and changed pages and upsert by url
• AI agents — give an agent current, readable text of a site without a browser tool
• Change monitoring — watch competitors' pricing, docs or terms pages: status: "changed" plus previousContentHash tells you which page moved
• Content audits — word counts, headings and missing descriptions for every page of a site

Input

{
  "startUrls": ["https://docs.apify.com/academy"],
  "maxPages": 50,
  "maxDepth": 3,
  "scope": "path",
  "useSitemap": true,
  "includeGlobs": [],
  "excludeGlobs": ["**/tag/**"],
  "changedOnly": false,
  "minTextLength": 200,
  "includeHtml": false,
  "includeImages": false
}

Globs match the whole address (https://example.com/blog/**; * stops at /, ** does not) — or only the path when they start with / (/blog/**). Start URLs are always crawled.

The state for new / changed / unchanged is kept per start URL in a named key-value store in your Apify account (website-markdown-crawler-state), so the second run of the same start URL already knows the first. Schedule it (Apify → Schedules) for a daily refresh.

Pricing

$1 per 1,000 pages (a page row), plus $0.01 per run. Summary, note and error rows are free. With Return only new and changed pages, unchanged pages are not returned and not charged: a 500-page documentation site where 12 pages changed costs about $0.02 after the first run.

Notes

• Plain HTTP, no JavaScript. Server-rendered sites (documentation, blogs, news, shops, company sites) work. A page that only shows text after JavaScript runs comes back as a free needs-javascript note, never as invented content. Pages behind a login cannot be read. Sites that block automated traffic from data centers (some news and shop sites answer HTTP 403) come back as free error rows with errorKind: "blocked"; the Actor uses no proxy and does not try to get around the block.
• robots.txt is respected for User-agent: * and for the groups addressed to Claude crawlers (ClaudeBot, Claude-User…) — a page is read only if both allow it (Disallow, Allow, * and $, the longest rule wins). A disallowed start URL is not fetched. Crawl-delay is not applied; the Actor makes at most 5 requests at a time per start URL. If robots.txt cannot be read, the crawl goes on.
• Where it goes. The start page first, then the sitemap (from robots.txt, /sitemap.xml, sitemap indexes and .gz files), then links, breadth first. By default it stays on the same site and under the start URL's folder (https://example.com/docs reads /docs and below); set Crawl scope to Whole site for everything on the host. Other websites are never followed. With a large sitemap index, reading stops once enough addresses are queued, so pagesFound counts what was found so far. Maximum pages per start URL counts fetched pages, including notes and errors.
• Main content. <main>, <article> or role="main", otherwise the block that holds most of the text. Navigation, headers and footers, sidebars, forms, cookie banners, share bars, comment sections and text hidden with CSS are left out.
• No contact data. E-mail addresses and obvious phone numbers are removed from the text, and mailto: and tel: links are dropped (an address on example.com is kept — it is not personal data). A link whose address contains an e-mail address is neither followed nor listed. Links to author pages and elements marked as bylines are left out, and pages that exist to show one person (/author/name/, /@name, /u/name) are not crawled unless you give one as a start URL; names that are written inside the body text stay, because that is the page's own text.
• What "changed" means. The Markdown of a page differs from the last run. Text that changes on every load (a visit counter, "posted 3 minutes ago") makes a page look changed every time, and so does switching Keep images in the Markdown on or off (every page's Markdown changes once). Up to 20,000 pages are remembered per start URL, the oldest dropped first. A page reached under two addresses is returned once when both name the same canonical address and have the same text.
• Language and location. Requests are plain and come from Apify's servers with Accept-Language: en-US. A site that picks its language, currency or prices by location or browser language may answer with its English or US version — start from the language folder you want (https://example.com/ja/).
• Limits. A page larger than 3 MB is cut at 3 MB, and a page with more than 60,000 HTML elements is read only as far as that (truncated: true; the largest pages we tried had about 22,000); non-HTML addresses (PDF, images) are skipped; addresses on private networks are refused; up to 5,000 pages per start URL and 100 start URLs per run.

For a blog's feed instead of its pages, see RSS Feed Reader. To see what a site is built with, see Tech Stack Detector. For pages that no longer exist, see Wayback Machine Scraper.

For AI agents

{"startUrls": ["https://docs.apify.com/academy"], "maxPages": 3} returns up to 3 rows with type: "page" (use markdown, keyed by url) and one type: "summary" row. To keep an index fresh, run on a schedule with "changedOnly": true and upsert the returned pages by url; rows with type note or error are free and explain what was skipped.

**Something broke or looks wrong?** Open an issue on this Actor's Issues tab and I'll look at it. **Found it useful?** A short review on the Store page helps other people find it.