Website to Markdown Crawler
Pricing
from $2.00 / 1,000 page saveds
Crawls a public website and saves clean Markdown for AI tools, docs and content audits. Same-site pages only, with robots.txt checked on every URL, heading-aware chunks, repeated boilerplate removed, and an optional change-only recrawl.
Public http or https pages to start from. The crawl stays on those hosts. Use a site you own, or a site whose robots.txt allows WebsiteToMarkdownCrawler. At most 10 start URLs. Private addresses are blocked.
[ "https://docs.python.org/3/tutorial/index.html"]How many links away from a start URL to follow. 0 reads the start URLs only. The ceiling is 5.
Most pages fetched in one run, from 1 to 50. The sample uses 10. URLs past the cap are not fetched and are not charged.
Optional patterns. When this list is not empty, a URL path must match one of them. Use * as a wildcard. The sample keeps the Python tutorial. robots.txt, sitemap.xml and llms.txt are still read.
[ "/3/tutorial/*"]Optional patterns. A matching URL is out of scope and is not charged. * is a wildcard.
How many fetches to run at once, from 1 to 4. Requests to the same host are still spaced out, and a crawl-delay is honoured up to 5 seconds.
Target size of a chunk, in estimated tokens. The ceiling is 2000. A heading starts a new chunk when the current one is already full.
Estimated tokens repeated at the start of the next chunk when a long section is split. The overlap is kept at or below half the chunk size.
When on, compare this crawl with the snapshot in the state store and return only added, changed or removed pages. Unchanged pages are not charged. Leave this off for a full crawl. The default run does not open a named store.
Named key-value store for the snapshot. Used only when Changes only is on. Letters, digits and hyphens. Leave empty to use website-markdown-state. If this token cannot open that store, the run uses the default store and still finishes.