# Changelog of Website to Markdown Crawler | LLM & RAG Ready, $1.50/1K (`aurelix/website-to-markdown`) Actor

- **URL**: https://apify.com/aurelix/website-to-markdown/changelog.md
- **Full Actor documentation**: https://apify.com/aurelix/website-to-markdown.md

## Changelog: Website to Markdown Crawler

### 0.1

- Store page: flat-price line and a comparison with other website-to-Markdown Actors; prices unchanged.
- The prefilled first run is capped at 20 pages so a test run costs only a few cents.
- A site that throttles the crawl (HTTP 429) is now named in the status message, so you know to try again later.
- Store page corrected: throughput is about 1,000 pages an hour (not 5,000), up to 3 requests are in flight per site, and the $0.001 start fee is stated.
- Cleaner Markdown: prose sidebars and footnotes inside the content are kept; Wikipedia language and category boxes, edit links and version switchers are removed.
- Start URLs that redirect within the same site are kept and crawled instead of being dropped.
- The run now stops itself before its timeout and ends SUCCEEDED with the pages done so far.
- About 3 times faster conversion, so roughly 1,000 large documentation pages fit in an hour at 256 MB.
- Pages over 2 MB are now skipped for free instead of exhausting the run's memory.
- First release: main-content Markdown, headings, links, language, last-modified, path or site scope, URL globs, robots.txt always obeyed.