Markdownee

Also available as: Online playground npm package CLI & lib Python library on PyPI Source code on GitHub Docs Actor guide Social Star us on GitHub Follow us on GitHub

Markdownee is a web scraper that crawls websites and saves their content as Markdown, HTML or plain text for LLMs, retrieval pipelines, and research datasets.

Choose which links to follow, set page and depth limits, and select how much page content to keep. Control tables, links, images, and user comments separately.

• Boilerplate removal uses the TypeScript implementation of Trafilatura Core, our open-source extraction engine based on Trafilatura.

  The Core in its name means it is reduced to one task: extracting main content by removing boilerplate. Other packages handle output conversion, including Markdown.

  Core's TypeScript engine ports the original Python Trafilatura, with go-trafilatura as a DOM translation aid. Its native Python engine translates that port.

• Crawlee handles crawling and uses Playwright for browser rendering.

• Two language versions — TypeScript and Python: self-host with the npm CLI, npm library, or Python library, or use the hosted Apify Actor. Source code is on GitHub.

• Save image files with the optional image downloading mode.

This Actor saves the selected formats in Apify storage.

Select a Dataset route when content should appear in each result record, or a Key-value store route when you want downloadable files. The Actor can retain both representations. Start with a bounded run and inspect its outputs before increasing the crawl limits.

What Markdownee does

Select HTTP or browser fetching, choose precision/balanced/recall extraction or whole-document cleanup, and control links, tables, images, and user comments. Crawl scope uses link selectors, URL patterns, sitemaps, and request/depth limits. Optional image downloading keeps assets with the extracted content.

How to run a crawl

Open Input, provide starting URLs and settings, then select Start. This HTTP-only example extracts one Wikipedia page and saves Markdown in the key-value store:

{
  "startUrls": [{ "url": "https://en.wikipedia.org/wiki/Web_scraping" }],
  "crawlerType": "cheerio",
  "maxRequestsPerCrawl": 1,
  "save": ["markdown-kvs"]
}

Inspect the run's Dataset record and follow its Markdown key in the Key-value store. To follow links, add a selector and limits; globs alone do not discover links. The Input tab contains the full settings reference.

Results and storage

Dataset records describe outcomes, metadata, crawl information, and selected formats. A *-dataset route stores content inline; a *-kvs route keeps a file and its key. Select both to retain both representations. markdownSource identifies an accepted origin-published Markdown source when discovery is enabled.

See Output for record fields and API for starting runs and retrieving results. The Actor guide covers input examples and interpreting outcomes.

Pricing

Compute, storage, proxies, rendering, and crawl settings affect pay-per-use costs. Start with a representative bounded run. Apify offers $5 of free usage monthly; check its pricing page for current rates.

Troubleshooting

For missing content, compare extraction modes and the captured original. Use browser rendering and bounded waits for JavaScript content. Review filters, retries, and failed records when a crawl stops early; target sites can throttle or block requests.

Follow target-site terms and applicable rules when collecting and using content.

Support

Report reproducible Actor problems through its Issues tab with the settings and observed outcome.

License

Licensed under Apache-2.0.