Website Pages to Clean Markdown for LLM and RAG

This tool does website to markdown that is llm ready for rag: paste public page URLs or a sitemap and get one clean row per page.

Each row is the page title, the Markdown, a word count, and the fetch status. Headings, paragraphs, lists, links, code blocks and simple tables stay. Menus, headers, footers, sidebars, forms, scripts and cookie banners are dropped. robots.txt is always checked and cannot be turned off.

Input

Field	What it does
Page URLs	One or more public https page URLs. You can leave this empty if you set a sitemap.
Sitemap URL	Optional public https sitemap. A urlset is read directly. A sitemap index is read one level down, up to 10 child sitemaps.
Same site only	On by default. Sitemap links and redirects that leave the site of the URL you gave are not fetched. URLs you typed yourself are still fetched.
Maximum pages	How many pages to attempt. Default 50. Hard cap 5000.
Maximum rows	Hard ceiling on rows returned, and therefore on the cost of the run. Default 50. Hard cap 5000. The run stops at the smaller of this and Maximum pages.
Proxy configuration	Optional Apify proxy. Off by default, including on a local run.

There is no switch for robots.txt. It is always on.

Pricing

Pay per result. $0.003 per row. No start fee.

Every dataset row is one result, including a row that says the page was missing, blocked, disallowed, or not HTML. A run that returns no rows produces nothing to charge. Maximum rows is the spend cap. On Apify this package calls the charge event named result once per row. It does not call that charge on a local run.

Output (one row per page)

One row from a real local run on 2026-09-22 against https://docs.python.org/3/tutorial/index.html. The markdown field here is only the start of that page. word_count is the full count from that run: 2593.

{
  "url": "https://docs.python.org/3/tutorial/index.html",
  "final_url": "https://docs.python.org/3/tutorial/index.html",
  "http_status": 200,
  "status": "ok",
  "title": "The Python Tutorial — Python 3.14.7 documentation",
  "markdown": "# The Python Tutorial[¶](https://docs.python.org/3/tutorial/index.html#the-python-tutorial \"Link to this heading\")\n\nTip\n\nThis tutorial is designed for\n*programmers* that are new to the Python language,\n**not** *beginners* who are new to programming.\n\nPython is an easy to learn, powerful programming language. ...",
  "word_count": 2593,
  "fetched_at": "2026-09-22T19:31:23Z",
  "error": null
}

Field	What it holds
url	The URL requested
final_url	The URL after redirects, or null if the page was not fetched
http_status	HTTP status, or null if the page was not fetched
status	ok, robots_disallowed, not_html, not_found, blocked, off_domain, empty, or error
title	The HTML title, or null
markdown	The page as Markdown, or null when there is no content
word_count	Words in markdown, or null
fetched_at	When this run handled the URL, in UTC
error	null on success. A short reason otherwise

A disallowed URL has status robots_disallowed and no content. A PDF or image has status not_html and no content.

What this does not do

• It does not run a browser, log in, send cookies, or solve a CAPTCHA.
• It does not read text out of PDFs or images. Those come back as not_html.
• It does not turn robots.txt off. A disallowed path is not fetched.
• It does not follow a redirect onto another site when Same site only is on.
• It does not keep the footer. On the Python docs the copyright line sits in that footer, so the Markdown may not include it. If you republish the text, keeping the notice is your job.
• It does not give you a licence to republish anyone's site. You supply the URLs. Each site's terms still apply.
• It does not invent a title or a paragraph when the page could not be read. Unreadable fields are null.
• It does not fetch http:// URLs, local hosts, or URLs that contain a password.

Limits

• maxPages and maxItems are each 1–5000 (default 50). The smaller one wins.
• At least one second between requests to the same host. A long list is slow on purpose. One local page plus its robots.txt took 1.11 seconds on 2026-09-22. That is one measurement, not a promise.
• A page or sitemap over 5 MB is refused, not half-converted.
• If robots.txt asks for a crawl delay over 60 seconds, that host is not fetched.
• Plain HTTPS GET only.

Refunds and support

Refunds: if a run returns zero rows for a valid input, email operations@ustechautomations.com within 7 days and we refund that run. Support: same address, replies within 2 business days.