Website to Markdown for LLM & RAG + llms.txt Generator
Pricing
$1.00 / 1,000 page converted to markdowns
Website to Markdown for LLM & RAG + llms.txt Generator
Convert any web page or whole site (sitemap or crawl) to clean Markdown with metadata and token-sized chunks ready for RAG, embeddings and AI agents. Generates llms.txt and llms-full.txt. Respects robots.txt. Flat price per page. By IntegriApps.
Pricing
$1.00 / 1,000 page converted to markdowns
Rating
0.0
(0)
Developer
IntegriApps .com
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
12 days ago
Last modified
Categories
Features
Share
Turn any web page — or a whole website via its sitemap.xml or a same-site crawl — into clean Markdown ready for LLMs, RAG pipelines, embeddings, vector databases and AI agents. Every page comes with metadata (title, description, language, canonical URL, dates), a token estimate and token-sized chunks with overlap. With one switch it also generates llms.txt and llms-full.txt for your site.
Part of IntegriApps MicroApps. Support Sunday to Friday.
What it does
- Clean Markdown: removes navigation, headers, footers, sidebars, cookie banners, scripts and ads-like boilerplate; keeps headings, lists, links (made absolute), code blocks and tables as GitHub-flavored Markdown.
- Three modes:
urls(only the pages you give),sitemap(reads sitemap.xml from robots.txt or/sitemap.xml, including sitemap indexes),crawl(follows same-site links up to a depth). - RAG-ready chunks: each page is split into chunks of ~N tokens (1 token ≈ 4 characters) on heading/paragraph boundaries, with configurable overlap and the nearest heading.
- llms.txt generator: builds an llms.txt index and an
llms-full.txtwith all the Markdown, saved in the run's key-value store. - Native Markdown: if a site already serves Markdown (
text/markdown), it is used as is. - Deduplication: pages with identical content are skipped and not charged.
- Polite by design: always respects robots.txt (including
Crawl-delay), uses plain HTTP requests (no browser, no proxies), and does not bypass logins, paywalls or CAPTCHAs.
Use cases
- Feed documentation, help centers or blogs into a RAG chatbot or vector database (Pinecone, Qdrant, pgvector, Weaviate…).
- Give an AI agent / MCP client a tool that reads a URL and returns Markdown.
- Publish an llms.txt for your own website so AI assistants understand it.
- Archive or diff the text content of your pages.
Input
| Field | Description | Default |
|---|---|---|
startUrls | Pages (or site start points) to convert | — (empty = free demo with example.com) |
mode | urls, sitemap or crawl | urls |
maxPages | Max pages converted in the run (1–10,000) | 100 |
maxDepth | Link depth in crawl mode | 2 |
includeUrlPattern / excludeUrlPattern | Regular expressions to filter discovered URLs | — |
chunkSize / chunkOverlap | Chunk size and overlap in tokens (0 = no chunks) | 800 / 100 |
onlyMainContent | Keep only the main content | true |
keepImages | Keep image links | false |
removeSelectors | Extra CSS selectors to delete | — |
generateLlmsTxt, siteName, siteDescription | Build llms.txt / llms-full.txt | false |
maxConcurrency | Parallel requests (1–20) | 5 |
Example — whole docs section from the sitemap, with llms.txt:
{"startUrls": [{ "url": "https://crawlee.dev/" }],"mode": "sitemap","includeUrlPattern": "/js/docs/","maxPages": 25,"generateLlmsTxt": true}
Output
One dataset item per URL (real output from a crawl run with chunkSize: 300, shortened):
{"url": "https://crawlee.dev/js/docs/quick-start","status": "ok","httpStatus": 200,"metadata": { "title": "Quick Start", "description": null, "language": null, "canonicalUrl": null, "servedAsMarkdown": true },"markdown": "# Quick Start\n\nWith this short tutorial you can start scraping with Crawlee...","wordCount": 1301,"tokenEstimate": 4552,"contentHash": "…","chunks": [{ "index": 0, "text": "# Quick Start\n\n…", "tokens": 293, "heading": "Quick Start" }]}
status is one of ok, blocked_by_robots_txt, error, unsupported_content_type (e.g. PDF, images) or duplicate_content. Only ok is charged.
Key-value store: llms.txt, llms-full.txt (when enabled) and RUN_SUMMARY (counts per status and pages charged). Export the dataset as JSON, JSONL, CSV or Excel from Apify Console or the API.
Pricing
Pay per event: USD 0.001 per page converted (USD 1 per 1,000 pages), flat — no extra charge for chunks, metadata or llms.txt. Pages blocked by robots.txt, failed requests, unsupported files and duplicates are free. You can cap spending with the run's maximum charge setting; the Actor stops cleanly when the limit is reached.
Use it from your code
curl -X POST "https://api.apify.com/v2/acts/integriapps~web-to-markdown/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \-H "Content-Type: application/json" \-d '{"startUrls":[{"url":"https://example.com/"}],"chunkSize":0}'
It also works from the Apify API clients (JavaScript/Python), Make, Zapier, n8n, LangChain/LlamaIndex Apify integrations and the Apify MCP server.
FAQ
Does it render JavaScript? No. It uses fast HTTP requests, which works for most documentation sites, blogs and server-rendered pages. Pages that need a browser to show content may come out empty.
Can it read pages behind a login or a paywall? No, and it never tries to bypass logins, paywalls or CAPTCHAs.
Why was a page skipped? Check its status. blocked_by_robots_txt means the site's robots.txt does not allow it (or robots.txt was unreachable, in which case we stay on the safe side).
Is it legal to use? You are responsible for having the right to use the content you convert (your own site, public documentation, content whose terms allow it). The Actor respects robots.txt but does not read each site's terms of service for you.
How are tokens counted? As an estimate of 4 characters per token, which is close to common LLM tokenizers for English text.
Can it convert PDFs? Not in this version; PDFs are returned as unsupported_content_type and are not charged.
Changelog
See ./CHANGELOG.md.
Part of IntegriApps MicroApps — small tools by IntegriApps, a SaaS for SMBs. Support Sunday to Friday.