Website to Markdown for LLM & RAG + llms.txt Generator avatar

Website to Markdown for LLM & RAG + llms.txt Generator

Pricing

$1.00 / 1,000 page converted to markdowns

Go to Apify Store
Website to Markdown for LLM & RAG + llms.txt Generator

Website to Markdown for LLM & RAG + llms.txt Generator

Convert any web page or whole site (sitemap or crawl) to clean Markdown with metadata and token-sized chunks ready for RAG, embeddings and AI agents. Generates llms.txt and llms-full.txt. Respects robots.txt. Flat price per page. By IntegriApps.

Pricing

$1.00 / 1,000 page converted to markdowns

Rating

0.0

(0)

Developer

IntegriApps .com

IntegriApps .com

Maintained by Community

Actor stats

0

Bookmarked

2

Total users

1

Monthly active users

12 days ago

Last modified

Features

Limited permissions

Share

Turn any web page — or a whole website via its sitemap.xml or a same-site crawl — into clean Markdown ready for LLMs, RAG pipelines, embeddings, vector databases and AI agents. Every page comes with metadata (title, description, language, canonical URL, dates), a token estimate and token-sized chunks with overlap. With one switch it also generates llms.txt and llms-full.txt for your site.

Part of IntegriApps MicroApps. Support Sunday to Friday.

What it does

  • Clean Markdown: removes navigation, headers, footers, sidebars, cookie banners, scripts and ads-like boilerplate; keeps headings, lists, links (made absolute), code blocks and tables as GitHub-flavored Markdown.
  • Three modes: urls (only the pages you give), sitemap (reads sitemap.xml from robots.txt or /sitemap.xml, including sitemap indexes), crawl (follows same-site links up to a depth).
  • RAG-ready chunks: each page is split into chunks of ~N tokens (1 token ≈ 4 characters) on heading/paragraph boundaries, with configurable overlap and the nearest heading.
  • llms.txt generator: builds an llms.txt index and an llms-full.txt with all the Markdown, saved in the run's key-value store.
  • Native Markdown: if a site already serves Markdown (text/markdown), it is used as is.
  • Deduplication: pages with identical content are skipped and not charged.
  • Polite by design: always respects robots.txt (including Crawl-delay), uses plain HTTP requests (no browser, no proxies), and does not bypass logins, paywalls or CAPTCHAs.

Use cases

  • Feed documentation, help centers or blogs into a RAG chatbot or vector database (Pinecone, Qdrant, pgvector, Weaviate…).
  • Give an AI agent / MCP client a tool that reads a URL and returns Markdown.
  • Publish an llms.txt for your own website so AI assistants understand it.
  • Archive or diff the text content of your pages.

Input

FieldDescriptionDefault
startUrlsPages (or site start points) to convert— (empty = free demo with example.com)
modeurls, sitemap or crawlurls
maxPagesMax pages converted in the run (1–10,000)100
maxDepthLink depth in crawl mode2
includeUrlPattern / excludeUrlPatternRegular expressions to filter discovered URLs—
chunkSize / chunkOverlapChunk size and overlap in tokens (0 = no chunks)800 / 100
onlyMainContentKeep only the main contenttrue
keepImagesKeep image linksfalse
removeSelectorsExtra CSS selectors to delete—
generateLlmsTxt, siteName, siteDescriptionBuild llms.txt / llms-full.txtfalse
maxConcurrencyParallel requests (1–20)5

Example — whole docs section from the sitemap, with llms.txt:

{
"startUrls": [{ "url": "https://crawlee.dev/" }],
"mode": "sitemap",
"includeUrlPattern": "/js/docs/",
"maxPages": 25,
"generateLlmsTxt": true
}

Output

One dataset item per URL (real output from a crawl run with chunkSize: 300, shortened):

{
"url": "https://crawlee.dev/js/docs/quick-start",
"status": "ok",
"httpStatus": 200,
"metadata": { "title": "Quick Start", "description": null, "language": null, "canonicalUrl": null, "servedAsMarkdown": true },
"markdown": "# Quick Start\n\nWith this short tutorial you can start scraping with Crawlee...",
"wordCount": 1301,
"tokenEstimate": 4552,
"contentHash": "…",
"chunks": [{ "index": 0, "text": "# Quick Start\n\n…", "tokens": 293, "heading": "Quick Start" }]
}

status is one of ok, blocked_by_robots_txt, error, unsupported_content_type (e.g. PDF, images) or duplicate_content. Only ok is charged.

Key-value store: llms.txt, llms-full.txt (when enabled) and RUN_SUMMARY (counts per status and pages charged). Export the dataset as JSON, JSONL, CSV or Excel from Apify Console or the API.

Pricing

Pay per event: USD 0.001 per page converted (USD 1 per 1,000 pages), flat — no extra charge for chunks, metadata or llms.txt. Pages blocked by robots.txt, failed requests, unsupported files and duplicates are free. You can cap spending with the run's maximum charge setting; the Actor stops cleanly when the limit is reached.

Use it from your code

curl -X POST "https://api.apify.com/v2/acts/integriapps~web-to-markdown/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
-H "Content-Type: application/json" \
-d '{"startUrls":[{"url":"https://example.com/"}],"chunkSize":0}'

It also works from the Apify API clients (JavaScript/Python), Make, Zapier, n8n, LangChain/LlamaIndex Apify integrations and the Apify MCP server.

FAQ

Does it render JavaScript? No. It uses fast HTTP requests, which works for most documentation sites, blogs and server-rendered pages. Pages that need a browser to show content may come out empty.

Can it read pages behind a login or a paywall? No, and it never tries to bypass logins, paywalls or CAPTCHAs.

Why was a page skipped? Check its status. blocked_by_robots_txt means the site's robots.txt does not allow it (or robots.txt was unreachable, in which case we stay on the safe side).

Is it legal to use? You are responsible for having the right to use the content you convert (your own site, public documentation, content whose terms allow it). The Actor respects robots.txt but does not read each site's terms of service for you.

How are tokens counted? As an estimate of 4 characters per token, which is close to common LLM tokenizers for English text.

Can it convert PDFs? Not in this version; PDFs are returned as unsupported_content_type and are not charged.

Changelog

See ./CHANGELOG.md.

Part of IntegriApps MicroApps — small tools by IntegriApps, a SaaS for SMBs. Support Sunday to Friday.