Website to Markdown Crawler for LLM, RAG & AI Context
Pricing
$1.00 / 1,000 page converted to markdowns
Website to Markdown Crawler for LLM, RAG & AI Context
Crawl any website and get each page as clean Markdown with title, description, word count and token estimate. Menus, footers and cookie banners removed. Built by Physea Labs (Liminal). Launch Sale: 90% OFF for the first week. $0.001 per page.
Pricing
$1.00 / 1,000 page converted to markdowns
Rating
0.0
(0)
Developer
jay casey
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
a day ago
Last modified
Categories
Share
Give it a URL and get every page of the site as clean Markdown, ready to paste into ChatGPT, load into a vector database or feed an agent. Navigation, footers, cookie banners and scripts are stripped before conversion.
🚀 Launch Sale: 90% OFF for the first week! Prices below are the regular rates; launch-week runs are billed at a tenth of them.
What can it do?
| What you get | Features |
|---|---|
| 📝 Clean Markdown: headings, lists, tables, code blocks, links | ⚡ Plain HTTP, no browser: 5 pages in about 2 seconds |
| 🔢 Word count and token estimate per page | 🤖 robots.txt respected, same-site and same-path scoping |
| 🏷️ Title, meta description, language, canonical URL | 📤 JSON, CSV, API, LangChain / LlamaIndex loaders |
Who this is for
- RAG pipelines and vector databases that need clean page text
- Building a custom GPT or agent knowledge base from docs sites
- Content audits and migrations: every page's text in one dataset
- Feeding competitor docs or help centers into an LLM for analysis
What you get back
| Field | Type | What you get | Example |
|---|---|---|---|
url | string | Page URL | https://crawlee.dev/python/docs/quick-start |
title | string | Page | `Quick start |
markdown | string | Clean Markdown of the main content | # Quick start\n\nThis short tutorial... |
word_count | integer | Words in the Markdown | 1268 |
est_tokens | integer | Rough LLM token estimate (chars / 4) | 3790 |
depth | integer | Link hops from the start URL | 0 |
description | string | Meta description | This short tutorial will help you... |
canonical_url | string | Canonical URL | https://crawlee.dev/python/docs/quick-start |
status | string | ok, error or skipped | ok |
Every run also writes RUN_SUMMARY.json to the key-value store (counts, spend-limit flag, pricing).
What you need to provide
| Field | Type | What it does | Example |
|---|---|---|---|
startUrls | array | Pages to start from. Links on the same site (and under the same path, by default) are followed. | [{"url": "https://crawlee.dev/python/docs/quick-start"}] |
maxPages | integer | Stop after this many pages are converted. | 5 |
maxDepth | integer | 0 = only the start URLs, 1 = plus pages they link to, and so on. | 1 |
stayUnderStartPath | boolean | Only follow links below the start URL's folder (e.g. /docs/). | true |
includeGlobs | array | Optional glob patterns, e.g. https://example.com/blog/* | - |
excludeGlobs | array | Optional glob patterns, e.g. /tag/ | - |
keepLinks | boolean | Off gives plain text-only Markdown (fewer tokens). | true |
respectRobotsTxt | boolean | Skip pages the site disallows for crawlers. | true |
maxConcurrency | integer | Pages fetched at the same time. | 5 |
Quick start
- Click Try for free.
- Press Start: the sample input finishes in under a minute with no API keys or setup.
- Replace the sample with your own values and run again.
- Open the dataset, or export it as JSON, CSV or Excel.
Pricing
🚀 Launch Sale: 90% OFF for the first week!
page-converted: $0.001 per page converted to Markdown ($1 per 1,000 pages).- Failed, empty, non-HTML and robots-blocked pages are not charged.
- Set a max cost per run in the run options and the run stops cleanly when it hits it.
- Normal Apify platform usage applies.
Limits and honest notes
- Pages are fetched as HTML without a browser. Sites that render their text only with JavaScript come back as
errorrows ("no readable text") and are not charged. - PDFs, images and other non-HTML files are skipped.
- The crawl stays on the start URL's host; use several start URLs for several sites.
- No other Apify actor is called: everything is collected in-house over plain HTTP.
Run it from code
Same input as the Console sample:
It also works from Zapier, Make, n8n and Apify schedules.
About Physea Labs · Liminal
Built and maintained by Physea Labs: the same extraction layer that feeds Liminal, our grounded research and decision engine. If it is good enough to feed Liminal's citations, it is good enough for your pipeline.
