Website Content to Markdown (LLM-ready) — by Brainiall

Turn any website into clean, structured Markdown that is ready to feed into LLMs, RAG pipelines, AI agents, vector databases and knowledge bases — without writing a single selector.

Give it one or more URLs and get back distraction-free Markdown: headings, paragraphs, lists and links preserved; navigation chrome, cookie banners, scripts and boilerplate stripped out.

What it does

• Single-page scrape — convert exact URLs to Markdown (default mode).
• Site crawl — set Max pages per site above 1 and each start URL becomes the entry point of a bounded crawl (up to 25 pages, configurable link depth). Every page becomes one dataset item.
• Metadata included — page title, language, description and word count with every item.
• Links on demand — optionally return the hyperlinks found on each page.

Powered by the Brainiall Web engine (api.brainiall.com) — a production web-intelligence service built for AI workloads.

Who it's for

• RAG builders — ingest documentation sites, blogs and product pages straight into your vector store. Markdown chunks cleanly and embeds better than raw HTML.
• AI agent developers — give your agents reliable, token-efficient web content instead of noisy HTML.
• Data & content teams — archive or migrate site content as portable Markdown.
• LLM fine-tuning — collect clean text corpora from public websites.

Compliance-first by design

This Actor inherits the ethics posture of the Brainiall Web engine:

• robots.txt is honored — pages disallowed for crawlers are skipped, and crawl-delay directives are respected.
• No bot-detection bypassing — sites that choose to block automated access stay blocked.
• SSRF-guarded — internal/private addresses are never fetched.

If your compliance team asks how the data was collected, you have a clean answer.

Input

{
    "startUrls": [
        { "url": "https://docs.example.com" },
        { "url": "https://blog.example.com/post" }
    ],
    "maxPagesPerUrl": 10,
    "maxDepth": 2,
    "includeLinks": false
}

Field	Description
startUrls	Pages (or crawl entry points) to convert.
maxPagesPerUrl	1 = scrape only the exact URL. 2-25 = crawl the site from that URL.
maxDepth	Link hops allowed from the start URL (crawl mode).
includeLinks	Also return hyperlinks found on the page (single-page mode).

Output

One dataset item per page:

{
    "url": "https://example.com/",
    "requested_url": "https://example.com",
    "title": "Example Domain",
    "markdown": "# Example Domain\n\nThis domain is for use in documentation examples...",
    "word_count": 19,
    "metadata": { "description": "", "lang": "en" },
    "http_status": 200
}

Export as JSON, CSV, Excel or consume via the Apify API — ready for LangChain, LlamaIndex or any custom pipeline.

Pricing

You pay per result (per page successfully converted to Markdown). Pages that fail or contain no extractable content are not charged. No subscriptions, no minimums — costs scale exactly with what you extract.

Tips

• For documentation sites, start at the docs root with maxPagesPerUrl: 25, maxDepth: 3 to capture a whole section in one run.
• Keep includeLinks off unless you need them — smaller items mean faster downstream processing.
• Deduplicate by url if you crawl overlapping start URLs.

---

Built and maintained by Brainiall — production AI APIs for speech, documents, vision and the web.