LLM-Ready Web Extractor — URL to Clean Markdown & JSON

Turn any web page or whole site into clean, LLM-ready Markdown and structured JSON. This web scraper extracts the main content of a page — stripping navigation, ads, cookie banners, and boilerplate — and returns tidy Markdown (or plain text / cleaned HTML) plus metadata. It's built for feeding LLMs, RAG pipelines, AI agents, and fine-tuning datasets clean web content instead of raw, noisy HTML.

Point it at one URL or hundreds, optionally crawl a site, and get back one structured record per page.

What it does

• Website to Markdown — converts article/doc/blog pages into clean Markdown that's ready to drop into an LLM prompt or vector database.
• Main-content extraction — removes menus, sidebars, footers, ads, and cookie notices, keeping the text that matters.
• Metadata — title, author, publish date, description, site name, and language where available.
• Optional crawling — follow same-domain links up to your page/depth limits to turn a whole docs site or blog into a corpus.
• Structured output — one JSON record per page (url, title, content, wordCount, metadata), exportable to JSON, CSV, or Excel.

Use cases

• Build a RAG knowledge base from documentation, help centers, or blogs.
• Feed LLM agents clean page content instead of raw HTML.
• Assemble fine-tuning / training datasets from public web pages.
• Give a chatbot up-to-date context from a set of URLs.

Input

Field	Description
startUrls	One or more pages to extract (entry points when crawling).
crawl	Follow same-domain links from the start URLs.
maxPages	Hard cap on pages extracted.
maxDepth	How many link-hops to follow when crawling.
onlySameDomain	Keep the crawl on the start URL's domain.
outputFormat	markdown (LLM-ready), text, or html.
includeLinks	Keep hyperlinks in the Markdown.

Output

Each page becomes one dataset item:

{
  "url": "https://example.com/docs/intro",
  "title": "Introduction",
  "content": "# Introduction\n\nClean markdown of the main content...",
  "format": "markdown",
  "wordCount": 812,
  "metadata": { "author": null, "date": "2025-04-01", "sitename": "Example Docs", "language": "en" }
}

Pricing

Pay-per-result: you're charged per page successfully extracted — no monthly fee, no charge for pages that fail or have no main content.

Notes

• Works best on server-rendered content (docs, articles, blogs, news). Heavily JavaScript-rendered pages are on the roadmap (browser-rendering toggle).
• Respects public, logged-out pages only — it does not log in or bypass paywalls.

FAQ

Does it handle a whole website? Yes — enable crawl and set maxPages / maxDepth.

What formats can I export? The dataset exports to JSON, JSONL, CSV, or Excel, or via the Apify API.

Is the Markdown clean enough for an LLM? Yes — that's the point: main content only, no nav/ads, links preserved.