Website to Markdown Scraper for AI & LLMs

Website to Markdown Scraper for AI & LLMs crawls a website from the start URLs you give it, renders each page in a real headless Chrome browser, and returns the main page content as clean Markdown with its URL and title. The output is ready for LLMs, RAG pipelines, vector databases, LangChain or LlamaIndex, chatbots and content analysis. Think of it as a simple website-to-Markdown API.

It stays on the start site's domain, removes navigation, headers, footers, scripts and images, plus a few common cookie-consent phrases, and does not log in, solve CAPTCHAs or bypass paywalls.

What does Website to Markdown Scraper for AI & LLMs do?

• Starts from one or more start URLs and follows internal links up to Maximum Depth, until Max URLs pages are scraped.
• Renders pages in headless Chrome, so JavaScript-built pages work in most cases.
• Extracts the page's main content once (a single <main> or <article> element when it holds most of the page text, otherwise <body>), so text is not repeated.
• Converts it to Markdown and cleans it: navigation, headers, footers, scripts and images are removed, plus inline SVGs and a few common cookie-consent phrases. Other cookie-banner text can remain in content.
• Follows a start URL that redirects to another domain (for example example.fr → example.com) and crawls the final domain.
• Optionally asks a search engine (Google, Bing or DuckDuckGo) for extra pages on the same domain before crawling, or skips that step with None.

Who is Website to Markdown Scraper for AI & LLMs for?

• AI and LLM developers building RAG, chatbots or knowledge bases from websites.
• Data and research teams collecting website text for analysis or classification.
• Marketers and SEO teams archiving or comparing site content.
• Sales teams that summarise prospect websites with an LLM.

Use cases for website-to-Markdown conversion

• Load a documentation site or help center into a vector database for a support chatbot.
• Give an AI agent the readable text of a company's website.
• Monitor content changes on competitor pages with a scheduled run.
• Build a clean text corpus for summarisation or classification.

What data can Website to Markdown Scraper for AI & LLMs extract?

Field	Type	Description
url	string	URL of the scraped page
title	string	Page title as shown in the browser tab
content	string	Main page content as cleaned Markdown

One dataset item per successfully scraped page.

How to convert a website to Markdown for LLMs

1. Open the Actor and go to the Input tab.
2. Add one or more Start URLs, for example the home page of a documentation site.
3. Set Maximum Depth (how many link levels to follow) and Max URLs (total pages).
4. Choose a Search Engine for extra page discovery, or None to crawl only from your start URLs.
5. Click Start and download the Markdown from the Output tab, or read it through the API.

How much will it cost to scrape a website to Markdown?

You pay only per result, no subscription and no rental. One result is one page stored in the dataset. Pages that fail are not stored and not charged. A small standard Actor start event also applies.

Current prices, including discounts on higher Apify plans, are on the Pricing tab. To keep costs predictable:

• Set Max URLs to the number of pages you need.
• Keep Maximum Depth at 1 unless you need deeper pages.
• Set Search Engine to None to skip the discovery step.
• Set a maximum cost per run in the run options.

Input

See the Input tab for all options. This is the input of the run used for the output below:

{
    "start_urls": [{ "url": "https://apify.com" }],
    "max_depth": 1,
    "max_urls": 5,
    "search_engine": "Google"
}

Output

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel. First item from a real 5-page run on apify.com, content shortened:

[
    {
        "url": "https://apify.com",
        "title": "Apify: Marketplace of ready-to-run tools for AI",
        "content": "# 89,424 tools for your AI Apify is a marketplace of ready-to-run tools for AI. Get real-time web data, track competitors, generate leads, … (3,443 characters in the full result)"
    }
]

The same run also crawled /store, /actors and the Console sign-in and sign-up pages linked from the homepage. Login pages contain little text, so their content is short.

Where to find the output

The run's Output tab lists these outputs, by the title it shows:

• Results: the default dataset, one row per page with url, title and content (view URLs and their titles; the full items, including content, are in the dataset and the API).

This Actor writes no extra key-value store records.

Tips for clean Markdown output

• Start from a content-heavy page (docs, blog, help center) rather than a page full of login links.
• Use a start URL on the final domain to skip a redirect.
• Scraping several pages per run is more efficient than many single-page runs, because the browser starts once per run.
• Search discovery depends on the search engine's results page and may add no pages. Your crawl from the start URLs works without it.

Integrations: API, AI agents (MCP), Make, Zapier and n8n

• Apify API: start a crawl and read the Markdown from Python, JavaScript or cURL. See the API tab.
• AI agents (MCP): add quaking_pail/ai-website-content-markdown-scraper through the Apify MCP server so Claude, Cursor or your own agent can read websites as Markdown.
• LLM frameworks: load the dataset into LangChain, LlamaIndex or a vector database with the Apify integrations or a simple API call.
• No-code: connect runs to Make, Zapier, n8n, Google Sheets or webhooks.

FAQ, disclaimers and support

Is it legal to scrape website content?

You should be aware that your results could contain personal data. Personal data is protected by the GDPR in the European Union and by other regulations around the world. You should not scrape personal data unless you have a legitimate reason to do so. If you're unsure whether your reason is legitimate, consult your lawyers.

Respect each website's terms of service and robots.txt, and check copyright before reusing content.

Does it work on JavaScript-heavy sites?

Pages are rendered in Chrome, so most JavaScript sites work. Sites that block bots or detect automation can still fail.

Why did I get fewer pages than Max URLs?

The site has fewer internal links within Maximum Depth, or some pages failed to load. Raise the depth or add more start URLs.

How do I report a problem?

Open an issue in the Issues tab with the run ID. If the Actor saved you time, a rating on the Reviews tab helps other users find it.

Related actors by quaking_pail

• SEO Audit & Markdown Content Scraper: the same Markdown content plus an on-page SEO audit and HTTP details for every page.
• Google Search Results to LLM Text: Google results for a query as one LLM-ready text.
• Website Contact Scraper: Emails & Phones: emails, phones and team names from company websites.
• LinkedIn Jobs Scraper: LinkedIn job listings with full descriptions.