Website to Markdown Crawler avatar

Website to Markdown Crawler

Pricing

from $0.80 / 1,000 pages

Go to Apify Store
Website to Markdown Crawler

Website to Markdown Crawler

Crawl any website or search the web and get each page as clean Markdown for AI, RAG and LLM apps: main content without menus or ads, title, description, language and links. From $1 per 1,000 pages.

Pricing

from $0.80 / 1,000 pages

Rating

0.0

(0)

Developer

Tapaswai Choudhary

Tapaswai Choudhary

Maintained by Community

Actor stats

0

Bookmarked

2

Total users

1

Monthly active users

18 hours ago

Last modified

Features

Limited permissions

Share

Turn any website or web search into clean Markdown for AI, RAG and LLM apps. Give it a site and it reads the pages it links to; give it a question and it reads the top search results. Each page comes back as Markdown of its main content, without menus, headers, footers, cookie banners or ads, plus its title, description, language, author, publish date and, if you want, every link. Export to JSON, CSV or Excel, load into a vector store, or call it as an API.

Built for AI developers feeding knowledge bases and RAG pipelines, agents that need fresh web context, teams moving docs or help centres into a chatbot, and anyone who wants pages as clean text.

Why this crawler

  • Main content only. Navigation, sidebars, footers and ads are dropped; headings, lists, tables, code blocks and links are kept.
  • Crawl or search, or both. Start addresses are crawled on the same site; searches read the top results with their rank and snippet.
  • Stays where you point it. Starting at /docs follows /docs/... pages only. Skip links by text (/tag/, ?page=).
  • No duplicates. The same page reached by two links, a trailing slash or index.html is read once.
  • Fast and cheap. Pages are read five at a time; a 50-page docs site takes seconds.
  • Ready for chunking. Word count on every row; skip thin pages with a minimum word count.

How to use it

  1. Add Websites to crawl (for example a docs site), Web searches, or both.
  2. Set Link depth and Max pages per website.
  3. Press Start. Download results from the Output tab, or connect an integration.
{
"startUrls": ["https://docs.apify.com/academy"],
"queries": ["how does retrieval augmented generation work"],
"maxResultsPerQuery": 5,
"maxPagesPerSite": 50,
"maxDepth": 2
}

What you get

One row per page:

{
"url": "https://docs.example.com/guide/getting-started",
"title": "Getting started | Example Docs",
"description": "Install the app and run your first project in five minutes.",
"language": "en",
"markdown": "# Getting started\n\nInstall the app and run your first project...",
"wordCount": 2473,
"depth": 1,
"startUrl": "https://docs.example.com/guide",
"statusCode": 200,
"scrapedAt": "2026-11-15T18:56:22Z"
}

Search rows also carry query, rank, searchTitle and searchSnippet. Optional fields: text (plain text) and links.

Pricing

Pay per page: $1 per 1,000 pages, plus a small start fee per run. You pay only for pages delivered. Set a spending limit on the run to cap the cost.

FAQ

Does it read pages that need JavaScript? It reads the page as the server sends it, which covers docs sites, blogs, news, help centres and most company sites. Pages that only draw their text with JavaScript come back thin; set a minimum word count to skip them.

Can I get the whole page instead of the main content? Yes, turn on Whole page.

Does it follow links to other sites? No, crawls stay on the start address's site. Search results can come from any site.

Can I keep a knowledge base fresh? Yes. Schedule the run and send the dataset to your vector store or a webhook.

Support

Questions or a site that does not come out right? Open an issue on the Issues tab and include the run link.