Website to Markdown Crawler avatar

Website to Markdown Crawler

Pricing

from $0.50 / 1,000 page saveds

Go to Apify Store
Website to Markdown Crawler

Website to Markdown Crawler

Turn a website into clean Markdown and plain text for AI: crawls the pages of a site, strips menus, footers and scripts, and saves each page's main content with headings, lists, links, tables and code blocks intact. Fast, with no browser, and it obeys robots.txt.

Pricing

from $0.50 / 1,000 page saveds

Rating

0.0

(0)

Developer

jake mcclure

jake mcclure

Maintained by Community

Actor stats

0

Bookmarked

2

Total users

1

Monthly active users

an hour ago

Last modified

Features

Limited permissions

Share

Turn a website into clean Markdown and plain text, ready for an AI assistant, a RAG pipeline, a knowledge base or a search index.

Give it a start page. It follows links to the other pages of the same website and saves the main content of each one: headings, paragraphs, lists, links, tables and code blocks are kept; menus, headers, footers, sidebars, cookie notices, forms and scripts are dropped.

No browser is used, so it is fast and cheap: a few hundred pages in a couple of minutes.

Use it to:

  • feed your documentation, help centre or blog to a chatbot or a custom GPT
  • build a RAG or fine-tuning dataset from a list of websites
  • keep an AI knowledge base up to date by running it on a schedule
  • convert a handful of pages to Markdown (set the page limit to 1)
  • archive or migrate the text of a site

Input

FieldWhat it does
Start pagesOne per line: the page to start from on each website, such as https://docs.example.com/ or just example.com.
Pages per websiteThe most pages saved for each start page. Default 50. Set it to 1 to convert only the pages you list.
Stay under the start page's folderOn: starting at example.com/docs/intro reads only pages under example.com/docs/. Off (default): any page of the website.
Keep images in the MarkdownInclude images as Markdown image links. Off by default.
Obey robots.txtOn by default: pages a website asks crawlers not to read are skipped.
{
"urls": ["https://docs.example.com/"],
"maxPagesPerWebsite": 200,
"stayUnderStartPath": true
}

Output

One record per page:

{
"startUrl": "https://docs.example.com/",
"url": "https://docs.example.com/guides/install",
"title": "Install | Example Docs",
"description": "How to install Example on Linux, macOS and Windows.",
"language": "en",
"depth": 2,
"foundOn": "https://docs.example.com/guides/",
"wordCount": 412,
"markdown": "# Install\n\nExample runs on Linux, macOS and Windows.\n\n## Requirements\n\n- Node.js 22 or newer\n- 200 MB of disk space\n\n```\nnpm install example\n```\n\nSee the [quick start](https://docs.example.com/guides/quick-start) next.",
"text": "Install\n\nExample runs on Linux, macOS and Windows.\n\nRequirements\n\n- Node.js 22 or newer\n- 200 MB of disk space\n\nnpm install example\n\nSee the quick start next.",
"scrapedAt": "2026-11-18T12:00:00.000Z"
}
  • markdown keeps headings, lists, links (as full addresses), tables, quotes and code blocks. text is the same content with no markup at all.
  • depth is how many clicks the page is from the start page, and foundOn is the page that linked to it.
  • title, description and language are what the page declares about itself.
  • Pages are read breadth first, so the pages nearest the start page come first when the limit cuts a crawl short.

A SUMMARY record in the run's key-value store lists, for each website, how many pages were saved and how many were skipped.

Pricing

Pay per result. You are charged once for each page saved to the dataset, at the per-result price shown on this page.

Pages that could not be read, pages robots.txt does not allow, files such as PDFs and images, and pages with no text are not saved and cost nothing. The run stops as soon as it reaches the maximum charge you set for it.

Limits

  • Pages are read without running their JavaScript. A site that builds its pages entirely in the browser (some single-page apps) yields little or no text; documentation sites, blogs, shops and company sites almost always work.
  • Only links to the same website are followed. A subdomain counts as a different website; add it as its own start page.
  • The main content is found from the page's own structure (<main>, <article>, roles and well-known class names). On unusual layouts a stray menu item may be kept, or a side note dropped.
  • The pages of one website are read one after another, half a second apart, so 1,000 pages of one site take about ten minutes. Several websites are crawled side by side.
  • Some sites refuse automated requests. The Actor identifies itself honestly and does not disguise itself or work around a refusal.
  • At most 5 MB of each page is read, and at most 20,000 pages per start page.

Is this allowed?

The Actor reads public pages the way a search engine's crawler does: it identifies itself in its User-Agent, obeys robots.txt by default and paces its requests to each site. You are responsible for how you use the text; most website content is protected by copyright, so check a site's terms before republishing it or training on it.