# Website to Markdown Crawler | LLM & RAG Ready, $1.50/1K (`aurelix/website-to-markdown`) Actor Extract website pages to Markdown at a flat $1.50 per 1,000 pages: 34% under Apify's RAG Web Browser, no compute-unit billing like Apify's Website Content Crawler, and failed, duplicate or empty pages never charged. Clean main-content Markdown for RAG and LLMs, headings and links kept. - **URL**: https://apify.com/aurelix/website-to-markdown.md - **Developed by:** [Aurelix](https://apify.com/aurelix) (community) - **Categories:** AI, Developer tools, MCP servers - **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $1.05 / 1,000 page results This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ### What is Website to Markdown Crawler? **Turn websites into clean Markdown your LLM can use, with menus, footers and cookie banners stripped, at a flat $1.50 per 1,000 pages, no login.** For RAG pipelines, AI agents and docs search. Plain HTTP with no browser, so runs start in seconds; robots.txt is always obeyed, and failed, duplicate or empty pages are free. Your first run with the example input costs about $0.031 (20 pages in our test run). [![Website to Markdown Crawler: example output from a real run](https://api.apify.com/v2/key-value-stores/twkCCtTWzsi8fVbx5/records/website-to-markdown-card-v3.png)](https://console.apify.com/actors/2kV4J94aGKwhX9obW?addFromActorId=2kV4J94aGKwhX9obW) [![Website to Markdown Crawler: 75-second demo video](https://api.apify.com/v2/key-value-stores/twkCCtTWzsi8fVbx5/records/website-to-markdown-demo-thumb-v1.png)](https://api.apify.com/v2/key-value-stores/twkCCtTWzsi8fVbx5/records/website-to-markdown-demo-v1.mp4) **[▶ Watch the 75-second demo](https://api.apify.com/v2/key-value-stores/twkCCtTWzsi8fVbx5/records/website-to-markdown-demo-v1.mp4)** | | | |---|---| | **Start from** | One or more start URLs | | **Scope** | Under the start path or the whole site; include and exclude URL globs | | **Output** | 13 fields per page; JSON, CSV, Excel, API | | **Automate** | Schedules, webhooks, Zapier, Make, n8n, Google Sheets, MCP for AI agents | | **Price** | $1.50 per 1,000 pages ($1.05 per 1,000 pages on paid Apify plans) | | **Login needed** | No | ### Why Website to Markdown Crawler users pick this one - **No compute-unit surprises.** One flat price per page; failed, duplicate and empty pages are free. - **RAG-ready rows.** Title, headings, word count and URL on every page for chunking and citations. - **Structure kept.** Headings, lists, tables, code blocks and links stay Markdown; links made absolute. - **34% under RAG Web Browser.** $1.50 per 1,000 pages against $2.28 for Apify's RAG Web Browser. - **Starts in seconds.** Plain HTTP with no browser; robots.txt and Crawl-delay always obeyed. ### Get started in 3 steps 1. Paste one or more start URLs, e.g. https://docs.apify.com/academy, and set Max pages. 2. Choose the scope: 'Under the start path' for one docs section, or 'Whole site'. Optionally add include or exclude URL globs such as **/blog/**. 3. Click Start, then download JSON, CSV or Excel, or send the pages to your vector database, LangChain, LlamaIndex, Zapier, Make or n8n through the API. **[▶ Try it with the example input](https://console.apify.com/actors/2kV4J94aGKwhX9obW?addFromActorId=2kV4J94aGKwhX9obW)** The form opens with an example input already filled in: that run costs about $0.031 on the free plan. ### How much does it cost to convert a website to Markdown? Pay per event: you pay only for what the run returns, plus a small start fee. Bronze, Silver, Gold and higher Apify plans pay less per result. | Pages | Free plan | Paid plans (Bronze, Silver, Gold) | |---|---|---| | 100 | $0.15 | $0.11 | | 1,000 | $1.50 | $1.05 | | 10,000 | $15.00 | $10.50 | Totals include the $0.001 start fee per run, rounded to the cent. Apify's free plan includes $5 of credit a month, enough for about 3,300 pages. Flat $1.50 per 1,000 pages: no compute-unit surprises. Skipped pages are not charged: pages blocked by robots.txt, fetch errors, non-HTML responses, duplicate URLs or identical Markdown, pages that need JavaScript and pages with no main content. #### Billing events | Event | Free plan | Paid plans (Bronze, Silver, Gold) | |---|---|---| | `page-result` | $1.50 per 1,000 pages | $1.05 per 1,000 pages | | `apify-actor-start` | $0.001 per run | $0.001 per run | ### Website to Markdown Crawler vs alternatives | | **This Actor** | apify/website-content-crawler | apify/rag-web-browser | |---|---|---|---| | **Price per 1,000 pages, free plan** | $1.50 | billed by compute units | $2.28 | | **Price per 1,000 pages, paid plans** | $1.05 | billed by compute units | $1.90 | | **Start fee per run** | $0.001 | none (compute units) | $0.0018 at 8 GB | | **Platform usage billed on top** | No | Yes | No | | **Crawls a whole site from start URLs** | ✓ | ✓ | ✗ (search results or given URLs) | | **Default memory per run** | 256 MB | 8 GB | 8 GB | | **Failed, duplicate and empty pages** | Not charged | Compute still billed | not stated | | **robots.txt** | Always obeyed | not stated | not stated | | **JavaScript rendering** | ✗ (server HTML only) | ✓ | ✓ | Prices checked regularly on each Actor's Store page. ### Benchmarks: how we compare | Test | Input | Result | Run time | Peak memory | |---|---|---|---|---| | First run (prefilled input) | docs.apify.com/academy, Max pages 20 | 20 pages for $0.031 in total | 28 s | 66 MB | | Throughput | docs.python.org/3/library, run timeout 600 s | 172 pages, stopped itself cleanly before the timeout (about 1,300 pages an hour) | 512 s | 129 MB | | Content accuracy | 10 real pages: Wikipedia, Python docs, MDN, GNU, Go, Martin Fowler, PostgreSQL docs, Apify docs, 2 practice shops | 98.5% to 100% of the content blocks kept on every page; 0 boilerplate blocks leaked | 6 s | 74 MB | | Huge page | one 15.8 MB HTML page | skipped, not charged, run SUCCEEDED | 2 s | 51 MB | | Large pages at once | 3 Wikipedia pages of about 1.9 MB each | 3 pages converted | 5 s | 130 MB | | Mixed bad inputs | an HTTP 429 URL, an HTTP 404 URL and 2 good sites, Max pages 4 | 4 pages; the 429 and the 404 named in the status and not charged | 16 s | 92 MB | | Blocked site | 3 start URLs all disallowed by robots.txt | 0 pages, 0 page charges, run SUCCEEDED | 3 s | 48 MB | Measured on Apify cloud runs at the default 256 MB, plain HTTP, no proxy, one run per row. Accuracy compares each page's Markdown with an independent parse of the same page's content blocks (10 pages). The first-run cost includes the $0.001 start fee; its time and memory come from a cost-test run on the same input. ### What data you get **Website to Markdown Crawler** crawls a website and turns each page into one row of clean Markdown of its main content, with navigation, headers, footers, sidebars, cookie banners and scripts stripped and headings, lists, tables, code blocks and links kept. Each row also carries the URL, title, meta description, language, last-modified date, headings, links and word count for chunking and citations. Crawl one docs section or the whole site, with include and exclude URL globs. - url (final, after redirects), title, meta description, language, last_modified (ISO 8601 UTC) - markdown: the page's main content as Markdown - headings: \[{level, text}] of the main content - links: absolute URLs linked from the main content - word_count, crawl_depth, http_status, scraped_at ![Website to Markdown Crawler: a real web page and the output this Actor returned for it](https://api.apify.com/v2/key-value-stores/twkCCtTWzsi8fVbx5/records/website-to-markdown-before-after-v1.png) | url | title | word_count | language | last_modified | markdown | |---|---|---|---|---|---| | https://docs.apify.com/academy | Apify Academy | Academy | Apify Documentation | 231 | en | 2026-11-04T01:19:44Z | # Apify Academy Learn everything about web scraping and ... | | https://docs.apify.com/academy/scraping-with-ai | Web scraping with AI and Apify | Academy | Apify Docume... | 419 | en | 2026-11-04T01:19:43Z | # Web scraping with AI and Apify \*\*Learn how to use AI t... | | https://docs.apify.com/academy/api-scraping | API scraping | Academy | Apify Documentation | 889 | en | 2026-11-04T01:19:43Z | # API scraping \*\*Learn all about how the professionals s... | | https://docs.apify.com/academy/anti-scraping | Anti-scraping protections | Academy | Apify Documentation | 2442 | en | 2026-11-04T01:19:43Z | # Anti-scraping protections \*\*Understand the various ant... | One full page record, exactly as a real run returned it (long text and lists shortened here). From a real run: ```json { "url": "https://docs.apify.com/academy", "title": "Apify Academy | Academy | Apify Documentation", "description": "Learn everything about web scraping and automation with our free courses that will turn you into an expert scraper devel... [5 more characters]", "language": "en", "last_modified": "2026-11-04T07:17:58Z", "markdown": "# Apify Academy\n\nLearn everything about web scraping and automation with our free courses that will turn you into an exp... [2378 more characters]", "headings": [ { "level": 1, "text": "Apify Academy" }, { "level": 2, "text": "Beginner courses" }, { "level": 4, "text": "Web scraping with AI" }, "... [8 more]" ], "links": [ "https://docs.apify.com/academy/scraping-with-ai", "https://docs.apify.com/academy/scraping-basics-javascript", "https://docs.apify.com/academy/scraping-basics-python", "... [5 more]" ], "word_count": 231, "crawl_depth": 0, "http_status": 200, "source_url": "https://docs.apify.com/academy", "scraped_at": "2026-11-04T11:25:08Z" } ``` ### Use cases - AI developers loading documentation sites into a vector database for RAG - Teams building a chatbot or support assistant on their own help center - Researchers turning blogs and knowledge bases into clean text - AI agents that need a site's pages as Markdown through MCP ### Use-case recipes #### Load a docs section into a vector database For AI developers building RAG on product documentation. Set `startUrls` to the docs section and keep `scope` on `path`. Split each row's `markdown` on its `headings`, embed the chunks, and store `url` and `title` with each chunk so answers can cite the page. ```json { "startUrls": [ "https://docs.apify.com/academy" ], "maxPages": 50, "scope": "path" } ``` #### Build a support chatbot from your help center For support teams. Point `startUrls` at the help center home, set `scope` to `domain`, and exclude pages that are not answers, such as search and tag listings. Feed `markdown` to the chatbot's knowledge base and use `last_modified` to spot articles nobody has updated in a long time. ```json { "startUrls": [ "https://example.com/help/" ], "maxPages": 100, "scope": "domain", "excludeGlobs": [ "**/search**", "**/tags/**" ] } ``` #### Copy a blog into clean text for research For researchers and writers. Use `includeGlobs` to keep only article URLs. Sort the rows by `word_count` to find the long-form pieces, and read `description` and `headings` for a quick outline before opening the full `markdown`. ```json { "startUrls": [ "https://example.com/blog/" ], "maxPages": 30, "scope": "path", "includeGlobs": [ "https://example.com/blog/**" ] } ``` #### Keep the corpus fresh after the first copy For teams that re-embed on a schedule. Run this Actor once for the full copy. Then schedule Website Change Tracker on the same `startUrls` and re-run this Actor only on the pages it reports as added or changed, so you re-embed only those. ```json { "startUrls": [ "https://docs.apify.com/academy" ], "maxPages": 20, "scope": "path" } ``` ### Input | Field | Type | What it does | Default | |---|---|---|---| | `startUrls` (Start URLs) | array | Pages to start from, e.g. \['https://docs.apify.com/academy']. | \["https://docs.apify.com/academy"] | | `maxPages` (Max pages) | integer | Stop after this many pages are converted (and charged), e.g. 100. | 50 | | `scope` (Crawl scope) | string | 'path' = only URLs under the start URL's folder (start https://example.com/docs/ crawls /docs/...), 'domain' = the whole site of each... | "path" | | `includeGlobs` (Include URL globs) | array | Follow only links whose full URL matches one of these patterns, e.g. \['https://docs.apify.com/academy/**']. | | | `excludeGlobs` (Exclude URL globs) | array | Never follow links whose full URL matches one of these patterns, e.g. \['**/changelog/**', '**?page=\*']. | | All 5 fields, with examples: see the **Input** tab. ### Switching from apify/website-content-crawler Already running `apify/website-content-crawler`? Map its input fields to this Actor's like this: | `apify/website-content-crawler` input | This Actor's input | Notes | |---|---|---| | `startUrls` | `startUrls` | Same field. Plain URL strings or `{"url": ...}` objects both work; a URL without http(s):// gets https://. | | `maxCrawlPages` | `maxPages` | Ours counts only pages converted and charged; skipped pages (robots.txt, errors, duplicates, non-HTML) do not count. Range 1 to 10,000. | | `maxCrawlDepth` | no equivalent | There is no depth limit; limit the crawl with `scope`, `includeGlobs`, `excludeGlobs` and `maxPages`. Each row reports its `crawl_depth`. | | `includeUrlGlobs` | `includeGlobs` plus `scope` | Theirs extends the crawl beyond the start URLs; ours narrows it: only links matching a pattern are followed. To go beyond the start folder, set `scope` to `domain`. | | `excludeUrlGlobs` | `excludeGlobs` | Same idea: `**` matches anything, `*` anything except `/`. | | `crawlerType` | not needed | Always plain HTTP with no browser. Pages that need JavaScript are skipped, listed in the log and not charged. | | `respectRobotsTxtFile` | not needed | robots.txt and Crawl-delay are always obeyed and cannot be switched off. | | `proxyConfiguration` | no equivalent | No proxy is used; pages are fetched directly with a named crawler user agent. | | `removeElementsCssSelector` / `htmlTransformer` | no equivalent | Main-content detection is automatic: menus, headers, footers, cookie banners and scripts are removed for you. | | `useSitemaps` | no equivalent | The crawl follows links from the start URLs only; sitemaps are not read. | - Price model changes from compute units to a flat $1.50 per 1,000 converted pages plus $0.001 per run; skipped pages are never charged. - Default memory is 256 MB, against 8 GB for website-content-crawler (comparison table above). - Each row carries `url`, `title`, `description`, `language`, `last_modified`, `markdown`, `headings`, `links`, `word_count`, `crawl_depth` and `http_status`; rename your downstream columns to match. - Duplicates are removed twice: a URL is returned once, and a page whose Markdown is identical to one already returned is skipped for free. ### Use with AI agents (Apify MCP server, Claude, ChatGPT, n8n AI agent) Website to Markdown Crawler is a ready-made tool for AI agents. Through the Apify MCP server an agent reads this Actor's input schema, fills in the input from your request, starts the run and reads the pages back. You pay the same per-result price as a run started by hand, from your own Apify account. #### Example prompts - "Crawl https://docs.apify.com/academy, up to 100 pages, and give me each page as Markdown with its title and URL so I can load it into a vector database" - "Turn every page under https://example.com/help/ into Markdown, skip anything under /changelog/, and list the pages with their word counts in a table" - "Fetch the whole site https://example.com as Markdown, at most 50 pages, and tell me which pages are longest and what headings they use" #### Claude (claude.ai and Claude Desktop) Open **Settings > Connectors**, choose **Add custom connector**, paste `https://mcp.apify.com/?tools=aurelix/website-to-markdown` and sign in to Apify when asked. Then ask Claude one of the prompts above. #### Claude Code, Cursor, VS Code and other MCP clients ```bash claude mcp add --transport http apify "https://mcp.apify.com/?tools=aurelix/website-to-markdown" ``` Clients configured with JSON take the same URL: ```json { "mcpServers": { "apify": { "url": "https://mcp.apify.com/?tools=aurelix/website-to-markdown" } } } ``` A client without Apify sign-in sends the header `Authorization: Bearer `; to run the server locally instead, use `npx -y @apify/actors-mcp-server --actors aurelix/website-to-markdown` with `APIFY_TOKEN` set. #### ChatGPT Where your ChatGPT plan offers custom connectors (developer mode), add `https://mcp.apify.com/?tools=aurelix/website-to-markdown` as a connector, sign in to Apify, and ask for the data in plain words. #### n8n AI agent Give an **AI Agent** node an **MCP Client Tool** with the endpoint `https://mcp.apify.com/?tools=aurelix/website-to-markdown` and a Bearer credential holding your Apify token. The agent can then call Website to Markdown Crawler whenever a workflow step needs pages; for a fixed, non-agent workflow use Apify's n8n node with **Run Actor** instead. ### Use it from AI agents (MCP), the API or no-code #### Copy this to your AI assistant ```text Use the Apify Actor aurelix/website-to-markdown (Website to Markdown Crawler) through Apify's MCP server or API. Main input: startUrls (Start URLs). Example input: {"startUrls": ["https://docs.apify.com/academy"], "maxPages": 100, "scope": "path", "excludeGlobs": ["**/changelog/**"]}. Each result is one page with fields such as url, title, description, language, last_modified, markdown, headings, links. It costs $1.50 per 1,000 pages on Apify's free plan plus a $0.001 start fee per run, and needs no login. ``` MCP server URL for this Actor: `https://mcp.apify.com/?tools=aurelix/website-to-markdown` (setup for each client in the section above). #### API Start a run and read the dataset with Apify's REST API or the Python and JavaScript clients: ```python from apify_client import ApifyClient client = ApifyClient("") run = client.actor("aurelix/website-to-markdown").call(run_input={"startUrls": ["https://docs.apify.com/academy"], "maxPages": 100, "scope": "path", "excludeGlobs": ["**/changelog/**"]}) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) ``` #### No-code: schedule it in 3 clicks Send the results to Google Sheets, Zapier, Make, n8n, Slack or any webhook URL. 1. Fill in the input, then click **Save as a new task**. 2. In the task, click **Schedule** and pick daily or weekly (Apify Schedules). 3. Open the task's **Integrations** tab to send every run's results to Slack, Google Sheets, email or any webhook. Prefer n8n or Make? Ask in the **Issues** tab and we will share a ready-made workflow for it. ### FAQ #### Can I convert a website to Markdown without login? Yes. It never logs in and needs no account on the site, only a free Apify account to run it. It reads the pages a logged-out visitor can see. #### Is there a website to Markdown API? You can use this Actor as one. Start it with Apify's REST API or the Python and JavaScript clients and read the results as JSON (see the code above), or call it from an AI agent through MCP. #### How much does it cost to convert a website to Markdown? About $1.50 for 1,000 pages on Apify's free plan and $1.05 on paid plans, start fee included. The free $5 monthly credit covers about 3,300 pages. #### Can I export the Markdown to Excel or Google Sheets? Yes. Download the results as Excel, CSV, JSON, XML or HTML from the run's **Storage** tab, or send them to Google Sheets with Apify's Google Sheets integration or the API. #### Is it legal to scrape websites for RAG? It reads only public data any logged-out visitor can see: the website you choose (crawl only sites you have the right to crawl). It goes at a polite pace, respects robots.txt, never logs in and never solves CAPTCHAs. It returns each page's main content as Markdown with title, headings and links; the content is returned as the site published it; copyright in it stays with its owner. To have a record removed, open an issue on this Actor's Issues tab; if a site owner formally asks us to stop, this Actor will stop. You are responsible for using the data lawfully, incl. anti-spam and privacy laws; this is not legal advice. #### Does it work for JavaScript-heavy sites? No. It uses plain HTTP without a browser, so pages that need JavaScript to show their content are skipped and not charged. #### Does it respect robots.txt? Always, and it cannot be switched off. Disallowed URLs are never requested, Crawl-delay is honoured, redirects into disallowed paths are not followed, and a site whose robots.txt returns a server error is not crawled. Links on pages marked nofollow are not followed. #### What is 'main content'? The page's
or
element when it has one, otherwise the largest content block. Navigation, headers, footers, tables of contents, cookie banners, site banners, version switchers, Wikipedia language and category boxes, share buttons, forms, scripts and hidden elements are removed. Prose sidebars and footnotes inside the content are kept; link-list sidebars are dropped. #### Does it run JavaScript? No. It reads the HTML the server sends, which covers documentation sites, blogs, news, knowledge bases and most company sites, including server-rendered React and Next.js. Pages that are an empty JavaScript app shell are detected, listed in the log and not charged. A headless browser would need about 1 GB of memory per run, so it is left out to keep runs cheap. #### How do I crawl only part of a site? Use the start URL's folder (scope 'Under the start path'), or include and exclude globs: '**' matches anything, '\*' anything except '/'. Example: include https://example.com/docs/** and exclude **/v1/**. #### Are duplicate pages charged? No. A URL is returned once, and a page whose Markdown is identical to one already returned (e.g. / and /index.html) is skipped for free. #### How do I keep the corpus fresh? Run this Actor once for the full copy, then re-crawl only what changed with our Website Change Tracker on the same start URLs: it returns added, changed and removed pages, so you re-embed only those. #### What am I charged when a run converts nothing? Only the $0.001 start fee. Pages blocked by robots.txt, fetch errors, HTTP errors such as 404, 429 and 500, non-HTML files, duplicates, pages over 2 MB, pages that need JavaScript and pages with no main content are never charged. The run's status message lists how many pages were skipped for each reason. #### Does Max pages count skipped pages? No. Max pages counts only pages converted and charged, so a crawl with many errors or duplicates still returns up to that many pages if the site has them. The highest value is 10,000. #### How many pages fit in one run? About 1,000 to 1,300 large documentation pages an hour at the default 256 MB, because the conversion is CPU-bound. A run with the default 1-hour timeout fits roughly 1,000 to 3,000 pages, depending on page size. The run stops itself before its timeout and ends SUCCEEDED with the pages done so far, so raise the timeout or the memory for bigger crawls. #### Does it retry when a site is slow or throttles it? Yes. HTTP 429 and 5xx responses and connection errors are retried up to 3 times in all, waiting for the site's Retry-After header when it sends one (up to 30 seconds). A page still failing after that is skipped and not charged; a 429 is named as throttled in the status message, so you know to try again later. #### Does it use proxies? No. It fetches pages directly, identifies itself as AurelixBot in the user agent, and sends at most two new requests a second per site with up to 3 in flight. Sites that block bots or show a challenge page will not be converted. #### Does it follow links to other subdomains or sites? No. www. and the bare domain count as one site; other subdomains and other sites are not followed. Add each subdomain as its own start URL if you need it. #### What do crawl_depth and http_status mean? crawl_depth is the number of links followed from a start URL to reach the page (0 for the start URL itself). http_status is the final HTTP status of the page after redirects; it is always 200 on returned rows, because other statuses are skipped. #### Does it work in other languages? Yes. The Markdown keeps the page's own text in any language, and the language field is read from the page's HTML lang attribute (empty when the page does not set one). ### Limits (stated honestly) - No JavaScript rendering: client-side-only pages (an empty app shell) are skipped and listed in the log, not converted. - Only HTML pages are converted; PDFs, images and other files are skipped. - Scope is one site per start URL: www. and the bare domain count as one site, but other subdomains are not followed. - At most two new requests per second per site (slower when robots.txt sets a Crawl-delay). Conversion is CPU-bound: about 1,000 pages an hour on large documentation pages at 256 MB, so the default 1-hour run fits roughly 1,000 to 3,000 pages. The run stops itself before its timeout and ends SUCCEEDED with the pages done so far; raise the timeout or memory for bigger crawls. - HTML pages over 2 MB are skipped with a log line and not charged, to keep memory low. - Main-content detection is automatic; on unusual layouts a little boilerplate can remain or a side block can be dropped. - Pages behind a login, paywall or bot challenge are not accessible. ### Related Aurelix tools Public Aurelix Actors, grouped by the data they collect. All are pay-per-result and need no login. #### AI, content and SEO - **[Google Trends Scraper](https://apify.com/aurelix/google-trends-scraper)**: Google Trends interest over time, by region and rising queries for any number of keywords, plus trending searches. #### Jobs and hiring data - **[Greenhouse Jobs Scraper](https://apify.com/aurelix/greenhouse-jobs-scraper)**: Live jobs from a company's Greenhouse, Lever, Ashby or SmartRecruiters board, with salary and Canadian province tags. - **[LinkedIn Jobs Scraper](https://apify.com/aurelix/linkedin-jobs)**: Public LinkedIn job postings by keyword and country: title, company, location, seniority, job type, salary text, description. #### E-commerce and product data - **[AliExpress Scraper](https://apify.com/aurelix/aliexpress-product-search)**: AliExpress search results by keyword: price, discount, rating, units sold, ship-from country and sponsored flag. - **[Amazon Best Sellers Scraper](https://apify.com/aurelix/amazon-best-sellers)**: Amazon Best Sellers and New Releases top 100 on 10 marketplaces: rank, ASIN, price, rating, rank change. #### Leads and company data - **[Google Maps Business Scraper](https://apify.com/aurelix/google-maps-scraper)**: Local business lists from Google Maps: phones, websites, ratings, hours, coordinates; optional website emails and socials. - **[Recently Funded Companies](https://apify.com/aurelix/recently-funded-companies)**: US companies that just filed an SEC Form D: amount raised, industry, state, executives, confirmed website. #### Ad intelligence - **[Facebook Ad Library Scraper](https://apify.com/aurelix/facebook-ad-library-scraper)**: Facebook and Instagram business ads from the Meta Ad Library: copy, creatives, landing pages, dates, optional EU reach. #### Canada and government data - **[Canada Building Permits Scraper](https://apify.com/aurelix/canada-building-permits-scraper)**: Building permits and development applications from 7 Canadian cities: address, type, status, dates, value, contractor company. ### Recent changes - **0.1**: Store page: flat-price line and a comparison with other website-to-Markdown Actors; prices unchanged. The full history is on the **Changelog** tab. ### Tested daily An automated health check runs this Actor every day on a fixed input and checks that at least 4 pages come back with `url`, `title`, `markdown`, `word_count`, `source_url`, `scraped_at` filled in; a failed check alerts the maintainer straight away. ### Your feedback Missing a field, found a bug or need another filter? Open an issue on the **Issues** tab; we aim to answer within a day. If this Actor saved you time, a short review helps other people find it. # Changelog This Actor's version history is a separate document: https://apify.com/aurelix/website-to-markdown/changelog.md # Actor input Schema ## `startUrls` (type: `array`): Pages to start from, e.g. \['https://docs.apify.com/academy']. Links on these pages are followed within the chosen scope. A URL without http(s):// gets https://. ## `maxPages` (type: `integer`): Stop after this many pages are converted (and charged), e.g. 100. Pages that are skipped (robots.txt, errors, duplicates, non-HTML) do not count. About 1,000 large documentation pages fit in an hour at 256 MB; if the run timeout comes first, the run stops cleanly with the pages done so far. ## `scope` (type: `string`): 'path' = only URLs under the start URL's folder (start https://example.com/docs/ crawls /docs/...), 'domain' = the whole site of each start URL (www. and the bare domain count as one site; other subdomains are not followed). ## `includeGlobs` (type: `array`): Follow only links whose full URL matches one of these patterns, e.g. \['https://docs.apify.com/academy/**']. '**' matches anything, '\*' anything except '/'. Empty = every in-scope URL. Start URLs are always fetched. ## `excludeGlobs` (type: `array`): Never follow links whose full URL matches one of these patterns, e.g. \['**/changelog/**', '\*\*?page=\*']. ## Actor input object example ```json { "startUrls": [ "https://docs.apify.com/academy" ], "maxPages": 20, "scope": "path", "includeGlobs": [], "excludeGlobs": [] } ``` # Actor output Schema ## `pages` (type: `string`): One item per converted page: URL, title, Markdown of the main content, headings, links and metadata. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ "https://docs.apify.com/academy" ], "maxPages": 20 }; // Run the Actor and wait for it to finish const run = await client.actor("aurelix/website-to-markdown").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": ["https://docs.apify.com/academy"], "maxPages": 20, } # Run the Actor and wait for it to finish run = client.actor("aurelix/website-to-markdown").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ "https://docs.apify.com/academy" ], "maxPages": 20 }' | apify call aurelix/website-to-markdown --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,aurelix/website-to-markdown" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/2kV4J94aGKwhX9obW/builds/5yZDMwV7KbDNAzORi/openapi.json