# Website Content to Markdown (LLM-ready) (`vivid_astronaut/website-content-to-markdown`) Actor
Turn any website into clean, LLM-ready Markdown for RAG pipelines, AI agents and knowledge bases. Scrape single pages or crawl entire sites. Compliance-first: robots.txt honored.
- **URL**: https://apify.com/vivid\_astronaut/website-content-to-markdown.md
- **Developed by:** [Fabio Suizu](https://apify.com/vivid_astronaut) (community)
- **Categories:** AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet
## Pricing
$2.00 / 1,000 page converted to markdowns
This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event
## What's an Apify Actor?
Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".
## How to integrate an Actor?
If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.
In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md):
```bash
npm install apify-client
```
In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md):
```bash
pip install apify-client
```
In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md):
````bash
# MacOS / Linux
curl -fsSL https://apify.com/install-cli.sh | bash
# Windows
irm https://apify.com/install-cli.ps1 | iex
```bash
In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md).
If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md).
For usage examples, see the [API](#api) section below.
For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).
# README
## Website Content to Markdown (LLM-ready) — by Brainiall
Turn any website into **clean, structured Markdown** that is ready to feed into LLMs, RAG pipelines, AI agents, vector databases and knowledge bases — without writing a single selector.
Give it one or more URLs and get back distraction-free Markdown: headings, paragraphs, lists and links preserved; navigation chrome, cookie banners, scripts and boilerplate stripped out.
### What it does
- **Single-page scrape** — convert exact URLs to Markdown (default mode).
- **Site crawl** — set *Max pages per site* above 1 and each start URL becomes the entry point of a bounded crawl (up to 25 pages, configurable link depth). Every page becomes one dataset item.
- **Metadata included** — page title, language, description and word count with every item.
- **Links on demand** — optionally return the hyperlinks found on each page.
Powered by the **Brainiall Web engine** ([api.brainiall.com](https://app.brainiall.com)) — a production web-intelligence service built for AI workloads.
### Who it's for
- **RAG builders** — ingest documentation sites, blogs and product pages straight into your vector store. Markdown chunks cleanly and embeds better than raw HTML.
- **AI agent developers** — give your agents reliable, token-efficient web content instead of noisy HTML.
- **Data & content teams** — archive or migrate site content as portable Markdown.
- **LLM fine-tuning** — collect clean text corpora from public websites.
### Compliance-first by design
This Actor inherits the ethics posture of the Brainiall Web engine:
- **robots.txt is honored** — pages disallowed for crawlers are skipped, and crawl-delay directives are respected.
- **No bot-detection bypassing** — sites that choose to block automated access stay blocked.
- **SSRF-guarded** — internal/private addresses are never fetched.
If your compliance team asks how the data was collected, you have a clean answer.
### Input
```json
{
"startUrls": [
{ "url": "https://docs.example.com" },
{ "url": "https://blog.example.com/post" }
],
"maxPagesPerUrl": 10,
"maxDepth": 2,
"includeLinks": false
}
````
| Field | Description |
|-------|-------------|
| `startUrls` | Pages (or crawl entry points) to convert. |
| `maxPagesPerUrl` | `1` = scrape only the exact URL. `2-25` = crawl the site from that URL. |
| `maxDepth` | Link hops allowed from the start URL (crawl mode). |
| `includeLinks` | Also return hyperlinks found on the page (single-page mode). |
### Output
One dataset item per page:
```json
{
"url": "https://example.com/",
"requested_url": "https://example.com",
"title": "Example Domain",
"markdown": "# Example Domain\n\nThis domain is for use in documentation examples...",
"word_count": 19,
"metadata": { "description": "", "lang": "en" },
"http_status": 200
}
```
Export as JSON, CSV, Excel or consume via the Apify API — ready for LangChain, LlamaIndex or any custom pipeline.
### Pricing
You pay **per result** (per page successfully converted to Markdown). Pages that fail or contain no extractable content are **not** charged. No subscriptions, no minimums — costs scale exactly with what you extract.
### Tips
- For documentation sites, start at the docs root with `maxPagesPerUrl: 25`, `maxDepth: 3` to capture a whole section in one run.
- Keep `includeLinks` off unless you need them — smaller items mean faster downstream processing.
- Deduplicate by `url` if you crawl overlapping start URLs.
***
Built and maintained by [Brainiall](https://www.brainiall.com) — production AI APIs for speech, documents, vision and the web.
# Actor input Schema
## `startUrls` (type: `array`):
Pages or sites to convert to Markdown. With Max pages per site = 1 each URL is scraped as a single page; with a higher value each URL is used as the entry point of a site crawl.
## `maxPagesPerUrl` (type: `integer`):
1 = scrape only the exact URL. 2-25 = crawl the site starting from the URL, following internal links, up to this many pages. Each page becomes one dataset item.
## `maxDepth` (type: `integer`):
How many link hops away from the start URL the crawler may go (crawl mode only).
## `includeLinks` (type: `boolean`):
Also return the hyperlinks found on each page (single-page scrape mode only).
## Actor input object example
```json
{
"startUrls": [
{
"url": "https://example.com"
}
],
"maxPagesPerUrl": 1,
"maxDepth": 2,
"includeLinks": false
}
```
# API
You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.
## JavaScript example
```javascript
import { ApifyClient } from 'apify-client';
// Initialize the ApifyClient with your Apify API token
// Replace the '' with your token
const client = new ApifyClient({
token: '',
});
// Prepare Actor input
const input = {
"startUrls": [
{
"url": "https://example.com"
}
]
};
// Run the Actor and wait for it to finish
const run = await client.actor("vivid_astronaut/website-content-to-markdown").call(input);
// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
console.dir(item);
});
// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs
```
## Python example
```python
from apify_client import ApifyClient
# Initialize the ApifyClient with your Apify API token
# Replace '' with your token.
client = ApifyClient("")
# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://example.com" }] }
# Run the Actor and wait for it to finish
run = client.actor("vivid_astronaut/website-content-to-markdown").call(run_input=run_input)
# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
print(item)
# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start
```
## CLI example
```bash
echo '{
"startUrls": [
{
"url": "https://example.com"
}
]
}' |
apify call vivid_astronaut/website-content-to-markdown --silent --output-dataset
```
## MCP server setup
```json
{
"mcpServers": {
"apify": {
"command": "npx",
"args": [
"mcp-remote",
"https://mcp.apify.com/?tools=vivid_astronaut/website-content-to-markdown",
"--header",
"Authorization: Bearer "
]
}
}
}
```
## OpenAPI specification
```json
{
"openapi": "3.0.1",
"info": {
"title": "Website Content to Markdown (LLM-ready)",
"description": "Turn any website into clean, LLM-ready Markdown for RAG pipelines, AI agents and knowledge bases. Scrape single pages or crawl entire sites. Compliance-first: robots.txt honored.",
"version": "1.0",
"x-build-id": "AC6TEkb1TEgEI0Etv"
},
"servers": [
{
"url": "https://api.apify.com/v2"
}
],
"paths": {
"/acts/vivid_astronaut~website-content-to-markdown/run-sync-get-dataset-items": {
"post": {
"operationId": "run-sync-get-dataset-items-vivid_astronaut-website-content-to-markdown",
"x-openai-isConsequential": false,
"summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
"tags": [
"Run Actor"
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/inputSchema"
}
}
}
},
"parameters": [
{
"name": "token",
"in": "query",
"required": true,
"schema": {
"type": "string"
},
"description": "Enter your Apify token here"
}
],
"responses": {
"200": {
"description": "OK"
}
}
}
},
"/acts/vivid_astronaut~website-content-to-markdown/runs": {
"post": {
"operationId": "runs-sync-vivid_astronaut-website-content-to-markdown",
"x-openai-isConsequential": false,
"summary": "Executes an Actor and returns information about the initiated run in response.",
"tags": [
"Run Actor"
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/inputSchema"
}
}
}
},
"parameters": [
{
"name": "token",
"in": "query",
"required": true,
"schema": {
"type": "string"
},
"description": "Enter your Apify token here"
}
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/runsResponseSchema"
}
}
}
}
}
}
},
"/acts/vivid_astronaut~website-content-to-markdown/run-sync": {
"post": {
"operationId": "run-sync-vivid_astronaut-website-content-to-markdown",
"x-openai-isConsequential": false,
"summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
"tags": [
"Run Actor"
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/inputSchema"
}
}
}
},
"parameters": [
{
"name": "token",
"in": "query",
"required": true,
"schema": {
"type": "string"
},
"description": "Enter your Apify token here"
}
],
"responses": {
"200": {
"description": "OK"
}
}
}
}
},
"components": {
"schemas": {
"inputSchema": {
"type": "object",
"required": [
"startUrls"
],
"properties": {
"startUrls": {
"title": "Start URLs",
"type": "array",
"description": "Pages or sites to convert to Markdown. With Max pages per site = 1 each URL is scraped as a single page; with a higher value each URL is used as the entry point of a site crawl.",
"items": {
"type": "object",
"required": [
"url"
],
"properties": {
"url": {
"type": "string",
"title": "URL of a web page",
"format": "uri"
}
}
}
},
"maxPagesPerUrl": {
"title": "Max pages per site",
"minimum": 1,
"maximum": 25,
"type": "integer",
"description": "1 = scrape only the exact URL. 2-25 = crawl the site starting from the URL, following internal links, up to this many pages. Each page becomes one dataset item.",
"default": 1
},
"maxDepth": {
"title": "Max crawl depth",
"minimum": 0,
"maximum": 4,
"type": "integer",
"description": "How many link hops away from the start URL the crawler may go (crawl mode only).",
"default": 2
},
"includeLinks": {
"title": "Include page links",
"type": "boolean",
"description": "Also return the hyperlinks found on each page (single-page scrape mode only).",
"default": false
}
}
},
"runsResponseSchema": {
"type": "object",
"properties": {
"data": {
"type": "object",
"properties": {
"id": {
"type": "string"
},
"actId": {
"type": "string"
},
"userId": {
"type": "string"
},
"startedAt": {
"type": "string",
"format": "date-time",
"example": "2025-01-08T00:00:00.000Z"
},
"finishedAt": {
"type": "string",
"format": "date-time",
"example": "2025-01-08T00:00:00.000Z"
},
"status": {
"type": "string",
"example": "READY"
},
"meta": {
"type": "object",
"properties": {
"origin": {
"type": "string",
"example": "API"
},
"userAgent": {
"type": "string"
}
}
},
"stats": {
"type": "object",
"properties": {
"inputBodyLen": {
"type": "integer",
"example": 2000
},
"rebootCount": {
"type": "integer",
"example": 0
},
"restartCount": {
"type": "integer",
"example": 0
},
"resurrectCount": {
"type": "integer",
"example": 0
},
"computeUnits": {
"type": "integer",
"example": 0
}
}
},
"options": {
"type": "object",
"properties": {
"build": {
"type": "string",
"example": "latest"
},
"timeoutSecs": {
"type": "integer",
"example": 300
},
"memoryMbytes": {
"type": "integer",
"example": 1024
},
"diskMbytes": {
"type": "integer",
"example": 2048
}
}
},
"buildId": {
"type": "string"
},
"defaultKeyValueStoreId": {
"type": "string"
},
"defaultDatasetId": {
"type": "string"
},
"defaultRequestQueueId": {
"type": "string"
},
"buildNumber": {
"type": "string",
"example": "1.0.0"
},
"containerUrl": {
"type": "string"
},
"usage": {
"type": "object",
"properties": {
"ACTOR_COMPUTE_UNITS": {
"type": "integer",
"example": 0
},
"DATASET_READS": {
"type": "integer",
"example": 0
},
"DATASET_WRITES": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_READS": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_WRITES": {
"type": "integer",
"example": 1
},
"KEY_VALUE_STORE_LISTS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_READS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_WRITES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_INTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_EXTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_SERPS": {
"type": "integer",
"example": 0
}
}
},
"usageTotalUsd": {
"type": "number",
"example": 0.00005
},
"usageUsd": {
"type": "object",
"properties": {
"ACTOR_COMPUTE_UNITS": {
"type": "integer",
"example": 0
},
"DATASET_READS": {
"type": "integer",
"example": 0
},
"DATASET_WRITES": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_READS": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_WRITES": {
"type": "number",
"example": 0.00005
},
"KEY_VALUE_STORE_LISTS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_READS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_WRITES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_INTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_EXTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_SERPS": {
"type": "integer",
"example": 0
}
}
}
}
}
}
}
}
}
}
```