# Website to Markdown Crawler: Firecrawl Alternative for RAG (`nebulacolors/website-content-markdown`) Actor

Crawls a website from the start URLs you give and returns each page as clean Markdown: title, description, language and main text, with menus, footers, cookie banners and scripts removed. Obeys robots.txt. $0.002 per page. Set Max pages to 1 to fetch a single page.

- **URL**: https://apify.com/nebulacolors/website-content-markdown.md
- **Developed by:** [Nebula Colors LLC](https://apify.com/nebulacolors) (community)
- **Categories:** AI, Agents, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$2.00 / 1,000 page saveds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website to Markdown Crawler: Firecrawl Alternative for RAG

Turn a website into clean Markdown for an LLM or RAG index, the job Firecrawl and Jina Reader do, but pay only per page: $2 per 1,000 pages, failed pages free, no plan and no API key.

### Website to Markdown use cases

- Load a documentation site, help center or blog into a RAG index or a custom GPT.
- Feed your help center to a support chatbot.
- Give an AI agent the text of one page or a whole section of a site.
- Keep a Markdown copy of your own site's content.

### What you get for each page

Give it a website and get back every page as clean Markdown, ready for an LLM, a RAG index or an AI agent. Each page comes with its title, description and language, and the main text only: menus, footers, cookie banners, forms, scripts and images are taken out, and links are kept as full addresses. It obeys robots.txt and paces its requests. $0.002 per page, and failed pages are free. It works for people and for AI agents through Apify's MCP server.

### Input

| Field | What to put |
|---|---|
| Start URLs | Where to start, one per line, like `https://docs.example.com/guide/`. Required. |
| Max pages | Stop after this many pages. Default 50. Set 1 to fetch just the start URL. |
| Max link depth | How many links away from a start URL to go. Default 5. 0 saves only the start URLs. |
| Crawl the whole site | Off by default: the crawl stays under the start URL's folder (start at `/docs/` and it stays in `/docs/`). On: any page on the same site. |
| Skip URLs matching | Pages to leave out, like `https://example.com/blog/tag/**`. `*` matches within one part of the path, `**` across parts. |
| Proxy | Off by default. Turn on Apify Proxy for a site that blocks cloud servers. |

Example:

```json
{
  "startUrls": ["https://crawlee.dev/js/docs/quick-start"],
  "maxPages": 20,
  "maxDepth": 3
}
```

### Read Google results

Pass the result URLs from Google Search Scraper as Start URLs with Max link depth 0. This crawler does not run JavaScript, so pages that need it may come back thin.

### Output

One row per page. This one came from crawling crawlee.dev on October 15, 2026 (Markdown cut short here):

```json
{
  "type": "page",
  "url": "https://crawlee.dev/js/docs/quick-start",
  "title": "Quick Start | Crawlee for JavaScript · Build reliable crawlers. Fast.",
  "description": "With this short tutorial you can start scraping with Crawlee in a minute or two. To learn more, read the Introduction.",
  "language": "en",
  "canonicalUrl": "https://crawlee.dev/js/docs/quick-start",
  "markdown": "Version: 3.18\n\n# Quick Start\n\nWith this short tutorial you can start scraping with Crawlee in a minute or two. To learn in-depth how Crawlee works, read the [Introduction](https://crawlee.dev/js/docs/introduction), which is a comprehensive step-by-step guide for creating your first scraper.\n\n## Choose your crawler\n\n...",
  "wordCount": 897,
  "depth": 0,
  "statusCode": 200,
  "crawledAt": "2026-10-15T01:48:51.191Z"
}
```

- Tables stay tables, code stays in code blocks, and lists stay lists.
- A page that redirected has `requestedUrl`, the address that was asked for.
- A page that failed (a 404, a server error, or a file that isn't a web page, like JSON) has `error` and no Markdown, and isn't charged.

The run's `SUMMARY` record says how many pages were saved and failed, how many robots.txt kept out, and why the crawl stopped (`done`, `maxPages` or `chargeLimit`).

### Price

- $0.002 per page saved, so $2 per 1,000 pages.
- Failed pages, pages robots.txt disallows and files that aren't web pages are free.
- You can set a maximum charge for a run. The crawl stops when it's reached and keeps what it already saved.

### Good to know

- **Polite by design.** It reads each site's robots.txt and skips what it disallows. It fetches one page at a time with a one-second pause between pages, so expect about 2.5 seconds a page: 20 pages of crawlee.dev took 53 seconds on Apify, start-up included.
- **It reads the HTML the site sends.** Most docs sites, blogs and help centers work. A site that draws its text with JavaScript after loading gives empty Markdown and a `wordCount` of 0.
- **Only web pages are saved.** Links to PDFs, images, archives and other files are skipped.
- **Each page is saved once.** Links to the same page with a different #anchor, or through a redirect, don't make extra rows.
- **Redirects are followed,** including pages that only forward with a meta refresh. If a start URL moves to another address (like docs.example.com to example.github.io/docs), the crawl carries on there.
- **It doesn't log in** and only reads public pages.

### Use it from an AI agent

The Actor works through Apify's MCP server like any Store Actor. An agent can find it, read its input and call it. Only `startUrls` is required. To read one page, set `maxPages` to 1, which usually finishes within the MCP call's wait (up to 45 seconds). A bigger crawl takes longer, so the call returns the run's status and dataset ID, and the agent reads the rows with `get-dataset-items`.

### FAQ

**How do I convert a website to Markdown for RAG?** Give it the site's address. It crawls the pages and returns each one as clean Markdown with its title, description and language, ready to chunk and embed.

**Can it fetch just one page?** Yes. Set Max pages to 1. It usually finishes within an AI agent's wait.

**Does it work on sites built with JavaScript?** It reads the HTML the site sends. Docs sites, blogs and help centers work. A site that draws its text with JavaScript after loading gives empty Markdown, and the row shows a word count of 0.

**Does it respect robots.txt?** Yes, and it paces its requests.

### More tools from us

- [AI Brand Mention Checker](https://apify.com/nebulacolors/ai-visibility): is your brand named by ChatGPT and Gemini, and who comes first.
- [Google Ads Transparency Scraper](https://apify.com/nebulacolors/google-ads-transparency): every Google, YouTube and Display ad a competitor runs.
- [Website Screenshot](https://apify.com/nebulacolors/website-screenshot): full page or mobile screenshots of any URL, PNG or JPEG.
- [News & Blog Article Extractor](https://apify.com/nebulacolors/article-extractor): the newest articles from a site or RSS feed, as Markdown.
- [Image Downloader](https://apify.com/nebulacolors/image-downloader): every image on a page, full size, in one ZIP.

### Questions or problems

Open an issue on this Actor's Issues tab. I read them.

# Actor input Schema

## `startUrls` (type: `array`):

Where the crawl starts, one per line, like https://docs.example.com/guide/. It follows links on the same site under the start URL's folder.

## `maxPages` (type: `integer`):

Stop after this many pages are saved. Each saved page is one charged event. Set 1 to fetch just the start URL.

## `maxDepth` (type: `integer`):

How many links away from a start URL to go. 0 saves only the start URLs.

## `wholeSite` (type: `boolean`):

Off: only pages under the start URL's folder (start at /docs/ and it stays in /docs/). On: any page on the same site.

## `excludeUrlGlobs` (type: `array`):

Pages to leave out, as patterns: \* matches within one part of the path, \*\* across parts. Example: https://example.com/blog/tag/\*\*

## `proxyConfiguration` (type: `object`):

Off by default. Turn on Apify Proxy for a site that blocks cloud servers.

## Actor input object example

```json
{
  "startUrls": [
    "https://crawlee.dev/js/docs/quick-start"
  ],
  "maxPages": 20,
  "maxDepth": 5,
  "wholeSite": false,
  "excludeUrlGlobs": [],
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `pages` (type: `string`):

All page rows as JSON: url, title, description, language, markdown, wordCount, depth and statusCode. A failed page has an error and no markdown.

## `summary` (type: `string`):

Pages saved and failed, pages skipped because robots.txt disallows them or they were too deep, and why the crawl stopped.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://crawlee.dev/js/docs/quick-start"
    ],
    "maxPages": 20
};

// Run the Actor and wait for it to finish
const run = await client.actor("nebulacolors/website-content-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://crawlee.dev/js/docs/quick-start"],
    "maxPages": 20,
}

# Run the Actor and wait for it to finish
run = client.actor("nebulacolors/website-content-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://crawlee.dev/js/docs/quick-start"
  ],
  "maxPages": 20
}' |
apify call nebulacolors/website-content-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,nebulacolors/website-content-markdown"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/ITfyPVbtZwbyDPvW1/builds/fEyq6zTchQrtu87I7/openapi.json