# Website Content Crawler → Clean Text & Markdown for AI (`bgfc97/website-content-markdown-crawler`) Actor

🔥 Up to 30% OFF! Crawl any website → clean text + Markdown for AI/RAG, plus 40+ metadata fields: SEO tags, Open Graph, headings outline, reading time, links, images and JSON-LD structured data. No API key, no login. Feed LLMs, RAG & agents. ⚡

- **URL**: https://apify.com/bgfc97/website-content-markdown-crawler.md
- **Developed by:** [Bruno](https://apify.com/bgfc97) (community)
- **Categories:** AI, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.70 / 1,000 page crawleds

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website Content Crawler → Clean Text & Markdown for AI

Turn **any website into clean, LLM-ready content.** Give it a URL, and it crawls the site and returns, for **every page**, the main article content as **clean text and Markdown** — with navigation, ads, cookie banners, headers, footers and other boilerplate stripped out. Perfect for feeding **ChatGPT / Claude / RAG pipelines / AI agents / knowledge bases**.

### What it actually does (nothing invented)

- Crawls a website from your Start URLs, following links to the depth you choose (same-domain by default).
- Extracts the **main readable content** of each page using Mozilla Readability (the same engine behind Firefox Reader View).
- Returns both **clean plain text** and **clean Markdown** (headings, lists, links and code blocks preserved).
- Gives you page metadata: title, description, author (byline), site name, published time, word count.
- No API key, no login. You only pay per page crawled.

### Output (per page)

```json
{
  "url": "https://example.com/blog/post",
  "title": "How it works",
  "description": "A short summary of the page",
  "byline": "Jane Doe",
  "siteName": "Example",
  "publishedTime": "2026-02-01T00:00:00.000Z",
  "wordCount": 812,
  "charCount": 5120,
  "text": "Clean main text of the page...",
  "markdown": "# How it works\n\nClean **Markdown** of the page...",
  "crawlDepth": 1
}
```

### Great for

- 📚 **RAG / knowledge bases** — ingest docs, help centers and blogs as clean Markdown.
- 🤖 **LLM context** — feed pages to ChatGPT/Claude without the HTML noise.
- 🧠 **AI agents** — let an agent read websites as structured text.
- 📝 **Content & research** — archive and analyze site content at scale.

### Why choose this Actor

- ✅ **Clean Markdown out of the box** — not raw HTML; ready to paste into an LLM.
- ✅ **Pay only per page** — no subscription, no minimum, no API key.
- ✅ **Simple** — paste URLs, set depth, run.
- ✅ **Tunable** — depth, max pages, same-domain, min word count, optional raw HTML, optional residential proxy for tougher sites.

### Limits (honest)

- It fetches the server-rendered HTML (fast & cheap). Sites that render their content **only via JavaScript** may return little — enable the residential-proxy option for blocking sites, and for heavy JS SPAs expect reduced content.
- Content extraction is automatic; unusual page layouts may extract more or less than a human would.

### ⭐ Enjoying this Actor?

If it saved you time, please leave an **honest review** — it genuinely helps. Found a bug or want a feature? **Open an Issue** and we'll jump on it fast.

# Actor input Schema

## `startUrls` (type: `array`):

Websites or pages to crawl. The crawler extracts the main content of each page as clean text + Markdown.

## `maxPages` (type: `integer`):

Maximum number of pages to crawl in total.

## `maxCrawlDepth` (type: `integer`):

How many link levels to follow from each Start URL. 0 = only the given URLs.

## `sameDomainOnly` (type: `boolean`):

Only follow links on the same domain as the Start URL.

## `includeMarkdown` (type: `boolean`):

Return a clean Markdown version of each page (great for LLMs/RAG).

## `extractLinks` (type: `boolean`):

Include the internal and external links found on each page.

## `extractImages` (type: `boolean`):

Include the images (src + alt text) found on each page.

## `extractStructuredData` (type: `boolean`):

Include schema.org JSON-LD structured data and detected schema types.

## `minWordCount` (type: `integer`):

Skip pages whose main content has fewer words than this (filters out empty/boilerplate pages).

## `includeHtml` (type: `boolean`):

Also include the raw HTML of each page (truncated).

## `useResidentialProxy` (type: `boolean`):

Use residential IPs for sites that block datacenter traffic. Off = cheaper datacenter proxy.

## `maxConcurrency` (type: `integer`):

How many pages to fetch in parallel.

## Actor input object example

```json
{
  "startUrls": [
    "https://docs.apify.com/academy/web-scraping-for-beginners"
  ],
  "maxPages": 25,
  "maxCrawlDepth": 1,
  "sameDomainOnly": true,
  "includeMarkdown": true,
  "extractLinks": true,
  "extractImages": true,
  "extractStructuredData": true,
  "minWordCount": 0,
  "includeHtml": false,
  "useResidentialProxy": false,
  "maxConcurrency": 10
}
```

# Actor output Schema

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://docs.apify.com/academy/web-scraping-for-beginners"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("bgfc97/website-content-markdown-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": ["https://docs.apify.com/academy/web-scraping-for-beginners"] }

# Run the Actor and wait for it to finish
run = client.actor("bgfc97/website-content-markdown-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://docs.apify.com/academy/web-scraping-for-beginners"
  ]
}' |
apify call bgfc97/website-content-markdown-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,bgfc97/website-content-markdown-crawler"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/xn4Gar5kfEpY2jElF/builds/bhE2see0uLFRI3pmT/openapi.json