# Web Scraper & Crawler for RAG (Markdown) (`byteblisk/sitereader`) Actor

Turn any website into clean Markdown or structured JSON. Read pages, explore whole sites, discover every URL, pull AI fields, search the web and read PDFs. Also runs as a fast HTTP API in standby mode.

- **URL**: https://apify.com/byteblisk/sitereader.md
- **Developed by:** [byteblisk](https://apify.com/byteblisk) (community)
- **Categories:** AI, Developer tools
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### Give your AI the web as clean, ready-to-use text

Turn any page, or an entire website, into clean Markdown or structured JSON that your LLM can actually use: no
menus, no cookie banners, no boilerplate. **$1.50 per 1,000 pages**, and pages that come back empty are free.

#### What you get

- **Clean content for RAG and chatbots.** Only the main text, with links, images and page details when you want them.
- **Whole sites in one run.** Crawl with depth and path rules, or list every URL of a domain in seconds.
- **Data, not just text.** Describe the fields you need in plain words ("every job with title, location and link")
  and get JSON back, on your own Anthropic key at no extra charge from us.
- **Pages that need JavaScript.** A real browser is used only where a page needs one, so the rest stays fast and cheap.
- **Know when a page changes.** Watches re-check pages on a schedule and return only what changed.
- **Plug into any agent.** HTTP API and MCP server, ready for Claude, Cursor or your own code.

#### Built for

- **AI engineers** feeding RAG pipelines, chatbots and vector databases
- **Growth and sales teams** turning websites into lead and price lists
- **Researchers and analysts** collecting and monitoring web content
- **Agent builders** who need the live web as a tool

### What data can SiteReader extract?

| Field | Description |
|---|---|
| `url` | The page that was read |
| `markdown` | Main content as Markdown |
| `text` | The same content as plain text, ready for search indexes and vector databases |
| `html` / `raw_html` | Cleaned or original HTML |
| `links` / `images` | Every link and image on the page, as absolute URLs |
| `screenshot` | Full-page screenshot (PNG, base64) |
| `summary` | A short AI summary of the page |
| `fields` | The fields you asked for, extracted by AI |
| `structured` | What the page publishes for machines: schema.org items (jobs, products, articles), Open Graph, tables, headings - no AI cost |
| `info` | Title, description, language, status code, engine used, archive date when an archive copy was used |

### How to read a website as Markdown

1. Open the **Input** tab and choose **What to do**: Read pages, Explore a site, Discover a site's URLs, AI fields or Find.
2. Add one or more **URLs** (or a search query).
3. Pick the **Output** you need. For AI fields, choose *AI fields* and describe what to extract.
4. Click **Start**. Results appear in the **Output** tab; failed URLs are listed in the `ERRORS` record.

### How much does it cost?

SiteReader uses **pay per event**: you pay only for what is done, and your Apify free monthly credit covers trying it.

| Event | When it is charged |
|---|---|
| Page (fetch) | each page read with a fast HTTP fetch, a PDF/DOCX, or an archive copy |
| Page (browser) | each page that needs a real browser (JavaScript, steps, screenshots) |
| AI page | each page with an AI summary or AI fields, using Claude Haiku through SiteReader |
| AI page (premium) | the same with Claude Opus |
| Proxy page | each page read through a country proxy |
| Site map | each site whose URLs are discovered |
| Search query | each web search |

AI pages are not charged when you use your own Anthropic key. You can set a maximum cost for every run.

#### Cost examples

| Job | Cost |
|---|---|
| 1,000 simple pages as Markdown | $1.50 |
| 1,000 JavaScript-heavy pages (real browser) | $4.00 |
| 1,000 pages with AI fields, your own Anthropic key | $1.50 (AI billed to your key) |
| A site map of one domain | $0.01 |
| A page that comes back empty | free |

### Input

See the **Input** tab for every option. A few notes:

- **Browser steps** use a JSON list, for example
  `[{"do": "click", "selector": "#load-more"}, {"do": "wait", "ms": 1000}]`.
- **AI fields**: end a URL with `/*` to let SiteReader pick the most relevant pages of that site.
- **robots.txt** is ignored by default; choose *Respect* to follow each site's rules.
- **Discover** also reads a site's `/llms.txt` (a list of pages written for AI) when it has one.
- **Headers and cookies** are sent only to the URLs of your run; pages read with them are never cached.

### Output

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel.

```json
{
  "url": "https://books.toscrape.com/",
  "markdown": "# All products\n\n* [A Light in the Attic](https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html) £51.77 ...",
  "fields": {"books": [{"title": "A Light in the Attic", "price": "£51.77"}]},
  "info": {"title": "All products | Books to Scrape - Sandbox", "status_code": 200, "engine": "fetch"}
}
```

### Use it as an API

Run the Actor in **standby** mode and call it over HTTP:

```bash
curl -X POST "https://<your-standby-url>/v1/read" \
  -H "Authorization: Bearer $APIFY_TOKEN" -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "output": ["markdown", "links"]}'
```

Every answer has the same shape: `{"ok": true, ..., "cost": {"units": 1}}`, or `{"ok": false, "error": "..."}`.
Long tasks (`/v1/explore`, `/v1/read/batch`, `/v1/fields`, `/v1/lookup`) return a job; follow it at `/v1/jobs/{id}`,
or add `deliver` to have the results sent to your webhook or storage when done. Also: `/v1/watches`, `/v1/recipes`,
`/v1/feed` and the one-line `GET /v1/read?url=...`. The full API description is on the **Server** tab.

### Connect your AI agent (MCP)

SiteReader is also an **MCP server**, so Claude, Cursor, VS Code and any MCP client can use it as a tool. Add this to
your client's MCP settings (your Apify token goes in the header):

```json
{
  "mcpServers": {
    "sitereader": {
      "url": "https://<your-standby-url>/mcp",
      "headers": {"Authorization": "Bearer <APIFY_TOKEN>"}
    }
  }
}
```

Tools: `read`, `read_batch` (up to 10 pages), `discover`, `explore`, `fields`, `find`, `find_batch` (up to 5
searches) and `usage`. Pages that contain text trying to give orders to an AI are flagged with `content_warning`, so
your agent treats them as data.

### Tips

- Keep **Engine** on *Auto*: the browser is used only where it is needed, which keeps runs cheap.
- Use **Main content** and **Remove these parts** to keep the Markdown short for AI.
- For job boards and widgets that load late, set **Wait after load** to 1000-2000 ms.

### FAQ, disclaimers and support

**Is it legal to scrape websites?** Our Actors are ethical and do not extract any private user data, such as email
addresses, gender, or location. They only extract what the user has chosen to share publicly. We therefore believe
that our Actors, when used for ethical purposes by Apify users, are safe. However, you should be aware that your
results could contain personal data. Personal data is protected by the GDPR in the European Union and by other
regulations around the world. You should not scrape personal data unless you have a legitimate reason to do so. If
you're unsure whether your reason is legitimate, consult your lawyers.

**A site returned an error or blocked the request.** SiteReader stops reading that site for the run. Choose
**Use the archive copy** to get its latest archived version instead.

Found a bug or need a feature? Open an issue in the **Issues** tab. For programmatic access, see the **API** tab.

### byteblisk actors

Built and supported by **byteblisk**. Same record conventions, API style and MCP design across all three, so an AI
agent can use them together:

- 📄 **[SiteReader](https://apify.com/byteblisk/sitereader)**: any website to clean Markdown or JSON for AI, with AI fields, site maps and watches.
- ⭐ **[App Reviews & Insights](https://apify.com/byteblisk/app-reviews-insights)**: Google Play and App Store reviews in one format, with AI insights and alerts.
- 🔎 **[Google Play & Search Toolkit](https://apify.com/byteblisk/google-play-search-toolkit)**: Play apps, reviews and search, top-100 search rank, and exports to
  your warehouse, in one actor.

Something missing or broken? Open an issue in the **Issues** tab: we answer every one.

# Actor input Schema

## `mode` (type: `string`):

Read: each URL. Explore: follow links within each site. Discover: list a site's URLs. Fields: AI pulls the fields you ask for across the pages. Find: web search, optionally reading each result. Feed: a Greenhouse, Lever or Ashby job board, a Shopify store, a WordPress site or an RSS feed, as clean items. Watch: compare each page with the previous run and output what is new, changed or removed (run it on an Apify schedule). Find all: a table of every item matching your query across search results, each row with its sources.

## `urls` (type: `array`):

Pages or sites. For AI fields, end a URL with /\* to let SiteReader pick the most relevant pages of that site.

## `query` (type: `string`):

Find mode only.

## `intent` (type: `string`):

What you are really after; results are scored 0-1 and sorted by it.

## `columns` (type: `object`):

{"company": "company name", "role": "job title", "city": "location"} - the first column identifies a row.

## `pages` (type: `integer`):

Lookup: how many search result pages to read for the table.

## `feed_kind` (type: `string`):

Job boards and Google Play app or developer URLs are recognised by themselves; choose the type for Shopify, WordPress, RSS or your own app's reviews.

## `google_access_token` (type: `string`):

Only for reading your own app's reviews through Google's official API (last 7 days). Used for the call, never stored.

## `meta_access_token` (type: `string`):

Your Meta token: for Instagram Business Discovery (with your Instagram business account) or the Ad Library. Never stored.

## `youtube_api_key` (type: `string`):

Your own key from Google Cloud. Never stored.

## `feed_options` (type: `object`):

Instagram: {"ig_user_id": "<your business account id>"}. Ad Library: {"countries": \["US"], "page_ids": \[]}. Any: {"limit": 50}. For these feeds put usernames, search terms, @handles or video URLs in URLs (or Search query).

## `watch_name` (type: `string`):

Keeps this watch's memory apart from your other watches. Defaults to one built from the URLs.

## `only_changes` (type: `boolean`):

Watch: only new, changed and removed pages.

## `include_content` (type: `boolean`):

Watch: include the page Markdown.

## `output` (type: `array`):

What to return for each page.

## `content` (type: `string`):

Main: drop menus, footers and other boilerplate. Full: the whole page.

## `keep` (type: `array`):

For example article, .job-list, #main

## `drop` (type: `array`):

For example nav, footer, .cookie-banner

## `clean` (type: `boolean`):

Drops cookie / consent banners and overlays before the page is turned into text.

## `expand` (type: `boolean`):

Opens accordions, 'read more' toggles and <details> before reading. Uses the browser.

## `engine` (type: `string`):

Auto: a fast fetch, with a browser only when the page needs JavaScript. Browser: always render (slower, priced as a browser page).

## `wait_ms` (type: `integer`):

Extra time for JavaScript widgets to appear. Uses the browser.

## `steps` (type: `array`):

Steps before the page is read: wait, click, type, key, scroll, script. Example: \[{"do": "click", "selector": "#load-more"}, {"do": "wait", "ms": 1000}]

## `headers` (type: `object`):

Your own headers, e.g. for your own site: {"Authorization": "Bearer ..."}. Sent only to the URLs in this run.

## `cookies` (type: `array`):

Your own cookies for pages you are allowed to read: \[{"name": "session", "value": "...", "domain": "example.com"}]. Pages read with cookies are never cached or shared.

## `limit` (type: `integer`):

Explore: pages per site. Discover: URLs per site. Find: results.

## `depth` (type: `integer`):

How many path levels below the site's root to follow.

## `only_paths` (type: `array`):

For example ^/careers

## `skip_paths` (type: `array`):

For example ^/blog

## `ignore_query` (type: `boolean`):

Explore: URLs that differ only after the ? (sorting, filters, sessions) count as one page. Tracking parameters like utm\_ are always ignored.

## `time_budget_secs` (type: `integer`):

Explore: stop after this many seconds and keep the pages read so far. Pages are saved as they are read.

## `from_sitemap` (type: `boolean`):

Start exploring from the sitemap too.

## `filter` (type: `string`):

Words to look for in the URLs, best matches first.

## `read_results` (type: `boolean`):

Find: read each result page.

## `ai_instructions` (type: `string`):

For example: every open job with its title, location and link.

## `ai_schema` (type: `object`):

The fields you want back. Without it, the AI chooses a structure that fits the instructions.

## `ai_model` (type: `string`):

Haiku: fast and cheap, right for most pages. Opus: the strongest model, priced higher.

## `anthropic_api_key` (type: `string`):

Use your own Claude key and pay Anthropic directly; AI pages are then not charged here.

## `region` (type: `string`):

See the site as a visitor in this country, for example IL, US, DE. Uses Apify Proxy; priced per page.

## `language` (type: `string`):

Ask sites for this language, e.g. he, de, pt-BR. Works without a country; with a region, the region's languages are used.

## `proxy_type` (type: `string`):

Region: proxy type.

## `robots` (type: `string`):

Ignore (default) or respect each site's robots.txt rules.

## `fallback` (type: `string`):

Archive: return the site's latest Internet Archive copy, marked with its date.

## `delay_secs` (type: `number`):

Seconds between requests to one site.

## Actor input object example

```json
{
  "mode": "read",
  "urls": [
    {
      "url": "https://books.toscrape.com/"
    }
  ],
  "pages": 8,
  "feed_kind": "",
  "only_changes": true,
  "include_content": false,
  "output": [
    "markdown"
  ],
  "content": "main",
  "clean": true,
  "expand": false,
  "engine": "auto",
  "wait_ms": 0,
  "limit": 50,
  "depth": 3,
  "ignore_query": false,
  "from_sitemap": true,
  "read_results": false,
  "ai_model": "haiku",
  "proxy_type": "none",
  "robots": "ignore",
  "fallback": "none",
  "delay_secs": 2
}
```

# Actor output Schema

## `pages` (type: `string`):

No description

## `errors` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        {
            "url": "https://books.toscrape.com/"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("byteblisk/sitereader").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": [{ "url": "https://books.toscrape.com/" }] }

# Run the Actor and wait for it to finish
run = client.actor("byteblisk/sitereader").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    {
      "url": "https://books.toscrape.com/"
    }
  ]
}' |
apify call byteblisk/sitereader --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,byteblisk/sitereader"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/MOsDxz5FaAUu5D4Cl/builds/3tcMLusHmLVbs8hnx/openapi.json