# Website Intelligence Crawler (`dami_studio/website-intelligence-crawler`) Actor

Crawl a public website and get each page as clean text, Markdown, metadata, same-site links and embedding-ready chunks. Robots-aware, no API key, no LLM call, no browser. Built for the boring part of a RAG pipeline. $0.60 per 1,000 pages plus a $0.001 start fee.

- **URL**: https://apify.com/dami\_studio/website-intelligence-crawler.md
- **Developed by:** [Dami's Studio](https://apify.com/dami_studio) (community)
- **Categories:** Developer tools, SEO tools, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.60 / 1,000 page crawleds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Website Intelligence Crawler — site to clean text, Markdown and chunks

Give it one or more URLs. It crawls the same site, honours `robots.txt`, and returns each page as one
row: clean text, Markdown, page metadata, same-site links and character-bounded chunks ready to hand
to an embedding model.

This is the boring part of a RAG pipeline: turning a public website into rows you can embed. It's
also fine if you just want a site's readable text and would rather not write a parser. No API key,
no LLM call anywhere, no browser.

### What a row looks like

```json
{
  "ok": true,
  "requestedUrl": "https://example.com/",
  "url": "https://example.com/",
  "seedUrl": "https://example.com/",
  "siteKey": "example.com",
  "depth": 0,
  "status": 200,
  "title": "Example Domain",
  "description": null,
  "language": "en",
  "canonicalUrl": "https://example.com/",
  "contentType": "text/html",
  "text": "Example DomainThis domain is for use in documentation examples...",
  "markdown": "# Example Domain\n\nThis domain is for use in documentation examples...",
  "links": [],
  "wordCount": 17,
  "chunks": [
    { "chunkIndex": 0, "text": "Example Domain...", "charCount": 125, "estimatedTokens": 32, "relevanceScore": null }
  ],
  "query": null,
  "outputFormat": "both",
  "fetchedAt": "2026-08-21T16:20:24.453Z"
}
```

`text` is the readable body with script, style, nav, header, footer and form stripped out. `markdown`
keeps headings, paragraphs, list items, blockquotes and code blocks. `links` is deduplicated
same-site links with their anchor text, capped at 250 per page. `relevanceScore` is `null` unless you
pass a `query`.

#### How full the columns actually are

Measured on a 60-row run on 2026-08-10. Three seeds (`stripe.com`, `docs.github.com`, `python.org`)
at depth 2, 20 pages each. All 60 fetched, none blocked, no diagnostic rows, 14.3 seconds end to end.

| Field | Filled |
|---|---|
| `url`, `status`, `title`, `canonicalUrl` | 100% |
| `text`, `markdown`, `chunks`, `wordCount`, `fetchedAt` | 100% |
| `language` | 98.3% |
| `links` | 98.3% |
| `description` | 96.7% |

`description` and `language` only get emitted when the page actually publishes a `meta description` /
`og:description` and an `html lang`. Nothing is invented, so those columns are `null` on pages that
don't. `links` is empty on a page with no same-site links. Median page: 335 words.

### Why it gets pages a plain fetch gets 403 on

Requests carry a real browser TLS and HTTP/2 fingerprint (`impit`), sweeping a Chrome profile and
then a Firefox one. Plain Node `fetch` gets rejected at the TLS handshake by Cloudflare, Akamai and
friends before a single byte of the page is read. That one change is what took this from "works on
example.com" to fetching Stripe and GitHub Docs without a proxy or a browser.

There's still no headless browser, so a page that renders nothing server-side comes back thin. See
the limits.

### Input

| Field | What it does |
|---|---|
| `startUrls` / `urls` | Public HTTP(S) URLs. Bare domains accepted, `www.` normalised, fragments dropped, duplicates removed. Up to 200 seeds. |
| `maxPages` | Page records per seed. Default 10, max 100. |
| `maxDepth` | Same-site link depth. Default 1, max 5. `0` fetches only the seed. |
| `maxConcurrency` | Pages fetched at once. Default 3, max 10. |
| `useRobotsTxt` | On by default. Honours `Disallow`, and `Crawl-delay` up to a 10-second cap. |
| `query` | Whitespace-separated terms. Chunks get a `relevanceScore` (term hits plus a title-match boost) and are sorted best-first. Plain counting, no embedding service, no key. |
| `chunkSizeChars` / `chunkOverlapChars` | Default 1200 and 120 characters. |
| `outputFormat` | `json` drops `markdown`, `markdown` drops `text`, `both` keeps them. |
| `requestTimeoutSecs` / `maxResponseSizeKb` | Default 15 seconds and 2048 KB. |
| `fallbackToProxy` + `proxyConfiguration` | Off by default. When on, a page that comes back blocked or errored on direct traffic is retried once through the proxy you configured. |

Leave the input empty and you get exactly one labelled `_sample: true` row and zero charges.

### Billing

$0.60 per 1,000 pages, event name `page`, plus a $0.001 start fee. One charge per successful HTML
page row. On the 60-page run above the platform billed exactly `{apify-actor-start: 1, page: 60}` —
one event per real row, nothing else.

Never charged: the empty-input sample row, `ROBOTS_DISALLOWED` rows, `HTTP_ERROR`, `NETWORK`,
`TIMEOUT`, `SSRF_BLOCKED`, `RESPONSE_TOO_LARGE` and `UNSUPPORTED_CONTENT` diagnostics. A run that
fetches nothing charges nothing but the start fee.

Worth knowing where that price comes from: a crawl of a site that enforces a `Crawl-delay` spends
about $0.0002 a page in platform compute just waiting, and the price is set to stay above that on
the slow sites rather than only on the fast ones.

### What it doesn't do

- No headless browser. A page whose content only appears after client-side JavaScript returns
  whatever the server sent, which is often a thin shell. Use a browser-based crawler for those.
- No cross-site crawling. Links are followed only inside the seed's registrable host.
- No login, no cookies, no paywalled content.
- No AI summarisation, embeddings or entity extraction. `relevanceScore` is term counting.
- No PDF, DOCX or image parsing. A non-HTML response becomes an `UNSUPPORTED_CONTENT` diagnostic row.
- Sites behind an interactive challenge (hCaptcha, "press and hold") aren't solved.

### Questions

**How do I turn a website into Markdown for a RAG pipeline?** Put the home page in `startUrls`, set
`maxDepth` to 2–3 and `maxPages` to how many pages you want, then read the `markdown` and `chunks`
fields off each row.

**Does it need an OpenAI or Anthropic key?** No. There's no LLM call in this actor at all. `chunks`
and `relevanceScore` are deterministic.

**Does it respect robots.txt?** Yes, by default. Disallowed paths come back as uncharged
`ROBOTS_DISALLOWED` rows rather than being silently skipped. `Crawl-delay` is honoured up to a
10-second cap — Hacker News asks for 30 and the crawler waits 10. Googlebot ignores `Crawl-delay`
entirely; this is the middle ground. Set `useRobotsTxt: false` on sites you own.

**How many pages will it return?** `maxPages` per seed, capped at 100. Ten seeds at `maxPages: 100`
is 1,000 rows.

**Can I crawl a site that blocks bots?** Often, without a proxy — see the fingerprint note. If a
specific site still blocks you, turn on `fallbackToProxy` and attach an Apify Proxy configuration;
blocked pages get retried once through it.

**What happens if I give it a bad URL?** One uncharged diagnostic row with `errorCode: BAD_INPUT` and
the reason. Private, local and link-local addresses are refused with `SSRF_BLOCKED`.

# Actor input Schema

## `startUrls` (type: `array`):

Public HTTP(S) URLs to crawl. Values are normalized and deduplicated.

## `urls` (type: `array`):

Additional public HTTP(S) URLs. Same-site links are followed from each seed.

## `maxPages` (type: `integer`):

Maximum successful or diagnostic page records per seed URL.

## `maxDepth` (type: `integer`):

Maximum same-site link depth from each seed. Zero fetches only the seed page.

## `maxConcurrency` (type: `integer`):

Maximum requests processed concurrently for each seed site.

## `useRobotsTxt` (type: `boolean`):

Honor public robots.txt disallow and crawl-delay rules.

## `query` (type: `string`):

Whitespace-separated terms used for deterministic ranking. No AI or API key is used.

## `chunkSizeChars` (type: `integer`):

Target maximum characters in each deterministic content chunk.

## `chunkOverlapChars` (type: `integer`):

Characters repeated between adjacent chunks to preserve context.

## `outputFormat` (type: `string`):

Keep clean text, Markdown, or both representations in each page record.

## `requestTimeoutSecs` (type: `integer`):

Maximum time allowed for each HTTP request.

## `maxResponseSizeKb` (type: `integer`):

Maximum response body size accepted from one page in kilobytes.

## `fallbackToProxy` (type: `boolean`):

Retry blocked direct requests through the configured Apify Proxy.

## `proxyConfiguration` (type: `object`):

Optional proxy configuration. Direct traffic is always tried first.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://example.com/"
    }
  ],
  "maxPages": 10,
  "maxDepth": 1,
  "maxConcurrency": 3,
  "useRobotsTxt": true,
  "chunkSizeChars": 1200,
  "chunkOverlapChars": 120,
  "outputFormat": "both",
  "requestTimeoutSecs": 15,
  "maxResponseSizeKb": 2048,
  "fallbackToProxy": false
}
```

# Actor output Schema

## `results` (type: `string`):

Page records in the default dataset.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://example.com/"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("dami_studio/website-intelligence-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://example.com/" }] }

# Run the Actor and wait for it to finish
run = client.actor("dami_studio/website-intelligence-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://example.com/"
    }
  ]
}' |
apify call dami_studio/website-intelligence-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,dami_studio/website-intelligence-crawler"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/JYu9aJftUKpamOBdy/builds/hcoiN66fgkPkAeaiq/openapi.json