# Website Crawler – Whole-Site Text, Emails & Contacts (`locomint/website-crawler-content-contacts`) Actor

Give it a website, get every page as clean Markdown plus the contacts found on each: emails, social profiles, WhatsApp, contact page, tech stack. Follows internal links and the sitemap, respects robots.txt, capped by pages and depth. Fast HTTP crawl, no browser, flat price per page.

- **URL**: https://apify.com/locomint/website-crawler-content-contacts.md
- **Developed by:** [Umaid Ali](https://apify.com/locomint) (community)
- **Categories:**
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.80 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Website Crawler – Whole-Site Text, Emails & Contacts

Give it a website. Get back **every page as clean Markdown**, and on every page the
**business contacts** it holds: emails, Facebook / Instagram / LinkedIn / X / YouTube /
TikTok, WhatsApp, contact page, tech stack.

- 🕸️ **Follows the site itself** – internal links plus the sitemap, so pages that the
  homepage never links to are found too. Same host only (subdomains optional).
- 🤝 **Polite by default** – honours robots.txt, one site at a time, capped by pages and
  depth, a few parallel fetches. No hammering.
- ⚡ **No browser** – plain HTTP, hundreds of pages a minute, a fraction of the price of
  browser-based crawlers.
- 🧾 **Honest statuses** – `ok`, `thin` (client-side rendered), `parked`, `http_error`,
  `not_html`, `unreachable`, `blocked`, `refused`. Every URL comes back with one.
- 💵 **Flat price per page.** Nothing else.

***

### 🎯 What can you do with it?

- **Feed a RAG / LLM pipeline** with a whole documentation site or knowledge base.
- **Find every contact point** a business publishes across its site, not just the homepage.
- **Audit a site** – thin pages, broken links (`http_error`), parked domains, tech stack.
- **Monitor content** – schedule it weekly and diff the dataset.

Pair with **Website Email & Contact Scraper + Page Content Extractor** when you already
have the exact URLs, and with the **Google Maps** actors to go from "dentists in Lisbon"
to every clinic's full website.

***

### ▶️ How to use it

1. Paste the homepage(s) into **Websites to crawl**.
2. Set **Max pages per site** and **Max link depth** (defaults: 50 and 3).
3. Optionally restrict with **Only crawl URLs containing** (e.g. `/docs/`) or
   **Skip URLs containing** (e.g. `/tag/`).
4. Run. Download JSON / CSV / Excel, or push to your pipeline via the API.

### ⚙️ Input

| Field | Type | Default | Notes |
|---|---|---|---|
| `startUrls` | list | – | Site homepages, up to 50 per run |
| `maxPagesPerSite` | integer | 50 | 1–2,000 |
| `maxDepth` | integer | 3 | 0 = start page only |
| `includeSubdomains` | boolean | false | Follow blog.example.com etc. |
| `includeUrlPatterns` | string\[] | – | Keep only URLs containing one of these |
| `excludeUrlPatterns` | string\[] | – | Skip URLs containing any of these |
| `useSitemap` | boolean | true | Seed from sitemap.xml |
| `respectRobots` | boolean | true | Skip disallowed paths |
| `outputFormat` | `markdown` / `text` | `markdown` | |
| `maxChars` | integer | 50,000 | Cut and marked `truncated` |
| `includeContacts` | boolean | true | Emails, socials, WhatsApp, contact page, tech stack |
| `includeLinks` | boolean | false | Outgoing links per page (up to 500) |
| `concurrency` | integer | 5 | Parallel fetches per site (1–10) |

### 🧾 Output

One row per page – the same fields as the single-page extractor plus where it sits in
the site:

```json
{
  "site": "acme.example",
  "url": "https://acme.example/services/repair",
  "depth": 2,
  "found_on": "https://acme.example/services",
  "status": "ok",
  "http_status": 200,
  "title": "Widget repair – Acme",
  "description": "Same-day widget repair since 1999.",
  "language": "en",
  "headings": ["Widget repair", "Pricing"],
  "content": "# Widget repair\n\nSame-day repair…",
  "word_count": 512,
  "truncated": false,
  "emails": ["repairs@acme.example"],
  "socials": {"instagram": "https://www.instagram.com/acme"},
  "whatsapp": "+971501234567",
  "contact_form_url": "https://acme.example/contact",
  "tech_stack": ["wordpress", "google-analytics"],
  "fetched_at": "2026-09-06T12:00:00+00:00"
}
```

***

### 🛡️ What it deliberately does not do

- **No personal data.** Only role-based business mailboxes (`info@`, `sales@`…);
  person-named mailboxes are dropped. Nothing is collected from social profiles.
- **No JavaScript rendering.** Client-side apps come back `thin` rather than faked.
- **No bypassing.** A challenge or block is reported as `blocked` and we back off;
  robots.txt is honoured; private addresses are refused.
- **No cross-site wandering.** Only the site you gave (and its subdomains if you ask).

***

### ❓ FAQ

**How many pages will it find?** Everything reachable by links within the depth limit,
plus the sitemap, up to your page cap. `depth` and `found_on` show how each page was reached.

**Can I crawl only a section?** Yes – `includeUrlPatterns: ["/docs/"]`.

**Why is a page `thin`?** Fewer than ~40 words of server-rendered text – usually a
React/Vue app. Contacts may still be found in its HTML.

***

### 🧑‍💻 Support

Issues tab, or hello@locomint.io. Part of the Locomint local business data toolkit.

# Actor input Schema

## `startUrls` (type: `array`):

One or more site homepages (full URLs with https://). Each site is crawled separately, up to 50 sites per run.

## `maxPagesPerSite` (type: `integer`):

Stop after this many pages on a site. You pay per page delivered.

## `maxDepth` (type: `integer`):

0 = only the start page, 1 = pages linked from it, and so on.

## `includeSubdomains` (type: `boolean`):

Also follow links to blog.example.com, shop.example.com, etc.

## `includeUrlPatterns` (type: `array`):

Optional. Keep only URLs containing one of these texts, e.g. \["/docs/"]. The start page is always included.

## `excludeUrlPatterns` (type: `array`):

Optional. Skip URLs containing any of these texts, e.g. \["/tag/", "?page="].

## `useSitemap` (type: `boolean`):

Seed the crawl from sitemap.xml when the site has one - finds pages that are not linked from the homepage.

## `respectRobots` (type: `boolean`):

Skip pages the site asks crawlers not to visit. Leave on.

## `outputFormat` (type: `string`):

Markdown keeps headings, lists, links and code; plain text is just the words.

## `maxChars` (type: `integer`):

Longer content is cut here and marked truncated.

## `includeContacts` (type: `boolean`):

Business emails (role addresses only), social profiles, WhatsApp, contact page URL, tech stack - from every page, no extra cost.

## `includeLinks` (type: `boolean`):

Add the list of links found on each page (up to 500).

## `concurrency` (type: `integer`):

Pages fetched at the same time per site.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://www.apify.com/"
    }
  ],
  "maxPagesPerSite": 5,
  "maxDepth": 3,
  "includeSubdomains": false,
  "useSitemap": true,
  "respectRobots": true,
  "outputFormat": "markdown",
  "maxChars": 50000,
  "includeContacts": true,
  "includeLinks": false,
  "concurrency": 5
}
```

# Actor output Schema

## `pages` (type: `string`):

One row per crawled page: site, depth, status, title, clean Markdown/text content, and business contacts found on the page.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://www.apify.com/"
        }
    ],
    "maxPagesPerSite": 5
};

// Run the Actor and wait for it to finish
const run = await client.actor("locomint/website-crawler-content-contacts").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://www.apify.com/" }],
    "maxPagesPerSite": 5,
}

# Run the Actor and wait for it to finish
run = client.actor("locomint/website-crawler-content-contacts").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://www.apify.com/"
    }
  ],
  "maxPagesPerSite": 5
}' |
apify call locomint/website-crawler-content-contacts --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,locomint/website-crawler-content-contacts"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/uirTcQlqBVTe8rQdt/builds/dQUQM8NrSYq6clWhe/openapi.json