# Website Content Crawler to Markdown by Country (`zaiq/website-crawler-by-country`) Actor

Crawl a website and get each page's main content as clean Markdown or text for RAG, LLM and AI agents, fetched from the country you pick through residential (home) connections. Geo-targeted, with proof of the country for every page. JavaScript when needed. Blocked pages are free.

- **URL**: https://apify.com/zaiq/website-crawler-by-country.md
- **Developed by:** [Zaiq](https://apify.com/zaiq) (community)
- **Categories:** AI, Developer tools, SEO tools
- **Stats:** 3 total users, 2 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $8.00 / 1,000 crawled pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website Content Crawler to Markdown by Country

Crawl a website and get each page's main content as clean Markdown or plain text, ready for RAG pipelines, LLMs and
AI agents, with every page fetched from the country you choose through real home (residential) internet connections
there. Each row shows where its page was fetched from: the connection is confirmed in the country by two IP-location
services before the page, and confirmed again after it.

### What it does

- Starts from your start URLs and follows links within the scope you set (the start URL's folder, the whole site, or
  the whole domain), nearest pages first, up to your page and depth limits. Optionally adds the pages listed in the
  site's sitemaps.
- Fetches every page from one country (any two-letter country code) through residential connections, asking for that
  country's usual browser language.
- Returns each page's main content as Markdown (headings, lists, tables, links, code blocks with their language)
  and/or plain text. Navigation, site headers and footers, sidebars, cookie and consent banners, breadcrumbs, tables of
  contents and edit links are removed.
- Finds pages whose content appears only after JavaScript runs and loads just those in a real browser (Automatic
  mode). You can also use a browser for every page, or never.
- Reads robots.txt and skips what it disallows for all crawlers, honours `Crawl-delay`, and never fetches more than 10
  pages at once from one site.
- Skips duplicates: the same address with tracking parameters or a `#fragment`, redirects to a page already crawled, a
  page's canonical address, and pages whose text is identical to one already delivered.
- Gives each page's title, description, language, canonical URL, HTTP status, word count and number of links, and can
  save each page's full HTML as a file.
- Proof of location on every row: the country, city and network of the connection, the two services that confirmed
  it, when, and when the same address was confirmed again after the page.

If no connection can be confirmed in the country, the run says so and charges nothing. It never falls back to another
country.

### Who uses it

- **AI and RAG teams** building a knowledge base from a site as it is served in one market: regional help centres,
  documentation, policies and product pages.
- **Localisation and SEO teams** checking what visitors in one country get across a whole site: language, redirects,
  canonical addresses, missing pages.
- **Compliance and monitoring**: a record of a site's regional terms, notices and disclosures as served in a country,
  with evidence of where each page was fetched from.
- **Market research** on sites that show different content in different countries.

### Example

The prefilled input crawls 10 pages of the Python tutorial from the United States. A larger test run on 4 October
2026 crawled the Python standard library documentation (`https://docs.python.org/3/library/`) from Germany:

| | |
|---|---|
| Pages delivered | 40 of 40 tried (none failed, blocked or duplicated) |
| Time | 12 seconds of crawling |
| Text | 119,460 words, as Markdown and plain text |
| Connections | 5 home connections in Dortmund (2), Essen, Ludwigshafen am Rhein and Hannover; each confirmed by ipinfo.io and ifconfig.co before and after its pages |
| Traffic | 744 KB in total through the residential network, about 19 KB a page with all checks included |

Other test runs the same day:

- 400 pages of the Python documentation from the US: 400 delivered, 6 connections, 9.6 MB of traffic.
- 40 pages from South Africa (connections in Cape Town and Johannesburg): 40 delivered, 749 KB.
- A public practice site whose quotes are added by JavaScript, from the UK: the plain fetch saw 4 words, so the page
  was loaded in a browser, which found 256; a second address of the same page was recognised as a duplicate and
  skipped free.
- Twelve start URLs including a missing page (404), a page that answers 403, one that answers 500, an image, a JSON
  file, three addresses robots.txt disallows and a local address: 3 pages delivered and charged; the 5 failed ones came
  back as free rows, the 3 disallowed ones were skipped and the local address was refused.
- Connections that the two services placed in different countries (US and Switzerland, Canada and the US) were not
  used.

### Input

```json
{
  "startUrls": [{ "url": "https://docs.python.org/3/library/" }],
  "country": "de",
  "maxPages": 40,
  "crawlScope": "path",
  "renderMode": "auto",
  "outputFormats": ["markdown", "text"]
}
```

| Field | What it does |
|---|---|
| `startUrls` | Where the crawl starts. |
| `country` | Two-letter country code. Every page is fetched through a residential connection in this country. |
| `maxPages` | Stop after this many delivered pages (up to 10,000). Only delivered pages are charged. |
| `maxDepth` | How many links away from a start URL to go. 0 fetches only the start URLs. Default 20. |
| `crawlScope` | `path` (default): only pages under the start URL's folder. `hostname`: the whole site. `domain`: the domain and its subdomains. |
| `includeUrlPatterns`, `excludeUrlPatterns` | Globs such as `https://docs.example.com/api/**` or `/blog/*`, or regular expressions starting with `re:`, such as `re:[?&]page=`. Include patterns replace `crawlScope`; exclude patterns always apply. |
| `useSitemaps` | Also queue the pages listed in the site's sitemaps. |
| `renderMode` | `auto` (default): plain fetch, browser only for pages that need it. `never`: plain fetch only. `always`: every page in a browser. |
| `outputFormats` | `markdown` (default), `text`, `html` (the full HTML saved as a file and linked from the row). |
| `removeBoilerplate` | Keep only the main content. On by default. |
| `respectRobotsTxt` | On by default. |
| `includeFailedPages` | Add a free row for every page that failed or was blocked. On by default; switch it off when the results go straight into a vector database. |
| `contentSelector`, `removeSelector` | Optional CSS selectors: where the main content is, and elements to drop. |
| `maxConcurrency`, `maxTriesPerPage`, `acceptLanguage` | Advanced: pages at once (default 8), connections to try per page (default 3), and a fixed Accept-Language header. |

### Output

One row per page. A row from the German run above, with the Markdown and text shortened:

```json
{
  "url": "https://docs.python.org/3/library/copy.html",
  "finalUrl": "https://docs.python.org/3/library/copy.html",
  "country": "DE",
  "depth": 1,
  "referrerUrl": "https://docs.python.org/3/library/",
  "status": 200,
  "title": "copy — Shallow and deep copy operations — Python 3.14.8 documentation",
  "description": "Source code: Lib/copy.py Assignment statements in Python do not copy objects, ...",
  "language": "en",
  "canonicalUrl": "https://docs.python.org/3/library/copy.html",
  "markdown": "# `copy` — Shallow and deep copy operations\n\n**Source code:** [Lib/copy.py](https://github.com/python/cpython/tree/3.14/Lib/copy.py)\n\n---\n\nAssignment statements in Python do not copy objects, ...",
  "text": "copy — Shallow and deep copy operations\n\nSource code: Lib/copy.py\n\n---\n\nAssignment statements in Python do not copy objects, ...",
  "wordCount": 599,
  "linksFound": 22,
  "crawledAt": "2026-10-04T14:11:52Z",
  "exit": {
    "country": "DE",
    "city": "Dortmund",
    "network": "AS8881 1&1 Versatel GmbH",
    "verified": true,
    "checkedWith": ["ipinfo.io", "ifconfig.co"],
    "checkedAt": "2026-10-04T14:11:48Z",
    "reconfirmedAt": "2026-10-04T14:11:53Z"
  },
  "rendered": false,
  "blocked": false,
  "countryUnavailable": false,
  "error": null,
  "charged": true
}
```

- `exit.checkedAt` is when the connection was confirmed in the country before the page; `exit.reconfirmedAt` is when
  the same address was confirmed again after it.
- `rendered` is `true` when the content came from a real browser.
- Failed and blocked pages have `error` set, `charged: false` and no content.
- With full HTML on, rows also have `htmlUrl`. A very long page is cut at 1,000,000 characters and has
  `contentTruncated: true`.
- The run's key-value store also holds `CRAWL-SUMMARY`: pages delivered, failed, blocked, duplicated and excluded by
  robots.txt, the connections used and approximate traffic.

For LangChain, LlamaIndex or a vector database, use `markdown` (or `text`) as the document and `url`, `title` and
`exit.country` as metadata.

### Use it from code, automations and AI agents

- **API**: run it and get the results in one call: `POST https://api.apify.com/v2/acts/zaiq~website-crawler-by-country/run-sync-get-dataset-items?token=YOUR_TOKEN` with the input as JSON. The API tab on this page has ready-made Python, JavaScript and cURL examples.
- **Python** (`pip install apify-client`):

```python
from apify_client import ApifyClient

client = ApifyClient("YOUR_TOKEN")
run = client.actor("zaiq/website-crawler-by-country").call(run_input={'startUrls': [{'url': 'https://docs.python.org/3/tutorial/'}], 'country': 'us', 'maxPages': 10})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)
```

- **No code**: connect it to Google Sheets, Zapier, Make or n8n from the Integrations tab, schedule runs in Apify Console, and get a webhook when a run finishes.
- **AI agents (MCP)**: add it as a tool to Claude, Cursor or any MCP client through Apify's MCP server: `https://mcp.apify.com?tools=zaiq/website-crawler-by-country`.

### Pricing

You pay only for pages delivered from the requested country. Apify's platform usage is included.

| Event | Price |
|---|---|
| Page crawled (plain fetch) | $0.01 ($10 per 1,000 pages) |
| Page crawled in a browser (the content needed JavaScript) | $0.05 ($50 per 1,000 pages) |

Plus Apify's standard run start fee.

In Automatic mode the browser price applies only when the browser found clearly more text than the plain fetch;
otherwise the page is charged at the plain price.

Free: pages that fail or return an error (4xx or 5xx), pages blocked by the site, duplicates, pages excluded by
robots.txt, pages whose connection could not be confirmed again after them (they are fetched again through another
connection), and every retry. A row is charged only when its `charged` field is `true`. Your spending limit for the
run is respected: the crawl stops cleanly before a page would go over it.

### Limits

- Pages behind a login, forms, and content that appears only after scrolling or clicking are not covered.
- Only web pages (HTML) are returned. Links to PDFs, images and other files are skipped.
- Some sites block automated visits even from home connections. Those pages come back as `blocked` and cost nothing.
  If 50 fetches in a row deliver nothing (25 with a browser for every page), the crawl stops rather than keep trying.
- Countries with few home connections may return `countryUnavailable` (free).
- Up to 10,000 pages per run and 50 at once, never more than 10 at once from one site.
- Main-content detection is automatic. For unusual layouts, set `contentSelector` or `removeSelector`.
- Speed depends on the site and on the home connections available in the country.
- Crawl only sites you are allowed to access, and follow their terms. robots.txt is respected by default.

### FAQ

**How do I know each page really came from that country?** Before a connection is used, two IP-location services
(ipinfo.io and ifconfig.co, with ipapi.co and country.is as fallbacks) are asked, through that same connection, where
it is. It is used only if both place it in the country. After it has fetched a few pages it is checked again, and its
pages are written to the dataset only if the same address is still there and still in the country. If the address
changed in between (home connections can drop out), those pages are fetched again through another connection, free.
Each row's `exit` shows the result.

**Why does a browser page cost more?** A browser also downloads the page's scripts: about 1.2 MB a page in our tests,
against about 20 KB for a plain fetch, all through paid residential connections.

**What if the run is interrupted?** The crawl saves its progress. When Apify moves the run to another server, or you
resurrect an aborted run, it continues where it stopped, and pages already in the dataset are never written or charged
again.

**Does it work for sites built with JavaScript?** Yes. In Automatic mode a page that looks empty without JavaScript (an
empty app container, a notice asking for JavaScript, or almost no text) is loaded again in Chrome through the same
verified connection.

**Can I crawl several sites in one run?** Yes. Add several start URLs; each one sets the scope for its own site.

**Is the connection's IP address shown?** No. Rows show the country, city and network only.

# Actor input Schema

## `startUrls` (type: `array`):

Where the crawl starts. By default it follows links under each start URL's folder: https://docs.example.com/guide/ crawls https://docs.example.com/guide/...

## `country` (type: `string`):

Every page is fetched through a home (residential) internet connection in this country, confirmed by two IP-location services. Pick one, or type any two-letter country code.

## `maxPages` (type: `integer`):

The crawl stops after this many pages have been delivered. Only delivered pages are charged.

## `maxDepth` (type: `integer`):

How many links away from a start URL the crawl may go. 0 fetches only the start URLs.

## `crawlScope` (type: `string`):

Which links found on the pages are followed. Include patterns below, when given, replace this.

## `includeUrlPatterns` (type: `array`):

Optional. Globs such as https://docs.example.com/api/\*\* or /blog/\*, or regular expressions starting with re: (for example re:/v\[0-9]+/). \* matches anything except /, \*\* matches anything. When set, these decide which links are followed instead of the option above.

## `excludeUrlPatterns` (type: `array`):

Optional. Same pattern format, for example /tag/\*\* or re:\[?&]page=. Always applied.

## `useSitemaps` (type: `boolean`):

Read the sitemaps listed in robots.txt (or /sitemap.xml) and queue their pages that are in scope, as well as following links.

## `renderMode` (type: `string`):

Automatic fetches each page plainly first and loads it in a real browser only when the plain page is missing content that JavaScript adds; the browser price is charged only when the browser found more text.

## `outputFormats` (type: `array`):

Markdown keeps headings, lists, tables, links and code blocks. Full HTML is saved as a file in the run's key-value store and linked from each row.

## `removeBoilerplate` (type: `boolean`):

Keep only the page's main content. Switch off to get all of the page's text.

## `respectRobotsTxt` (type: `boolean`):

Skip pages the site's robots.txt disallows for all crawlers (User-agent: \*), and wait between requests when it sets a Crawl-delay (up to 10 seconds).

## `includeFailedPages` (type: `boolean`):

Add a free row for each page that failed, was blocked or returned an error (4xx/5xx). Switch off to keep only delivered pages, for example when the results go straight into a vector database.

## `maxConcurrency` (type: `integer`):

How many pages are fetched at the same time. Never more than 10 at once from one site.

## `maxTriesPerPage` (type: `integer`):

If a page fails or meets a bot wall, it is tried again through another address in the country, up to this many times. Failed tries are free.

## `acceptLanguage` (type: `string`):

Leave empty to send the country's usual browser language (for example de-DE in Germany). Set it to ask every page for one language.

## `contentSelector` (type: `string`):

Optional. Take the text only from elements matching this selector, for example article.post or #content. Leave empty to find the main content automatically.

## `removeSelector` (type: `string`):

Optional. Elements to drop before the text is taken, for example .related-posts, .ad-slot.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.python.org/3/tutorial/"
    }
  ],
  "country": "us",
  "maxPages": 10,
  "maxDepth": 20,
  "crawlScope": "path",
  "includeUrlPatterns": [],
  "excludeUrlPatterns": [],
  "useSitemaps": false,
  "renderMode": "auto",
  "outputFormats": [
    "markdown"
  ],
  "removeBoilerplate": true,
  "respectRobotsTxt": true,
  "includeFailedPages": true,
  "maxConcurrency": 8,
  "maxTriesPerPage": 3
}
```

# Actor output Schema

## `pages` (type: `string`):

Dataset with one row per crawled page (failed and blocked pages are free rows unless switched off).

## `summary` (type: `string`):

Pages delivered, failed, blocked, duplicates and robots.txt exclusions, the addresses used and approximate traffic.

## `files` (type: `string`):

Key-value store with each delivered page's full HTML (html-<id>) when Full HTML output is on.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.python.org/3/tutorial/"
        }
    ],
    "country": "us",
    "maxPages": 10
};

// Run the Actor and wait for it to finish
const run = await client.actor("zaiq/website-crawler-by-country").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://docs.python.org/3/tutorial/" }],
    "country": "us",
    "maxPages": 10,
}

# Run the Actor and wait for it to finish
run = client.actor("zaiq/website-crawler-by-country").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.python.org/3/tutorial/"
    }
  ],
  "country": "us",
  "maxPages": 10
}' |
apify call zaiq/website-crawler-by-country --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,zaiq/website-crawler-by-country"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/gSA7gf0wfsqSWkWdY/builds/HVHSfkYFlCTKY8Fh1/openapi.json