Website Content Crawler to Markdown by Country

Crawl a website and get each page's main content as clean Markdown or plain text, ready for RAG pipelines, LLMs and AI agents, with every page fetched from the country you choose through real home (residential) internet connections there. Each row shows where its page was fetched from: the connection is confirmed in the country by two IP-location services before the page, and confirmed again after it.

What it does

• Starts from your start URLs and follows links within the scope you set (the start URL's folder, the whole site, or the whole domain), nearest pages first, up to your page and depth limits. Optionally adds the pages listed in the site's sitemaps.
• Fetches every page from one country (any two-letter country code) through residential connections, asking for that country's usual browser language.
• Returns each page's main content as Markdown (headings, lists, tables, links, code blocks with their language) and/or plain text. Navigation, site headers and footers, sidebars, cookie and consent banners, breadcrumbs, tables of contents and edit links are removed.
• Finds pages whose content appears only after JavaScript runs and loads just those in a real browser (Automatic mode). You can also use a browser for every page, or never.
• Reads robots.txt and skips what it disallows for all crawlers, honours Crawl-delay, and never fetches more than 10 pages at once from one site.
• Skips duplicates: the same address with tracking parameters or a #fragment, redirects to a page already crawled, a page's canonical address, and pages whose text is identical to one already delivered.
• Gives each page's title, description, language, canonical URL, HTTP status, word count and number of links, and can save each page's full HTML as a file.
• Proof of location on every row: the country, city and network of the connection, the two services that confirmed it, when, and when the same address was confirmed again after the page.

If no connection can be confirmed in the country, the run says so and charges nothing. It never falls back to another country.

Who uses it

• AI and RAG teams building a knowledge base from a site as it is served in one market: regional help centres, documentation, policies and product pages.
• Localisation and SEO teams checking what visitors in one country get across a whole site: language, redirects, canonical addresses, missing pages.
• Compliance and monitoring: a record of a site's regional terms, notices and disclosures as served in a country, with evidence of where each page was fetched from.
• Market research on sites that show different content in different countries.

Example

The prefilled input crawls 10 pages of the Python tutorial from the United States. A larger test run on 4 October 2026 crawled the Python standard library documentation (https://docs.python.org/3/library/) from Germany:

Pages delivered	40 of 40 tried (none failed, blocked or duplicated)
Time	12 seconds of crawling
Text	119,460 words, as Markdown and plain text
Connections	5 home connections in Dortmund (2), Essen, Ludwigshafen am Rhein and Hannover; each confirmed by ipinfo.io and ifconfig.co before and after its pages
Traffic	744 KB in total through the residential network, about 19 KB a page with all checks included

Other test runs the same day:

• 400 pages of the Python documentation from the US: 400 delivered, 6 connections, 9.6 MB of traffic.
• 40 pages from South Africa (connections in Cape Town and Johannesburg): 40 delivered, 749 KB.
• A public practice site whose quotes are added by JavaScript, from the UK: the plain fetch saw 4 words, so the page was loaded in a browser, which found 256; a second address of the same page was recognised as a duplicate and skipped free.
• Twelve start URLs including a missing page (404), a page that answers 403, one that answers 500, an image, a JSON file, three addresses robots.txt disallows and a local address: 3 pages delivered and charged; the 5 failed ones came back as free rows, the 3 disallowed ones were skipped and the local address was refused.
• Connections that the two services placed in different countries (US and Switzerland, Canada and the US) were not used.

Input

{
  "startUrls": [{ "url": "https://docs.python.org/3/library/" }],
  "country": "de",
  "maxPages": 40,
  "crawlScope": "path",
  "renderMode": "auto",
  "outputFormats": ["markdown", "text"]
}

Field	What it does
startUrls	Where the crawl starts.
country	Two-letter country code. Every page is fetched through a residential connection in this country.
maxPages	Stop after this many delivered pages (up to 10,000). Only delivered pages are charged.
maxDepth	How many links away from a start URL to go. 0 fetches only the start URLs. Default 20.
crawlScope	path (default): only pages under the start URL's folder. hostname: the whole site. domain: the domain and its subdomains.
includeUrlPatterns, excludeUrlPatterns	Globs such as https://docs.example.com/api/** or /blog/*, or regular expressions starting with re:, such as re:[?&]page=. Include patterns replace crawlScope; exclude patterns always apply.
useSitemaps	Also queue the pages listed in the site's sitemaps.
renderMode	auto (default): plain fetch, browser only for pages that need it. never: plain fetch only. always: every page in a browser.
outputFormats	markdown (default), text, html (the full HTML saved as a file and linked from the row).
removeBoilerplate	Keep only the main content. On by default.
respectRobotsTxt	On by default.
includeFailedPages	Add a free row for every page that failed or was blocked. On by default; switch it off when the results go straight into a vector database.
contentSelector, removeSelector	Optional CSS selectors: where the main content is, and elements to drop.
maxConcurrency, maxTriesPerPage, acceptLanguage	Advanced: pages at once (default 8), connections to try per page (default 3), and a fixed Accept-Language header.

Output

One row per page. A row from the German run above, with the Markdown and text shortened:

{
  "url": "https://docs.python.org/3/library/copy.html",
  "finalUrl": "https://docs.python.org/3/library/copy.html",
  "country": "DE",
  "depth": 1,
  "referrerUrl": "https://docs.python.org/3/library/",
  "status": 200,
  "title": "copy — Shallow and deep copy operations — Python 3.14.8 documentation",
  "description": "Source code: Lib/copy.py Assignment statements in Python do not copy objects, ...",
  "language": "en",
  "canonicalUrl": "https://docs.python.org/3/library/copy.html",
  "markdown": "# `copy` — Shallow and deep copy operations\n\n**Source code:** [Lib/copy.py](https://github.com/python/cpython/tree/3.14/Lib/copy.py)\n\n---\n\nAssignment statements in Python do not copy objects, ...",
  "text": "copy — Shallow and deep copy operations\n\nSource code: Lib/copy.py\n\n---\n\nAssignment statements in Python do not copy objects, ...",
  "wordCount": 599,
  "linksFound": 22,
  "crawledAt": "2026-10-04T14:11:52Z",
  "exit": {
    "country": "DE",
    "city": "Dortmund",
    "network": "AS8881 1&1 Versatel GmbH",
    "verified": true,
    "checkedWith": ["ipinfo.io", "ifconfig.co"],
    "checkedAt": "2026-10-04T14:11:48Z",
    "reconfirmedAt": "2026-10-04T14:11:53Z"
  },
  "rendered": false,
  "blocked": false,
  "countryUnavailable": false,
  "error": null,
  "charged": true
}

• exit.checkedAt is when the connection was confirmed in the country before the page; exit.reconfirmedAt is when the same address was confirmed again after it.
• rendered is true when the content came from a real browser.
• Failed and blocked pages have error set, charged: false and no content.
• With full HTML on, rows also have htmlUrl. A very long page is cut at 1,000,000 characters and has contentTruncated: true.
• The run's key-value store also holds CRAWL-SUMMARY: pages delivered, failed, blocked, duplicated and excluded by robots.txt, the connections used and approximate traffic.

For LangChain, LlamaIndex or a vector database, use markdown (or text) as the document and url, title and exit.country as metadata.

Use it from code, automations and AI agents

• API: run it and get the results in one call: POST https://api.apify.com/v2/acts/zaiq~website-crawler-by-country/run-sync-get-dataset-items?token=YOUR_TOKEN with the input as JSON. The API tab on this page has ready-made Python, JavaScript and cURL examples.
• Python (pip install apify-client):

from apify_client import ApifyClient

client = ApifyClient("YOUR_TOKEN")
run = client.actor("zaiq/website-crawler-by-country").call(run_input={'startUrls': [{'url': 'https://docs.python.org/3/tutorial/'}], 'country': 'us', 'maxPages': 10})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

• No code: connect it to Google Sheets, Zapier, Make or n8n from the Integrations tab, schedule runs in Apify Console, and get a webhook when a run finishes.
• AI agents (MCP): add it as a tool to Claude, Cursor or any MCP client through Apify's MCP server: https://mcp.apify.com?tools=zaiq/website-crawler-by-country.

Pricing

You pay only for pages delivered from the requested country. Apify's platform usage is included.

Event	Price
Page crawled (plain fetch)	$0.01 ($10 per 1,000 pages)
Page crawled in a browser (the content needed JavaScript)	$0.05 ($50 per 1,000 pages)

Plus Apify's standard run start fee.

In Automatic mode the browser price applies only when the browser found clearly more text than the plain fetch; otherwise the page is charged at the plain price.

Free: pages that fail or return an error (4xx or 5xx), pages blocked by the site, duplicates, pages excluded by robots.txt, pages whose connection could not be confirmed again after them (they are fetched again through another connection), and every retry. A row is charged only when its charged field is true. Your spending limit for the run is respected: the crawl stops cleanly before a page would go over it.

Limits

• Pages behind a login, forms, and content that appears only after scrolling or clicking are not covered.
• Only web pages (HTML) are returned. Links to PDFs, images and other files are skipped.
• Some sites block automated visits even from home connections. Those pages come back as blocked and cost nothing. If 50 fetches in a row deliver nothing (25 with a browser for every page), the crawl stops rather than keep trying.
• Countries with few home connections may return countryUnavailable (free).
• Up to 10,000 pages per run and 50 at once, never more than 10 at once from one site.
• Main-content detection is automatic. For unusual layouts, set contentSelector or removeSelector.
• Speed depends on the site and on the home connections available in the country.
• Crawl only sites you are allowed to access, and follow their terms. robots.txt is respected by default.

FAQ

How do I know each page really came from that country? Before a connection is used, two IP-location services (ipinfo.io and ifconfig.co, with ipapi.co and country.is as fallbacks) are asked, through that same connection, where it is. It is used only if both place it in the country. After it has fetched a few pages it is checked again, and its pages are written to the dataset only if the same address is still there and still in the country. If the address changed in between (home connections can drop out), those pages are fetched again through another connection, free. Each row's exit shows the result.

Why does a browser page cost more? A browser also downloads the page's scripts: about 1.2 MB a page in our tests, against about 20 KB for a plain fetch, all through paid residential connections.

What if the run is interrupted? The crawl saves its progress. When Apify moves the run to another server, or you resurrect an aborted run, it continues where it stopped, and pages already in the dataset are never written or charged again.

Does it work for sites built with JavaScript? Yes. In Automatic mode a page that looks empty without JavaScript (an empty app container, a notice asking for JavaScript, or almost no text) is loaded again in Chrome through the same verified connection.

Can I crawl several sites in one run? Yes. Add several start URLs; each one sets the scope for its own site.

Is the connection's IP address shown? No. Rows show the country, city and network only.