Website Markdown Crawler

Crawl linked pages within a public website section and export Markdown, readable text and a source URL for each page. Start with a documentation index, blog or help center. The crawler follows links within that URL's origin and path subtree, removes common navigation noise, and records exactly which URLs it attempted.

For recurring imports, supply the previous run's content hashes. Changed pages include their new content; unchanged pages return a small metadata record so your downstream workflow can skip reprocessing them. Every exported page costs the same, including unchanged records.

Ordinary HTTP is the default. Enable renderJavaScript for public content generated by same-origin JavaScript. Both modes use the same extraction, link scope and change hashes. The Actor does not use login cookies, solve access challenges, download files or search the web.

Page-based pricing: $1 per 1,000 exported pages on the Free plan, plus the start event. Unchanged metadata-only records are also billable pages. Both HTTP and JavaScript-rendered pages use this price. Supply URLs; web search is not included.

Try a small crawl

Run the Python tutorial example with five pages. Inspect Overview, switch to Content for the Markdown, and open URL coverage report in the run output.

{
  "startUrls": ["https://docs.python.org/3/tutorial/"],
  "maxPages": 5,
  "maxDepth": 1
}

Start URLs are strings. maxDepth: 0 processes only those URLs. Depth one also follows links found on them. The default is ten page attempts and depth two; the hard limits are 200 attempts and depth five. A redirect or robots.txt check does not consume an extra page attempt.

If the start URL is https://example.com/docs/, /docs/setup is in scope and /blog/ is outside it. Other subdomains, origins and protocols are outside that start URL's scope. Supply their own start URL if you intend to crawl them. A cross-scope redirect stops the run with an explanation; use the final URL shown by your browser.

Render JavaScript content

When a page builds its content in JavaScript, enable rendering and wait for an element that signals the content is present. This example uses the public Quotes to Scrape practice site:

{
  "startUrls": ["https://quotes.toscrape.com/js/"],
  "maxPages": 2,
  "maxDepth": 1,
  "renderJavaScript": true,
  "waitForSelector": ".quote"
}

waitForSelector waits for the first matching element, up to ten seconds after the page load. It does not select the exported content; use contentSelector for that. With no wait selector, the crawler captures the DOM one second after page load. This does not guarantee that a timer, animation or later request has finished. Keep the same mode and selectors when comparing content hashes.

Rendering supports same-origin scripts, stylesheets and GET-based JSON requests. It blocks cross-origin resources, forms and other non-GET requests, frames, images, fonts, media, WebSockets and service workers. Sites that require those features may return partial content or fail the wait selector. Each page has a 45-second rendering limit, at most 40 resource requests and 20 MB of resource text. Failed allowed resources and exceeded limits fail visibly in COVERAGE; intentionally blocked resource requests are counted there. No click, login or challenge-solving step is provided.

The same public-address checks, robots rules and request delays apply to resources. Browser rendering uses more runtime than HTTP, so start with a small page count. The existing Python documentation example remains in HTTP mode at 512 MB.

Output

One dataset item represents one successfully extracted HTML page. The following fields are available in JSON, CSV and Excel exports:

Field	Meaning
url, requested_url	Final fetched page and its queued URL before redirects
title, canonical_url	Page title and declared canonical link, when present
markdown, text	Extracted content; null for an unchanged page
content_hash, previous_hash	SHA-256 of the Markdown and the caller's previous value
change_status	new, changed or unchanged
word_count	Word count of the current extracted text, also present for unchanged pages
depth, source_url	Link depth and the page that discovered this URL; a seed has depth zero and no source URL
scraped_at	UTC observation time

A metadata excerpt from the tested Python tutorial crawl:

{
  "url": "https://docs.python.org/3/tutorial/",
  "title": "The Python Tutorial \u2014 Python 3.14.7 documentation",
  "change_status": "new",
  "word_count": 1057,
  "content_hash": "b216133cb14e3dfcc3d354ef3a83bd77b1d1100d955064e1477e2eac4be3928e",
  "depth": 0
}

The full record also includes the Markdown and readable text.

Markdown retains headings, links, emphasis, lists, fenced code and simple tables. Links become absolute URLs so they still work after export. Code blocks retain whitespace. Complex layouts, table spans and visual components may not translate exactly. Images and their bytes are not exported.

The default content selection prefers a main element or a main role, then a single article, then the document body. When a page contains several articles, it uses the body rather than silently keeping only the first card. Scripts, styles, navigation, sidebars, form controls, footers and heading permalink controls are removed. Text inside a form remains available because some catalogs wrap their product listings in a form.

Select the content you need

Use contentSelector to select a specific element, or excludeSelectors to remove repeated material inside it:

{
  "startUrls": ["https://docs.python.org/3/tutorial/"],
  "maxPages": 5,
  "maxDepth": 1,
  "contentSelector": "div.body",
  "excludeSelectors": [".admonition"]
}

Both settings use CSS selectors. An invalid selector fails before crawling. The content selector must match exactly one element; zero or multiple matches fail at the affected page instead of quietly choosing a header or the first card. Inspect a small run before increasing the page limit. Link discovery uses the page's links independently of content removal.

Compare a later run

Build previousHashes from the final URLs and hashes in your last dataset:

previous_hashes = {row["url"]: row["content_hash"] for row in previous_rows}
next_input = {
    "startUrls": ["https://docs.python.org/3/tutorial/"],
    "maxPages": 5,
    "maxDepth": 1,
    "previousHashes": previous_hashes,
}

Supply next_input to a later run. Matching hashes produce unchanged records with null Markdown/text. Keep your earlier content for those URLs; replace it only when the new record contains content. Retain the same selectors and crawl scope between comparisons, since changing extraction settings also changes hashes.

The Actor stores no shared monitoring state. Your workflow owns the previous hash map and archived content. A page absent from a bounded crawl is not proof of deletion. Check the coverage report before interpreting a missing URL. Hashes compare the observed Markdown, not the site's publication date or legal availability status.

API example

The example below uses the standard Python library. Set APIFY_TOKEN in your environment, then run it to download five pages. Tokens belong in your environment, never in a public Task or shared input file.

import json
import os
from pathlib import Path
from urllib.request import Request, urlopen

actor = "benthepythondev~website-markdown-crawler"
url = f"https://api.apify.com/v2/acts/{actor}/run-sync-get-dataset-items?timeout=300"
payload = {
    "startUrls": ["https://docs.python.org/3/tutorial/"],
    "maxPages": 5,
    "maxDepth": 1,
}
request = Request(
    url,
    data=json.dumps(payload).encode(),
    headers={
        "Authorization": "Bearer " + os.environ["APIFY_TOKEN"],
        "Content-Type": "application/json",
    },
)
with urlopen(request, timeout=330) as response:
    pages = json.load(response)
Path("pages.json").write_text(json.dumps(pages, indent=2), encoding="utf-8")
print(f"Saved {len(pages)} pages")

For longer crawls, start an asynchronous Actor run and wait for its terminal status before reading the dataset. Make, n8n and other Apify integrations can run the same input and retrieve the dataset. Save a Task when you want to reuse a tested configuration; creating a Task does not automatically create a recurring schedule.

Pricing and limits

Free-plan pricing is $1 per 1,000 exported pages, plus $0.00005 per start at 512 MB. Ten exported pages cost $0.01005 at that tier. Bronze receives 10% off, Silver 15%, and Gold or higher 20% off both charge events. Apify's current Pricing tab is authoritative.

Unchanged pages still require a fetch and comparison, so their metadata records have the same page charge. Failed requests, non-HTML responses, robots-disallowed pages and duplicate URLs create no page result charge. Earlier successful records remain billable when a later page fails. The start charge still applies. The Actor observes Apify's maximum-charge setting and stops exporting when that limit is reached.

There is no paid upstream API or residential proxy requirement. Each page response is limited to 3 MB and extracted Markdown to 200,000 characters. The crawler stops with an error rather than silently truncating content beyond those limits. Discovery holds at most 5,000 distinct URLs in one run; the coverage report marks that ceiling if reached.

Coverage and failure behavior

The run's COVERAGE record lists attempted URLs, their depth, extraction status and errors. It also records the number exported, pending queued pages and why the crawl stopped. page_limit means there are discovered URLs left. Reaching a depth limit can also leave parts of a site unexplored, so a completed queue does not establish complete website coverage.

A source/network/extraction error stops the crawl and fails the run while preserving already exported pages and the coverage record. The crawler does not repeatedly retry denied requests. A run with no exported HTML pages fails visibly instead of treating a blocked or empty export as successful.

robots.txt is respected. A missing robots.txt allows normal crawling; an unavailable or redirected robots.txt is treated conservatively. Requests are sequential and at least half a second apart per origin, with longer declared crawl delays respected. Use only websites and material you are entitled to process, and follow their terms and content licenses.

Common questions

Does it work on any website? No. It needs a public page without login or a challenge. Optional rendering supports the same-origin JavaScript subset described above. PDFs, authenticated pages, cross-origin applications and interactive dashboards need a different permitted source or tool.

Does it discover every page? No. It follows links under the supplied URL subtrees within the depth/page limits. It does not consult sitemaps or search engines. Unlinked pages require explicit start URLs.

Why did my crawl stop after a redirect? The destination left the configured origin or subtree, repeated another queued URL, or exceeded the redirect limit. Inspect COVERAGE and use the intended final public URL.

Does an unchanged record overwrite the old content? Your integration decides. Retain your previous content when change_status is unchanged; the null fields mean no replacement content was sent.

Can it crawl my local network? No. It accepts public HTTP(S) websites on standard ports. Private and reserved addresses are rejected.

How do I report a problem? Open an issue on this Actor with the run link, a public test URL and the field you expected. Never post tokens, cookies or private documents. A small failing example is easier to diagnose than a large crawl.

For a supplied list of single pages, see Webpage Text Extractor. For URL discovery from XML sitemaps, see Sitemap URL Extractor. This crawler adds linked-page Markdown and content comparisons to those narrower workflows.