Docs to LLM Markdown (incremental)

Turns a documentation or help-centre site into clean Markdown, split into heading-aware chunks ready for a RAG pipeline or vector database. Turn on incremental mode for scheduled runs and it only delivers pages that changed since the last run, so you pay the full page price for, and re-embed, only what is new.

Incremental vs. full re-crawls: a full re-crawl delivers and charges every page on every run. In incremental mode an unchanged page costs a fifth of a delivered page and is not delivered again. For a 200-page site where 20 pages change a day, a daily full re-crawl costs $0.22 a day; incremental mode costs about $0.08 a day after the first run (worked out under Pricing).

What you get

One dataset item per page:

Field	Meaning
url, title	The page
status	new or changed since your last run
contentHash	SHA-256 of the page Markdown
tokenEstimate	Rough token count (about 4 characters per token)
markdown	The whole page as Markdown
chunks	index, headings (the heading path, such as ["Install", "With pip"]), markdown, tokenEstimate
crawledAt	UTC time of the fetch

Navigation, headers, footers, scripts and forms are removed, including menus, sidebars, tables of contents and breadcrumbs that a site builds from plain divs (recognised by their class, id or role, and only when they are mostly links and hold no main content). Code blocks keep their language, and tables become Markdown tables.

Example input

{
  "startUrl": "https://docs.example.com/guide/",
  "maxPages": 200,
  "incremental": true,
  "delaySeconds": 1,
  "maxChunkChars": 4000
}

Only startUrl matters for a first try; the rest are the defaults, except maxPages (default 50) and incremental (default off).

Example output item

One page from an example site (the site and text are made up for illustration):

{
  "url": "https://docs.example.com/guide/install",
  "title": "Install | Widget docs",
  "status": "changed",
  "contentHash": "51a0b14b5e2192b8d080165a70db11ee2d2b2cf396da4afec38d9ce946177505",
  "tokenEstimate": 34,
  "crawledAt": "2026-10-15T06:00:12+00:00",
  "markdown": "# Install\n\nWidget needs **Python 3.11** or later.\n\n## With pip\n\n```bash\npip install widget\n```\n\nCheck it worked with `widget --version`.",
  "chunks": [
    {
      "index": 0,
      "headings": ["Install"],
      "markdown": "# Install\n\nWidget needs **Python 3.11** or later.",
      "tokenEstimate": 12
    },
    {
      "index": 1,
      "headings": ["Install", "With pip"],
      "markdown": "## With pip\n\n```bash\npip install widget\n```\n\nCheck it worked with `widget --version`.",
      "tokenEstimate": 21
    }
  ]
}

How it crawls

• Reads robots.txt first and always obeys it, including Crawl-delay. This cannot be turned off.
• Uses the site's sitemaps first, then follows links. It stays on the same site and under the start URL's folder.
• Fetches one page at a time, with at least 0.5 seconds between requests, and stops at once if the site answers 429 or 503.
• Skips pages that need a login, pages marked noindex, and pages that look like people listings (team, staff or author pages, or pages with several email addresses). Any other email address or phone number is replaced with [email removed] or [phone removed].
• Refuses IP addresses and hosts that point to private or internal networks, including on redirects.

Your responsibility

Only crawl sites whose terms allow automated access and reuse of their content, such as your own docs or docs published under an open licence. The crawler takes public pages only and does not bypass logins or captchas.

For site owners

This crawler identifies itself as HustleStaxxDocsCrawler and obeys robots.txt. To block it, add this to your robots.txt:

User-agent: HustleStaxxDocsCrawler
Disallow: /

Pricing

You pay per event, in US dollars:

Event	Price	When
Run start	$0.02	Once per run
Page delivered	$0.001	Each new or changed page written to the dataset
Page unchanged	$0.0002	Incremental mode only: each page checked and found the same as last run

Skipped pages (blocked by robots.txt, needing a login, not HTML, and so on) cost nothing.

Worked example: a 200-page docs site (set maxPages to 200 or more)

• First run: $0.02 + 200 × $0.001 = $0.22.
• Each daily run after that in incremental mode, if 20 pages changed and 180 did not: $0.02 + 20 × $0.001 + 180 × $0.0002 = $0.02 + $0.02 + $0.036 = about $0.08.
• The same daily run as a full re-crawl: $0.22 every day.
• Over 30 days: about $2.42 in incremental mode ($0.22 + 29 × $0.076) against $6.60 for daily full re-crawls.

Set a maximum cost per run in the Apify Console to cap spending; the crawler stops cleanly when it is reached and the run still ends as a success with the pages delivered so far.

Limits

• One site per run: the same host as the start URL, and only pages under the start URL's folder.
• Up to 10,000 pages per run (maxPages, default 50). Pages that are skipped count toward the limit, because they were still fetched.
• One request at a time, at least 0.5 seconds apart (default 1 second, longer if robots.txt asks). A 200-page site takes at least 3 to 4 minutes.
• HTML pages only. PDFs and other files are skipped, and so are pages larger than 5 MB.
• It does not run JavaScript, log in or solve captchas.
• Reads at most 20 sitemap files.
• Chunk size is set by maxChunkChars (default 4,000 characters, minimum 500). Chunks split only between blocks, so a single code block, table or paragraph longer than the limit makes one chunk longer than the limit.

Troubleshooting

• The run failed with "Could not read any page: robots.txt does not allow the start URL". The site's robots.txt blocks this crawler or all crawlers from that folder. The crawler always obeys robots.txt and cannot be made to ignore it. Ask the site owner, or crawl a site you own or one that allows it.
• The run failed with "Could not read any page: ..." and another reason. No page could be read, so the run is marked as failed instead of succeeding with an empty dataset. The message says why, for example robots.txt returned 500 (the site is down; try later) or every page was skipped (12 needs login or forbidden) (the docs are behind a login). You still pay the $0.02 start fee for that run.
• Pages come back empty or are skipped as "no text content". The site is probably built with JavaScript only: the page text is drawn in the browser and is not in the HTML. This crawler does not run JavaScript. Look for a static version of the docs (many projects publish them on GitHub or as a plain HTML export) and use that as the start URL.
• The run stopped early with "site asked us to slow down (429)". Raise delaySeconds and run again.
• Some pages are missing. Check they sit under the start URL's folder, that they are linked from other pages or listed in the sitemap, and that maxPages is high enough. The status message lists every reason a page was skipped.
• Incremental mode delivered every page again. The first incremental run for a site has nothing to compare with, so every page is new. Changes are tracked per site and start folder, so a different start URL starts a fresh history.

Change log

0.2 (2026-10-15)

• A run that cannot read a single page (for example, blocked by robots.txt or the site is down) now fails with a clear message, instead of ending as a success with 0 pages. A run that stops at your spending limit still counts as a success.
• Menus, sidebars, tables of contents and breadcrumbs built from plain divs are now removed from the text.
• New tests: the page limit is never exceeded, div-built sidebars are removed without losing main content, and a run with nothing read fails.
• README: prices in money with a worked example, example input and output, limits, troubleshooting and this change log.

0.1 updates (2026-10-14)

• Incremental mode is now off by default (turn it on for scheduled runs), and the start URL defaults to the Apify Academy docs so a run with empty input works.
• Hardened against hostile pages: email and phone masking stays fast on crafted text, a page nested too deeply to read is skipped instead of failing the run, and pages over 5 MB are skipped.
• Added output and dataset schemas, so the Output tab shows an overview table.

0.1 (2026-10-14)

• First release.

Local development

python -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt pytest
python -m pytest -q   # uses a fake site, no network needed