Docs To Markdown
Pricing
from $1.00 / 1,000 results
Go to Apify Store
Docs To Markdown
Pricing
from $1.00 / 1,000 results
Where to start crawling. Paste the top of the docs section you want, for example https://docs.example.com/guide. By default the crawl stays inside that path, so a run started at /guide will not wander into /blog or /pricing.
[ { "url": "https://docs.apify.com/academy" }]Hard ceiling on pages fetched. This is a real limit, not a hint: the crawl stops the moment it is reached. Kept deliberately low by default so an exploratory run cannot turn into a surprise bill.
How many link hops from a start URL to follow. 0 crawls only the start URLs themselves.
Ignore the start URL's path and allow any page on the same domain. Leave this off to keep a docs crawl inside the docs.
Override the automatic scope with an explicit list of prefixes. A page is crawled only if its URL starts with one of them.
[]Regular expressions. Any URL matching one of them is skipped. Useful for changelogs, tag pages, or the /v1/ copy of versioned docs you do not want duplicated in your vector database.
[]Read /sitemap.xml and queue every in-scope URL it lists, in addition to following links. Sitemap URLs still obey the scope prefixes and the max pages limit.
Coverage is the share of a page's available content text that survived extraction. Below this value the page is flagged with a warning, and the extractor retries with a more permissive strategy before giving up. Raise it to be stricter about partial extractions.
Leave off to keep every page with its warning attached, which is usually what you want while you are still tuning. Turn on to keep the dataset clean for a production ingest.
Add the original HTML of each page to the output. Useful for auditing a page the coverage check flagged. Makes the dataset much larger.
How many pages to fetch in parallel. Lower this if the documentation site rate-limits you.
Seconds to wait for a single page before giving up on it and moving on.
Stop crawling after this many seconds and write out whatever has been extracted so far. 0 means no limit beyond the platform run timeout. A second guardrail against a crawl that never ends.
Override the browser User-Agent header sent with every request.