Site to Markdown Crawler

Crawls website pages into dataset rows with extracted body text and Markdown for AI ingestion, RAG pipelines or offline archiving. Uses plain HTTP (no browser). The default memory allocation is 512 MB; 256 MB is the selectable minimum.

Page cap: the default is 500 page attempts. Robots and sitemap discovery requests do not count toward this cap. Each attempt counts toward maxPages, including free error, blocked, robots-disallowed, no-content and spend-skipped rows. Set maxPages to a positive integer to choose a different limit. The form accepts values from 1 to 50,000; the runtime falls back to 500 for a zero value when schema validation is bypassed.

Spend cap: set the maximum total charge in Apify's run options. The crawler stops scheduling new pages when that pay-per-event limit is reached; pages already in progress can finish as free skipped rows. The JSON input field maxTotalChargeUsd does not set the platform run option.

What you get

One dataset row per crawled page:

Field	Description
url	Normalized requested URL
finalUrl	URL after redirects
httpStatus	HTTP status code
title	Page title, if found
markdown	Page body as Markdown: headings, lists, emphasis, tables, code blocks
wordCount	Word count of the extracted body
status	ok, no-content, blocked, disallowed-by-robots, error, or skipped
error	Explanation for an error, block, robots refusal or skipped page
fetchedAt	ISO timestamp of the fetch

no-content means the extractor found no main text, which can happen on navigation or index pages. skipped means a fetched page was left uncharged because the spend cap was reached.

How it crawls

• Checks robots rules before page and sitemap requests and their redirects. It uses the crawler-specific group when present, otherwise User-agent: *, and applies Crawl-delay up to 10 seconds.
• Seeds the URL queue from permitted sitemap files (advertised in robots.txt, or /sitemap.xml), then follows links on the start domains and their subdomains. Sitemap files, sitemap indexes and sitemap redirects must stay on the start URL's exact scheme, host and port. Sitemap and robots requests are separate from the page cap.
• Extracts body text and converts to Markdown using trafilatura (Apache-2.0). The library uses main-text heuristics to reduce navigation, header and footer boilerplate.
• Stops at the page cap or spend cap, whichever comes first.

When to use this

• Feeding a documentation site, blog or wiki into a RAG pipeline or AI assistant.
• Archiving a site as structured Markdown.
• Any job where you want clean text at a known cost rather than raw HTML.

Pricing

US$0.0005 per successfully extracted page (US$0.50 per 1,000 charged pages), plus a US$0.00005 start charge at the supported memory allocations.

Only pages returned with status: ok are charged. Pages that are blocked, disallowed by robots.txt, have no extractable text, or are skipped due to the spend cap are not charged.

The page price applies to ok rows. The total also includes the start charge, so 500 charged pages cost US$0.25005. A US$5.00 run charge limit allows at most 9,999 charged pages at these rates; you must also raise maxPages above its default to attempt that many pages.

Apify's Website Content Crawler offers a browser mode. This actor uses plain HTTP and does not render JavaScript. It uses a fixed price per charged page; no matched cost or quality comparison is claimed. This actor is not affiliated with Apify's Website Content Crawler.

Input

Field	Default	Description
startUrls	(required)	One or more URLs to start crawling from
maxPages	500	Limit page attempts and dataset rows, including free outcomes; enter 1 to 50,000
maxTotalChargeUsd	5.0	Legacy input field; does not set the spend cap. Use Apify run options
sameDomainOnly	true	Follow links on start domains and their subdomains. Keep true for link discovery; false uses only start and sitemap URLs
useSitemap	true	Seed the URL queue from permitted sitemaps on each start URL's exact origin
timeoutSecs	20	HTTP socket timeout in seconds, not a deadline for a complete page attempt
maxConcurrency	2	Concurrent page attempts, from 1 to 20. Higher values can increase memory use

Use with AI agents

Use the input form's Start URLs list. The runtime also accepts plain URL strings or a single URL string when schema validation is bypassed. Crawled page attempts are returned in the default dataset as structured rows. Filter by status: "ok" before chunking the markdown field for a prompt or vector store. Review a sample of the extracted content before ingestion; ok reports successful extraction, not completeness or usefulness for every page.

Robots.txt and terms of service

Before a page or sitemap request, the actor checks cached robots.txt rules for User-agent: MadrascoSiteMarkdownCrawler, falling back to User-agent: * when there is no matching group. The same rules and crawl-delay pacing apply to permitted page and sitemap redirects.

Robots fetch errors, server errors, HTTP 401/403/429 responses, redirects, oversized, compressed or non-UTF8 policies, and HTML returned as policy prevent requests to that origin. Other robots 4xx responses are treated as having no rules. Sitemap directives come from the same cached policy, without a separate robots fetch for discovery. Refused or failed sitemap resources are logged and skipped; the supplied start pages can still run if their robots checks allow them.

For page requests, HTTP 401, 403 and 429 responses, and recognized challenges on HTTP 503 responses, are recorded as blocked; they are not retried or bypassed. Challenge detection is limited and does not identify every captcha or login page.

Check the terms of every site you will crawl before running the actor, including linked subdomains and page-redirect destinations. Robots rules do not grant permission under a site's terms, and the actor does not automatically approve terms. Cross-origin sitemap resources and sitemap redirects are refused; add a destination as a separate start URL only after checking its terms. Page redirects can still reach another origin, subject to that destination's robots and public-host checks.

Limitations

• JavaScript-rendered content: the actor uses plain HTTP, not a browser. Pages that require JavaScript to render their main content will return incomplete or empty Markdown. For those sites, expect no-content results.
• Timeouts: timeoutSecs sets the timeout for HTTP socket operations. A page attempt can take longer because robots fetching, crawl-delay waits, redirects and parsing add time. It is not a whole-page or whole-run deadline; configure the run timeout in Apify run options separately.
• Large sites: the default page cap is 500. Set maxPages and the maximum total charge in Apify run options to match the job; a test run on a sample of URLs is a good first step for unfamiliar sites.
• Large pages: page bodies are limited to about 1 MB before parsing, so larger pages can be truncated.
• Cross-domain discovery: disabling sameDomainOnly stops link discovery; it still allows supplied start URLs and page URLs listed in permitted sitemaps. The same-domain setting does not restrict page-redirect destinations. Sitemap resources themselves must stay on their start origin.
• Extraction quality: main-text extraction can omit snippets or flatten formatting. Short index text can still be ok and chargeable, and separate URLs can return duplicate content. Review a sample before using it in a RAG pipeline.
• Sparse pages: pages with no extracted main text are returned with status: no-content rather than discarded, so you can see what the crawler found.

Data handling

Your run uses Apify-managed input storage, a request queue, a dataset, logs and an OUTPUT summary. These can contain supplied and discovered URLs, fetched text and Markdown, page metadata and errors. URLs and page content can contain personal information; the actor does not redact it.

Use only site content you are authorized to process, and avoid personal or sensitive information. Target sites receive the HTTP requests needed for the crawl. This source has no separate Madrasco data store or reuse pipeline, and it does not implement automatic deletion. Manage retention, access and deletion through your Apify storage settings and account controls.

Publisher

Built and maintained by Madrasco using AI agents. Support: use the Issues tab on this actor's page. Replies may be AI-assisted; a human owner can be reached on request.