Markdown Website Content Scraper

Turn any website into clean, structured Markdown for RAG pipelines, AI agents, vector stores and knowledge bases — without writing selectors. This markdown scraper takes one or more URLs and returns distraction-free page text: headings, paragraphs, lists and links kept; navigation chrome, cookie banners, scripts and boilerplate stripped. The Brainiall Web engine honors robots.txt and crawl-delay, does not bypass bot detection, and refuses internal or private addresses. If a page is disallowed or empty, it is skipped and not charged.

What it does

• Single-page conversion — default mode. Each start URL becomes one dataset item of Markdown.
• Bounded site crawl — raise Max pages per site above 1. Each start URL is an entry point; the Actor follows internal links up to the page cap and depth cap. Every page is one item.
• Page metadata — title, language, description, word count and HTTP status travel with the Markdown.
• Optional links — ask for the hyperlinks found on each page (single-page mode).

Give the Actor a public https:// URL. You get Markdown sized for retrieval, not a dump of raw HTML. Failed fetches, empty bodies and robots.txt blocks never create a page-converted event.

Why this Actor

• Markdown, not HTML. RAG builders ingest docs sites, blogs and product pages into a vector store. Markdown chunks and embeds more cleanly than raw markup.
• Bounded crawl with a hard cap. Raise maxPagesPerUrl only when you need a section, not an unbounded spider. Depth is capped at 4 hops.
• robots.txt is honored. The engine does not bypass bot detection. Internal, loopback and link-local addresses are refused (SSRF-guarded). You stay inside public pages.

Input

Field	Type	Default
startUrls	array of {url}	— (required)
maxPagesPerUrl	integer	1 (1 = that URL only; 2–25 = crawl)
maxDepth	integer	0 (0–4 hops in crawl mode)
includeLinks	boolean	false

{
  "startUrls": [
    { "url": "https://docs.example.com" },
    { "url": "https://blog.example.com/post" }
  ],
  "maxPagesPerUrl": 10,
  "maxDepth": 2,
  "includeLinks": false
}

Prefill a public https://example.com URL for the first run.

Output

One dataset item per page:

{
  "url": "https://example.com/",
  "requested_url": "https://example.com",
  "title": "Example Domain",
  "markdown": "# Example Domain\n\nThis domain is for use in documentation examples...",
  "word_count": 19,
  "metadata": { "description": "", "lang": "en" },
  "http_status": 200
}

Export JSON, CSV or Excel, or read the default dataset through the Store API. Downstream you can chunk markdown for retrieval.

Pricing

Pay-per-event. Current prices (do not assume other SKUs):

Event	USD
page-converted (primary)	$0.002 per page successfully converted to Markdown

Worked cost example:

• 50 pages → $0.10
• 500 pages → $1.00
• A docs section of 25 pages in one crawl → $0.05

Pages that fail, return no extractable content, or are blocked by robots.txt are never charged. No subscription and no monthly minimum. Set BRAINIALL_API_KEY as an Actor secret (key from https://app.brainiall.com). Keys are never shipped in the listing.

Use cases

1. RAG pipelines that turn a docs site or blog into Markdown documents ready to chunk and embed.
2. Agent context — fetch a public page as token-efficient Markdown instead of noisy HTML.
3. Content archives — migrate a public site section to portable Markdown with title, language and word count attached.

FAQ

How is this different from saving a page as HTML? HTML keeps chrome, scripts and layout. This Actor returns main content as Markdown sized for RAG and agents.

Will I be banned from the target site? The Brainiall Web engine does not bypass bot detection and respects robots.txt. Stay within the page and depth caps. You are responsible for the sites you fetch.

Can I crawl an entire large site? Not in one run. Cap is 25 pages per start URL. Split sections across runs and deduplicate by url.

Limits

• maxPagesPerUrl: 1–25 per start URL.
• maxDepth: 0–4 in crawl mode.
• HTTP(S) public URLs only. Internal, loopback and link-local addresses are refused.
• robots.txt Disallow and crawl-delay are honored. Sites that block automated access stay blocked.
• Charge is per successful Markdown page, not per start URL and not per failed fetch.

---

Built and maintained by Brainiall — production speech, document and web engines.