RAG Web Browser: URL & Website to Markdown for LLM and RAG

Give it a search query or a list of URLs. Get back clean Markdown, plain text or HTML, ready to chunk and embed. It reads the live web for your LLM the way a search tool inside a chat assistant does, without you running browsers, proxies or a bot-detection arms race.

Point it at one page, a whole site, or a question you want answered from current sources.

What it does

• Search the web, then read the results. Pass a query and it fetches the top N results and converts each page to Markdown. One step instead of a SERP scraper plus a content scraper.
• Convert known URLs in bulk. Pass a list and it processes them in parallel, streaming rows into the dataset as each finishes rather than making you wait for the slowest page.
• Crawl a site. Set a depth and it follows in-content links, stays on the domain unless you say otherwise, and stops at a hard page cap you control.
• Three output formats. Markdown, plain text, and HTML, any combination, on the same row.
• Pay per page, no run fee. Several actors in this category charge a fixed fee every time a run starts, on top of per-result pricing. This one does not, and it does not charge for pages it failed to fetch.

Quick start

Search the web and read the top 5 results:

{
  "query": "best vector database for RAG",
  "maxResults": 5
}

Convert specific pages, with plain text alongside the Markdown:

{
  "urls": [
    "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
    "https://example.com/pricing"
  ],
  "outputFormats": ["markdown", "text"]
}

Crawl a documentation site, capped at 50 pages:

{
  "urls": ["https://docs.example.com/getting-started"],
  "crawlDepth": 2,
  "maxCrawlPages": 50,
  "sameDomainOnly": true,
  "excludeUrlPatterns": ["/tag/", "/author/", "?replytocom"]
}

Input

Field	Type	Default	Description
query	string	—	Search the web and convert the top results. A single URL pasted here is fetched directly instead.
urls	array	—	Specific pages to convert. Bare domains like example.com are accepted. Works alongside query.
maxResults	integer	5	How many search results to fetch. Search mode only.
searchLanguage	string	—	Two-letter code (en, de, nl) to narrow results by language.
outputFormats	array	["markdown"]	Any of markdown, text, html.
includeMetadata	boolean	true	Adds title, description, publishedTime, finalUrl.
minContentLength	integer	200	Pages thinner than this are flagged, not dropped.
crawlDepth	integer	0	0 fetches only what you asked for. 1 also follows links found in those pages. Max 3.
maxCrawlPages	integer	10	Hard ceiling on pages fetched during a crawl. This is your cost cap.
sameDomainOnly	boolean	true	Keep the crawl on the starting domain.
excludeUrlPatterns	array	—	Skip URLs containing any of these substrings. Case-insensitive.
concurrency	integer	10	Pages in parallel. See the note below before raising it.
requestTimeoutSecs	integer	30	Per-attempt timeout for one page.
hedgeDelaySecs	integer	4	If the first extraction backend has not answered in this many seconds, a second is tried in parallel and the first good result wins.
maxRetries	integer	2	Retries per backend for a rate-limited or failed page.

Every field has a default. The minimum viable input is a query or a single URL.

Output

One row per page:

{
  "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
  "finalUrl": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
  "title": "Retrieval-augmented generation",
  "description": "Retrieval-augmented generation (RAG) is a technique that enables large language models to retrieve and incorporate new information from external data sources.",
  "publishedTime": "2023-11-05T13:19:20Z",
  "markdown": "From Wikipedia, the free encyclopedia\n\n**Retrieval-augmented generation**...",
  "wordCount": 2100,
  "length": 14822,
  "provider": "provider_a",
  "fetchedInSeconds": 0.61
}

Search results also carry searchQuery, searchRank, searchTitle, searchSnippet and searchEngine. Crawled pages carry crawlDepth and seedUrl. A page that could not be fetched comes back as a row with an error explaining why, so a partial failure never silently shrinks your dataset.

Results are pushed as each page completes, so you can start reading the dataset while the run is still going. Download as JSON, JSONL, CSV or Excel, stream via the Apify API, or pipe to a webhook, S3, or BigQuery.

Notes worth reading before you scale up

Concurrency. The default of 10 is a measured throughput peak, not a conservative guess. The extraction backends rate-limit per source IP: a 15-page burst at concurrency 10 from a single IP gets throttled on 14 of 15 requests. This Actor handles that by moving a throttled backend onto rotating egress for the rest of the run, which is why a 30-page batch completes in full rather than losing a handful of pages. Pushing concurrency to 25 made throughput worse in testing (3.4 pages/sec at 10, 1.1 at 25). Raise it only if you measure a gain.

HTML output is rendered from the extracted Markdown. It is clean, structured HTML, but it is not the origin page's own markup, which the extraction step has already discarded along with the nav bars and cookie banners.

Crawl links come from the extracted Markdown, so they are in-content links. Navigation chrome that the extractor stripped is not followed. For a RAG crawl that is usually what you want.

Where search results come from. Search mode queries a web index and each row records its searchEngine. It is not Google's index, and this Actor does not claim to be. If you need Google specifically, pair a dedicated SERP Actor with this one in URL mode.

How it compares

	This Actor	Firecrawl	Tavily	Website Content Crawler
Run without your own API key	Yes	No, needs a Firecrawl key	No, needs a Tavily key	Yes
Search query as input	Yes	Yes	Yes	No, URL only
Markdown, text and HTML	Yes	Yes	Markdown	Yes
Crawl with a hard page cap	Yes	Yes	Limited	Yes
Per-run start fee	None	Monthly credits	Monthly credits	Platform compute
Charges for failed pages	No	Varies	Varies	Compute is billed regardless

If you already run jobs on Apify, this keeps web-to-Markdown in the same account, dataset format and billing as everything else.

FAQ

How is this different from a plain URL to Markdown converter?

A converter takes one URL and returns one document. This takes a question and returns the current web's answer to it as documents, or takes a site and returns the whole readable surface of it. Search mode and crawl mode are the difference, and both feed the same clean row shape.

Why does my page come back with a contentWarning instead of content?

It has content, just less than minContentLength (200 characters by default). Genuinely short pages, stubs and brief news items are returned flagged rather than thrown away. Lower minContentLength to silence it, or raise it to be stricter about thin pages.

What happens when a page is behind a bot wall?

The Actor detects interstitials (Cloudflare challenges, "enable JavaScript" stubs, captcha pages) and refuses to pass them off as content. It tries the other extraction backend, and if the wall holds, the row comes back with an error naming the reason. You are not charged for it.

Can I use it as an HTTP endpoint instead of starting a run?

Yes. Standby mode serves the same pipeline over HTTP, so an agent can request a page mid-conversation without paying Actor start latency each call. Pass the same fields as query parameters, for example ?query=vector+databases&maxResults=3 or ?urls=https://a.com&urls=https://b.com.

How much does a run cost?

Pay-per-event: you are billed per page successfully converted, with no monthly minimum and no fee for starting a run. Failed pages and pages dropped by a plan limit are not billed. A 100-page crawl costs 100 page events. See the Store page for the current per-page price.

Can I schedule it?

Yes, via Apify's built-in scheduler. A common setup is a nightly crawl of a docs site with crawlDepth set, diffing against the previous run to re-embed only what changed.

Is there a free tier?

Free-plan Apify accounts get 10 results per run. Paid plans have no cap. The limit is per run, so free users can still evaluate every feature.