RAG Web Browser: Website to Markdown and URL to Markdown for LLM & RAG

RAG Web Browser converts any website to Markdown for LLM and retrieval augmented generation (RAG) workflows. Give it a search query, a list of URLs, or both, and receive one clean dataset row per page: readable Markdown by default, with optional plain text, rendered HTML, page metadata and link crawling. Every row keeps the original URL, so the documents you chunk, embed and index can always be cited back to their source.

What this website to Markdown tool does

Most web pages are built for browsers, not for language models. Navigation menus, cookie banners, scripts and layout markup waste tokens and confuse retrieval. RAG Web Browser fetches each page, extracts the readable document and returns it as Markdown with headings, lists, links and tables intact. You can use it in three ways:

• URL to Markdown: pass the pages you already know, from one URL to a list of up to 1000.
• Web search to Markdown: pass a query; the top results are discovered for you and converted, with their search rank, title and snippet attached.
• Website to Markdown crawl: start from one or more pages and follow links one to three hops deep, within a page ceiling you control.

The output is a plain Apify dataset, so you can download it as JSON, CSV or Excel, read it through the Apify API, or hand it straight to LangChain, LlamaIndex, a vector database loader or your own embedding script.

Website to Markdown quick start

Use a search query when you want relevant pages discovered for you. Use URLs when you already know which pages you need. You can combine both in one run: search results become starting pages, followed by the explicit URLs. Repeated starting pages are processed once. A complete URL pasted into query, without a separate URL list, is fetched directly.

This URL input was run successfully:

{
  "urls": [
    "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
    "example.com",
    "https://docs.python.org/3/library/json.html",
    "https://www.iana.org/this-page-does-not-exist-rag-baseline"
  ],
  "maxResults": 5,
  "searchLanguage": "",
  "outputFormats": [
    "markdown"
  ],
  "includeMetadata": true,
  "minContentLength": 200,
  "crawlDepth": 0,
  "maxCrawlPages": 10,
  "sameDomainOnly": true,
  "concurrency": 10,
  "requestTimeoutSecs": 30,
  "hedgeDelaySecs": 4,
  "maxRetries": 2
}

The urls field also accepts one string. Bare domains receive an https:// prefix. Empty entries are removed and a run accepts up to 1000 explicit URLs. For search mode, maxResults defaults to 5 and accepts 1 through 50. searchLanguage defaults to an empty string; setting a language such as en narrows the search by language. It does not select a country.

Input and options

All input fields are optional individually, but a run needs either query or urls. The existing names and defaults remain supported. outputFormats defaults to ["markdown"]; choose any combination of markdown, text and html. An empty format list uses Markdown. includeMetadata defaults to true and controls whether the page title, description, published time and final URL are included. Unrequested formats and disabled metadata are absent from the row, rather than present with a null value.

minContentLength defaults to 200 characters of readable text. A clean page shorter than the threshold can still be returned, with a contentWarning. This makes the difference between short but useful content and a failed fetch visible. Increasing the threshold adds warnings to thin pages; it is not a guarantee that every returned document has a particular length.

crawlDepth defaults to 0, meaning only starting pages are fetched. Depth 1 also follows links from those pages; depths 2 and 3 continue further. maxCrawlPages defaults to 10 and accepts up to 200. It includes starting pages, so it is the total crawl ceiling. sameDomainOnly defaults to true. excludeUrlPatterns contains case insensitive URL substrings to skip when following links. Links come from extracted page content and may omit navigation links removed during extraction.

This crawl input was also run successfully:

{
  "query": "crawlee python quick start",
  "urls": [
    "https://crawlee.dev/python/docs/quick-start"
  ],
  "maxResults": 3,
  "searchLanguage": "en",
  "outputFormats": [
    "markdown",
    "text",
    "html"
  ],
  "includeMetadata": true,
  "minContentLength": 500,
  "crawlDepth": 1,
  "maxCrawlPages": 8,
  "sameDomainOnly": true,
  "excludeUrlPatterns": [
    "/api/",
    "/blog/"
  ],
  "concurrency": 10,
  "requestTimeoutSecs": 30,
  "hedgeDelaySecs": 4,
  "maxRetries": 2
}

Transport options retain their original validation ranges: concurrency defaults to 10, requestTimeoutSecs to 30, hedgeDelaySecs to 4 and maxRetries to 2. Requests are processed in bounded batches. Values above the effective per-batch parallelism or page timeout are accepted, while the actual work remains within the request budget. The effective hedge delay is at most 4 seconds so other extraction attempts can still run within the page budget. A slow page may become a free error row. Large runs automatically continue across batches; regular Actor users do not need to manage continuation fields.

Output: Markdown page content and metadata

Each row always has url, provider, length, wordCount and fetchedInSeconds. The provider is an anonymous label such as provider_a, not a source service name. A successful row adds the requested formats and, if enabled, title, description, publishedTime and finalUrl. Published times retain the text supplied by the source and may be empty. HTML is rendered from the extracted Markdown, so it represents the readable document rather than the original source markup.

This row comes from a real search run for best vector database for RAG with searchLanguage set to en. The markdown value is shortened here after its first two paragraphs; in the dataset it holds the full article of 1975 words:

{
  "url": "https://www.braintrust.dev/articles/best-vector-databases-for-rag-2026",
  "provider": "provider_a",
  "length": 16026,
  "wordCount": 1975,
  "fetchedInSeconds": 0.41,
  "title": "Best vector databases for RAG in 2026",
  "description": "A vector database stores embeddings and returns the chunks a RAG app uses as context. When a user asks a question, the app embeds the query, searches for the closest stored chunks, and passes those chunks to the model for answer generation.",
  "publishedTime": "11 July 2026",
  "finalUrl": "https://www.braintrust.dev/articles/best-vector-databases-for-rag-2026",
  "markdown": "TL;DR\n\nA vector database stores embeddings and returns the chunks a RAG app uses as context. When a user asks a question, the app embeds the query, searches for the closest stored chunks, and passes those chunks to the model for answer generation.",
  "searchQuery": "best vector database for RAG",
  "searchRank": 2,
  "searchTitle": "Best vector databases for RAG in 2026 - Articles - Braintrust",
  "searchSnippet": "Compare the top vector databases for RAG in 2026: Pinecone, Weaviate, Qdrant, Chroma, and Turbopuffer. Learn how each handles hosting, hybrid search, metadata filtering, and storage architecture, and how to evaluate retrieval quality.",
  "searchEngine": "web",
  "fetchedAt": "2026-10-02T21:51:03.767Z",
  "source": "rag-web-browser",
  "fetched_at": "2026-10-02T21:51:03.767Z",
  "word_count": 1975,
  "final_url": "https://www.braintrust.dev/articles/best-vector-databases-for-rag-2026"
}

The following fields are copied from a real page response with metadata disabled and text requested:

{
  "url": "https://www.iana.org/help/example-domains",
  "provider": "provider_a",
  "length": 868,
  "wordCount": 114,
  "fetchedInSeconds": 0.3,
  "text": "A number of domains such as example.com and example.org are maintained for documentation purposes. These domains may be used as illustrative examples in documents without prior coordination with us. They are not available for registration or transfer.\n\nWe provide a web service on the example domain hosts to provide basic information on the purpose of the domain. These web services are provided as best effort, but are not designed to support production applications. We strongly recommend you do not design applications that require the example domains to have operating HTTP service.\n\nFurther Reading\n\n IANA-managed Reserved Domains\n Special-Use Domain Names (RFC 6761)\n* Reserved Top Level DNS Names (RFC 2606)\n\nLast revised 5 days ago."
}

Search rows also include searchQuery, searchRank, searchTitle, searchSnippet and searchEngine. The engine field is web. Use searchRank if you want search order: pages finish at different times, and row order follows completion. Crawl rows include crawlDepth and seedUrl, allowing you to identify the starting page and the number of hops. New fields fetchedAt and fetched_at provide a UTC collection timestamp; source, final_url and word_count make the same rows usable in shared data contracts while retaining the existing names.

A page that fails extraction produces a row with provider: null, zero length and wordCount, an error and providersTried. Other successfully fetched pages remain available. This is the real error row for a host that does not exist, from the same run as the text example above:

{
  "url": "https://rag-baseline-unresolvable-host.invalid/page",
  "provider": null,
  "length": 0,
  "wordCount": 0,
  "fetchedInSeconds": 4.12,
  "error": "provider_a: HTTP 422; provider_b: HTTP 403; provider_c: HTTP 404",
  "providersTried": [
    "provider_a",
    "provider_b",
    "provider_c"
  ],
  "fetchedAt": "2026-10-02T21:51:00.269Z",
  "source": "rag-web-browser",
  "fetched_at": "2026-10-02T21:51:00.269Z",
  "word_count": 0
}

The key value store contains a summary with rowsEmitted, rowsProduced, searchResults, crawlPages, errorCount and outputFormats. When errors occur, the errors record contains code and message pairs. Invalid input or an empty search finishes as a successful empty run, with the reason available in the records.

Use cases

• RAG knowledge bases: convert documentation, help centres and product pages to Markdown, then chunk by heading and embed. The url and title fields become citation metadata.
• AI agents that browse: give an agent one tool that searches the web and returns readable page content in a single call, instead of raw HTML it has to clean itself.
• Fresh context for answers: run a query on a schedule and feed the newest pages to a model, with fetchedAt showing when each document was collected.
• Documentation refreshes: re-run a fixed URL list and compare length, wordCount or the content itself to decide which documents need a new embedding.
• Content research: collect the top results for a topic with their search rank, snippet and full text in one dataset.

Comparison: how this compares with other ways to turn a website into Markdown

Approach	Search	Crawl links	Batch of URLs	Metadata per page	Runs without your own server
RAG Web Browser	yes	up to 3 hops	up to 1000	title, description, published time, final URL, search rank	yes
Copy and paste or a browser extension	no	no	no	manual	yes, one page at a time
An HTML to Markdown library in your code	no	you build it	you build it	you build it	no
A single-URL converter	no	no	one page per call	varies	yes

An HTML to Markdown library is the right choice when you already have clean HTML. RAG Web Browser is for the step before that: getting the page, removing everything that is not the document, handling pages that fail, and returning rows that are ready for an index. Failed pages are reported per row instead of failing the run, so a batch of 500 URLs with three broken links still delivers 497 documents.

Pricing and free tier

The price is $0.002 per successfully converted URL, using the scraped event. There is no run start fee. Only rows delivered without an error are charged. A short page returned with a contentWarning is still a converted page and counts as one event. Failed pages and an empty run incur no page events. For example, the real URL input above returned four page rows and four scraped events; the two-URL error test returned two rows with one successful page and one event. Converting 1000 pages therefore costs $2.

Free-plan Apify accounts receive up to 10 rows per run. Error rows also count toward that delivery limit, but remain free. Paid accounts can use the full input limits. The cap applies per run; smaller jobs can be scheduled separately when appropriate.

FAQ

How do I convert a website to Markdown?

Put the page addresses in urls and start the run. For a whole section of a site, add one starting page, set crawlDepth to 1 or 2 and choose a maxCrawlPages ceiling. Each page arrives as its own row with a markdown field.

Can I provide URLs without running a search?

Yes. URL-only mode skips search entirely. It is useful for a known document list, documentation refreshes, and processing URLs collected by another Actor. You can request text alongside Markdown or disable metadata if your downstream pipeline does not need it.

Is this the same as an HTML to Markdown converter?

It includes that step, but starts earlier. A converter needs HTML you already have; this Actor fetches the page, keeps only the readable document and then writes the Markdown. If you need HTML as well, add html to outputFormats.

How do I interpret a content warning?

A warning means the readable page content was shorter than your threshold. It still contains extracted content and is delivered as a successful conversion. Review that content before indexing it if your workflow needs a minimum document size. A fetch error has a separate error field and is not billed.

Does the crawl discover every link on a website?

It follows HTTP and HTTPS links found in extracted content, subject to your depth, domain, exclusion and total page limits. Binary assets and fragment-only links are skipped. Navigation removed during extraction cannot be discovered from that document, so provide additional starting URLs when coverage requires them.

Can I use it from LangChain, LlamaIndex or my own code?

Yes. Start the Actor through the Apify API or an Apify client, then read the dataset items. Each item is a flat JSON object, so the markdown field maps directly to a document body and the other fields to document metadata.

Can I schedule repeated runs?

Yes. Use Apify schedules with the same input and store the dataset between runs. Compare content or collection timestamps in your own indexing workflow when deciding which documents need another embedding. This Actor returns page content; it does not maintain your vector database or perform the comparison for you.

Is an empty search a failed job?

No. A search without organic results finishes successfully with an empty dataset and no page charges. The summary and errors records explain the outcome. This also allows scheduled workflows to handle missing results without treating the run as a crash.