# Search & URL Markdown Extractor for LLMs (`vexaloom-labs/rag-web-browser`) Actor

Turn a search query or a list of URLs into clean Markdown, plain text, or HTML pages, with filters for domain, keyword, and minimum content length. No login or cookies needed.

- **URL**: https://apify.com/vexaloom-labs/rag-web-browser.md
- **Developed by:** [vexaloom-labs](https://apify.com/vexaloom-labs) (community)
- **Categories:** AI, Agents, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.49 / 1,000 results

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Web Search & RAG Markdown Extractor for LLMs

Give your LLM or RAG pipeline fresh web content in one run: search the web, read specific pages, or crawl a small site — and receive clean Markdown, plain text, HTML, structured links, and citation metadata. The actor starts its search on Google and automatically falls back to Bing or DuckDuckGo when an engine is unavailable, so a query keeps returning results even when one search provider is having a bad day.

### What this actor does

- **Three intents in one actor:** search by keyword (`query` / `queries`), read exact pages (`startUrls`), or crawl a site by following links within the same domain (`maxCrawlDepth`)
- **Resilient search:** Google first, with automatic Bing and DuckDuckGo fallbacks; every result keeps its title, snippet, displayed address, position, and the engine that found it
- **RAG-ready output:** clean Markdown, plain text, or HTML, plus every link on the page as structured `{url, text}` pairs and optional heading-aware chunks that never cut code blocks or tables in half
- **Citation metadata:** canonical URL, author, published date, keywords, social image, and site name — each included when the page actually provides it
- **Token control:** `maxContentLength` caps the returned content per page; `includeMetadata` and `chunkMarkdown` are opt-in extras
- **Filters:** domain allowlist/blocklist, minimum text length, required keywords, and custom CSS selectors to strip or keep
- **No login, cookies, or API keys required.** Fields a page doesn't have are omitted from its record rather than returned empty

### Output per page

- `url` — the page's address; `loadedUrl` — where the fetch ended after redirects; `redirectedUrl` — the final address when it differs from `url`
- `domain` — host name of the page
- `title`, `description`, `languageCode`
- `markdown`, `text`, `html` — included only for the formats you select
- `links` — every http(s) link found on the page as `{url, text}`, when `links` is selected
- `chunks` — heading-aware content pieces with `index`, `text`, `headingPath`, `charCount`, when `chunkMarkdown` is on
- `wordCount`, `readingTimeMinutes` (at 220 words per minute), `contentLength` (characters of the emitted content), `contentHash` (SHA-256 for change detection)
- `canonicalUrl`, `author`, `publishedDate` (ISO 8601), `keywords`, `ogImage`, `siteName` — page metadata, when present
- `searchRank` — 1-based position of the page in the search results (search mode only)
- `searchResult` — `{url, title, snippet, displayedUrl, position, engine}` for pages that came from a search query
- `query` — the search query that produced the page; `engine` — which engine found it
- `crawlDepth` — how many link hops from a start URL (crawl mode only)
- `fetchError` — why a page could not be fetched, on metadata-only records (see below)
- `httpStatusCode` — the status the page answered with; pages that answer with an error status but still return content are kept like any other page
- `loadedTime` — seconds spent fetching the page
- `recordType: "page"`, `scrapedAt`

### Input

| Field | Type | Default | Description |
|---|---|---|---|
| `query` | string | `what is retrieval augmented generation` | Search terms, or a single page URL to read directly |
| `queries` | array | `[]` | More queries for batch runs; a URL here is fetched directly |
| `startUrls` | array | `[]` | Pages to read; with `maxCrawlDepth` > 0 these become crawl seeds |
| `searchEngine` | string | `auto` | `auto` / `google` / `bing` / `duckduckgo` |
| `searchLanguage` | string | `auto` | Language bias for results |
| `searchCountry` | string | `auto` | Country bias for results |
| `maxResults` | integer | `3` | Search results fetched per query (1–250; > 10 needs multiple result pages) |
| `maxItems` | integer | `0` | Global page cap for the whole run (0 = no cap) |
| `outputFormats` | array | `["markdown"]` | Any of `markdown`, `text`, `html`, `links` |
| `removeElementsCssSelector` | string | `nav, footer, aside, script, style, .ad, .advertisement, [aria-hidden='true']` | Elements removed before extraction |
| `keepElementsCssSelector` | array | `[]` | Keep only content inside these selectors |
| `htmlTransformer` | string | `readable-text` | `readable-text` (main article) or `none` (full page) |
| `maxContentLength` | integer | `0` | Truncate content per page (0 = full content) |
| `includeMetadata` | boolean | `true` | Include author, dates, canonical URL, and friends |
| `chunkMarkdown` | boolean | `false` | Emit heading-aware `chunks` |
| `chunkMaxChars` | integer | `2000` | Target chunk size (200–10000) |
| `chunkOverlapChars` | integer | `200` | Overlap between chunks (0–1000) |
| `maxCrawlDepth` | integer | `0` | Follow same-domain links up to this depth (0 = off) |
| `maxCrawlPages` | integer | `25` | Page cap in crawl mode |
| `domainAllowlist` | array | `[]` | Only keep pages whose host contains one of these |
| `domainBlocklist` | array | `[]` | Drop pages whose host contains one of these |
| `minTextLength` | integer | – | Drop pages with less extracted text than this |
| `excludeContentSelectors` | array | `[]` | Extra CSS selectors to strip |
| `keywordsAnyOf` | array | `[]` | Keep only pages containing at least one keyword |
| `emitFailedResults` | boolean | `true` | Emit a metadata-only record with `fetchError` for pages that could not be fetched |

Older tasks may still carry tuning fields such as `scrapingTool`, `requestTimeoutSecs`, `desiredConcurrency`, `maxRequestRetries`, `dynamicContentWaitSecs`, `removeCookieWarnings`, `useApifyProxy`, or `debugMode`. They are accepted and ignored: request timeouts, retries, rendering, proxy rotation, and diagnostics are decided automatically by the actor.

#### Example: Google a question and keep only substantial pages

```json
{
  "query": "vector database comparison 2026",
  "maxResults": 5,
  "outputFormats": ["markdown", "links"],
  "minTextLength": 1000,
  "domainBlocklist": ["pinterest.com"]
}
```

#### Example: batch queries with citation metadata

```json
{
  "queries": ["retrieval augmented generation evaluation", "chunking strategies for RAG"],
  "maxResults": 3,
  "includeMetadata": true,
  "chunkMarkdown": true,
  "chunkMaxChars": 1500
}
```

#### Example: read exact pages

```json
{
  "query": "",
  "startUrls": [
    "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
    "https://docs.langchain.com/oss/python/langchain/retrieval"
  ],
  "outputFormats": ["markdown", "text"],
  "htmlTransformer": "readable-text"
}
```

#### Example: crawl a documentation section

```json
{
  "query": "",
  "startUrls": ["https://docs.crawl4ai.com/core/"],
  "maxCrawlDepth": 2,
  "maxCrawlPages": 20,
  "outputFormats": ["markdown", "links"]
}
```

### Use cases

- **RAG ingestion** — pull the current top results for a topic and hand the Markdown (or ready-made chunks) straight to your embedding step
- **Agent context gathering** — give an AI agent a fresh, citation-bearing view of the web without building a search stack
- **Citation enrichment** — resolve each reference URL from a document into clean text plus author and publication date
- **Documentation snapshots** — crawl a docs section and keep `contentHash` values to detect what actually changed between runs
- **Competitive monitoring** — schedule queries with a `domainAllowlist` to watch specific companies or publications
- **Due-diligence research** — batch a list of questions, filter to long-form sources with `minTextLength`, and export one tidy dataset

### FAQ

#### Do I need to log in or provide cookies?

No. Every search and page request is anonymous: no account, no cookies, no API keys.

#### How does the actor choose a search engine?

`searchEngine: auto` (the default) tries Google first. If Google cannot be reached or returns no readable results, the actor automatically tries Bing and then DuckDuckGo. You can also pin one engine explicitly — useful when you want strictly one provider for reproducibility.

#### Do I need to set up a proxy?

No. The actor starts with direct requests and only switches connections when a page actually blocks it. Everything is handled automatically; there is nothing to configure. Blocked pages get a per-request escalation (datacenter rotation, then a rendering relay) before the actor gives up on them, and pages that still fail are reported through `fetchError`.

#### Why do some pages have fewer fields?

Metadata like `author` or `publishedDate` is emitted only when the page itself declares it. If a page has no publication date, the field is simply absent from that record — never `null`, never an empty string. The same applies to `chunks` (only with `chunkMarkdown`) and `links` (only when selected).

#### Can the actor read JavaScript-heavy pages?

Yes, automatically. The actor first requests every page with plain HTTP; when a page answers with a JavaScript challenge, a 202-style JS stub, or an empty app shell, it renders that page in a real browser (dismissing common consent banners and expanding collapsed sections) before extraction. No setting to change — the escalation is decided per page.

#### How do I control the size of the output?

Set `maxContentLength` to cap characters per page, or enable `chunkMarkdown` to get evenly sized pieces. Both help keep embedding and token costs predictable.

#### What happens when a page can't be fetched?

`maxResults` counts *search results*, so every requested result yields one record. When a site refuses the fetch (for example a paywall or a bot challenge that survives the actor's retry ladder), the record still carries `url`, `title`, `description`, `searchResult` and `searchRank`, plus a `fetchError` note explaining what happened — with no fabricated page content. Set `emitFailedResults` to `false` if you prefer the dataset to contain successfully fetched pages only.

#### What happens if a search returns nothing?

The run finishes with a status message explaining what happened and still succeeds — no placeholder rows. If every provider fails for a query, the log lists the reason.

#### Are file downloads and AI summaries supported?

No — this actor returns text and metadata for pages. It skips media and document files (video, audio, images, PDFs) because they carry no page text, and it does not call any paid AI service.

#### Is this affiliated with Google, Bing, or DuckDuckGo?

No. This is an independent, third-party actor. It reads public search result pages and public web pages; it is not operated by or affiliated with any search provider.

# Actor input Schema

## `query` (type: `string`):

What to look up, or a single web page URL to read. Examples: `vector database comparison`, `https://en.wikipedia.org/wiki/Retrieval-augmented_generation`. When `startUrls` or `queries` are also set, everything is processed in one run.

## `queries` (type: `array`):

Additional queries for batch research runs. Each query is searched and its top results are fetched. A query that is itself a URL is fetched directly.

## `startUrls` (type: `array`):

Pages to read directly. With `maxCrawlDepth` above 0, the actor walks links found on these pages within the same website instead.

## `searchEngine` (type: `string`):

Which search engine to use. `Automatic` tries Google first and silently falls back to Bing, then DuckDuckGo when an engine is blocked — recommended.

## `searchLanguage` (type: `string`):

Language for search results. `Automatic` lets each engine choose based on the query.

## `searchCountry` (type: `string`):

Country bias for search results. `Automatic` means no country bias.

## `maxResults` (type: `integer`):

How many top organic results to fetch and emit for each search query. Values above 10 need multiple search-result pages (slower). With `Emit results for pages that could not be fetched` on, each of the requested results yields one record — a page record when the fetch succeeds, or a metadata record with `fetchError` when it does not.

## `maxItems` (type: `integer`):

Overall cap on how many pages the run may emit across all queries, URLs, and crawled pages. 0 means no global cap.

## `outputFormats` (type: `array`):

Which representations to include per page, in any combination. `Links` adds every link found on the page as structured `{url, text}` pairs.

## `removeElementsCssSelector` (type: `string`):

Comma-separated CSS selectors removed before extraction (navigation, footers, ads, cookie bars). Set a non-matching selector such as `dummy_keep_everything` to keep everything.

## `keepElementsCssSelector` (type: `array`):

When set, only content inside these selectors is kept — everything else is dropped before extraction.

## `htmlTransformer` (type: `string`):

`Readable content` finds the article body and drops navigation and boilerplate. `None` keeps the full cleaned page.

## `maxContentLength` (type: `integer`):

Truncate the emitted text and Markdown of each page to this many characters. 0 returns the full content. Useful for controlling embedding cost.

## `includeMetadata` (type: `boolean`):

Add canonical URL, author, published date, keywords, social image, and site name (each only when the page provides it).

## `chunkMarkdown` (type: `boolean`):

Emit a `chunks` array per page — heading-aware Markdown chunks ready for embedding, never split inside code blocks or tables.

## `chunkMaxChars` (type: `integer`):

Target size of each chunk when `chunkMarkdown` is on. Code blocks and tables are kept whole even when larger.

## `chunkOverlapChars` (type: `integer`):

Approximate overlap between consecutive chunks when `chunkMarkdown` is on, so context is not lost at chunk boundaries.

## `maxCrawlDepth` (type: `integer`):

0 reads only the start URLs. 1+ follows links found on those pages, staying on the same website, up to this depth.

## `maxCrawlPages` (type: `integer`):

Safety cap on pages visited in crawl mode.

## `domainAllowlist` (type: `array`):

Keep only pages whose host contains one of these substrings (case-insensitive).

## `domainBlocklist` (type: `array`):

Drop pages whose host contains one of these substrings.

## `minTextLength` (type: `integer`):

Drop pages with fewer than this many characters of extracted text. Useful for skipping thin or placeholder pages.

## `excludeContentSelectors` (type: `array`):

Additional CSS selectors removed before extraction, on top of `Remove elements`.

## `keywordsAnyOf` (type: `array`):

Keep only pages whose title or content contains at least one of these keywords (case-insensitive).

## `emitFailedResults` (type: `boolean`):

When a search result page cannot be fetched after the full retry ladder, still emit one record for it with `url`, `title`, `description`, `searchResult`, `searchRank` and a `fetchError` note (no fabricated content). Turn off to skip failed pages entirely.

## Actor input object example

```json
{
  "query": "what is retrieval augmented generation",
  "queries": [],
  "startUrls": [],
  "searchEngine": "auto",
  "searchLanguage": "auto",
  "searchCountry": "auto",
  "maxResults": 3,
  "maxItems": 0,
  "outputFormats": [
    "markdown"
  ],
  "removeElementsCssSelector": "nav, footer, aside, script, style, .ad, .advertisement, [aria-hidden='true']",
  "keepElementsCssSelector": [],
  "htmlTransformer": "readable-text",
  "maxContentLength": 0,
  "includeMetadata": true,
  "chunkMarkdown": false,
  "chunkMaxChars": 2000,
  "chunkOverlapChars": 200,
  "maxCrawlDepth": 0,
  "maxCrawlPages": 25,
  "domainAllowlist": [],
  "domainBlocklist": [],
  "excludeContentSelectors": [],
  "keywordsAnyOf": [],
  "emitFailedResults": true
}
```

# Actor output Schema

## `pages` (type: `string`):

Dataset of fetched pages with content and metadata, ready for LLM and RAG pipelines.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "query": "what is retrieval augmented generation",
    "queries": [],
    "startUrls": [],
    "searchEngine": "auto",
    "searchLanguage": "auto",
    "searchCountry": "auto",
    "maxResults": 3,
    "maxItems": 0,
    "outputFormats": [
        "markdown"
    ],
    "removeElementsCssSelector": "nav, footer, aside, script, style, .ad, .advertisement, [aria-hidden='true']",
    "keepElementsCssSelector": [],
    "htmlTransformer": "readable-text",
    "maxContentLength": 0,
    "includeMetadata": true,
    "chunkMarkdown": false,
    "chunkMaxChars": 2000,
    "chunkOverlapChars": 200,
    "maxCrawlDepth": 0,
    "maxCrawlPages": 25,
    "domainAllowlist": [],
    "domainBlocklist": [],
    "excludeContentSelectors": [],
    "keywordsAnyOf": [],
    "emitFailedResults": false
};

// Run the Actor and wait for it to finish
const run = await client.actor("vexaloom-labs/rag-web-browser").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "query": "what is retrieval augmented generation",
    "queries": [],
    "startUrls": [],
    "searchEngine": "auto",
    "searchLanguage": "auto",
    "searchCountry": "auto",
    "maxResults": 3,
    "maxItems": 0,
    "outputFormats": ["markdown"],
    "removeElementsCssSelector": "nav, footer, aside, script, style, .ad, .advertisement, [aria-hidden='true']",
    "keepElementsCssSelector": [],
    "htmlTransformer": "readable-text",
    "maxContentLength": 0,
    "includeMetadata": True,
    "chunkMarkdown": False,
    "chunkMaxChars": 2000,
    "chunkOverlapChars": 200,
    "maxCrawlDepth": 0,
    "maxCrawlPages": 25,
    "domainAllowlist": [],
    "domainBlocklist": [],
    "excludeContentSelectors": [],
    "keywordsAnyOf": [],
    "emitFailedResults": False,
}

# Run the Actor and wait for it to finish
run = client.actor("vexaloom-labs/rag-web-browser").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "query": "what is retrieval augmented generation",
  "queries": [],
  "startUrls": [],
  "searchEngine": "auto",
  "searchLanguage": "auto",
  "searchCountry": "auto",
  "maxResults": 3,
  "maxItems": 0,
  "outputFormats": [
    "markdown"
  ],
  "removeElementsCssSelector": "nav, footer, aside, script, style, .ad, .advertisement, [aria-hidden='\''true'\'']",
  "keepElementsCssSelector": [],
  "htmlTransformer": "readable-text",
  "maxContentLength": 0,
  "includeMetadata": true,
  "chunkMarkdown": false,
  "chunkMaxChars": 2000,
  "chunkOverlapChars": 200,
  "maxCrawlDepth": 0,
  "maxCrawlPages": 25,
  "domainAllowlist": [],
  "domainBlocklist": [],
  "excludeContentSelectors": [],
  "keywordsAnyOf": [],
  "emitFailedResults": false
}' |
apify call vexaloom-labs/rag-web-browser --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,vexaloom-labs/rag-web-browser"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/svDKrzKCcdfn7GHRJ/builds/gPqkQf6syKae3DRXC/openapi.json