# Web Search & Page Markdown for AI Agents (`emttn/web-search-markdown`) Actor

Web search plus each result page, or your own URLs, as clean markdown in one call. For AI agents and RAG, payable per request via x402. Charged per search with results and per page delivered; failed pages are free. Each search also runs an Apify search actor, billed by Apify to your account.

- **URL**: https://apify.com/emttn/web-search-markdown.md
- **Developed by:** [Frameskim Tools](https://apify.com/emttn) (community)
- **Categories:** AI, Agents, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Web Search & Page Markdown for AI Agents

Give it a question, get back the top web pages already turned into clean markdown, ranked as the search engine ranked them. Built for AI agents, RAG pipelines and assistants that need to read the web in one call and pay per request.

- Search one or many queries and read the top 1 to 10 result pages of each as markdown.
- Turn your own list of URLs into markdown, no search needed.
- Main content only: menus, footers and cookie banners are left out; headings, lists, tables and links stay.
- Frameskim charges only per search with results and per page delivered; failed, blocked or empty pages are free. Each search also runs Apify's Google Search Results Scraper, billed by Apify to your account.
- Payable by AI agents per request: pay-per-event only, limited permissions, no Standby.

### What you get

One dataset item per result page or URL:

```json
{
  "source": "search",
  "query": "how does robots.txt work",
  "rank": 1,
  "url": "https://www.rfc-editor.org/rfc/rfc9309.html",
  "title": "RFC 9309: Robots Exclusion Protocol",
  "markdown": "# Robots Exclusion Protocol\n\n...",
  "truncated": false
}
```

- `source` is `search` for a search result and `url` for a page you listed.
- `rank` is the position in the results (1 is the first); `query` and `rank` are `null` for listed URLs.
- `truncated` is `true` when the page passed 3 MB or the markdown passed 200,000 characters and was cut.
- A page that could not be delivered has `error` instead of `markdown`, with the reason, and is not charged.

### Input

| Field | What it does |
|---|---|
| `queries` | Up to 20 queries per run. |
| `query` | One query as plain text, handy for agents; runs first. |
| `maxResults` | Pages to read per query, 1 to 10 (default 5). |
| `urls` | Up to 50 pages to convert without searching. |

Example for the API or an agent: `{"query": "markdown table syntax", "maxResults": 3}`.

### Pricing

This actor bills per event, nothing else (each search also runs an Apify actor that Apify bills apart, see below):

- `search`: one per query that returned results.
- `page`: one per page delivered as markdown.

A query that fails or finds nothing, and every page that fails, is blocked by its site or has no readable text, is not charged. A query with at least one result is charged even if none of its pages can be delivered, and what was delivered stays charged if the run later stops or fails. The run never charges more than the maximum you set; what does not fit comes back as an item that says so. Current prices are on the Pricing tab.

#### The search itself

Each query runs Apify's [Google Search Results Scraper](https://apify.com/apify/google-search-scraper) (`apify/google-search-scraper`) once, on your own Apify account, for one page of results. Apify bills that run to you apart, at that actor's prices (one `search-page-scraped` event at your plan's rate and one actor start), also when the query finds nothing. The run asks for one page with every paid add-on off (no ads, no AI overview, no AI engines, no HTML snapshots) and caps its time and charge. Apify does not count it in the maximum you set for this run, so this run takes what each search may cost off that maximum before it starts the next query. The search run, its input and its results stay in your account like any other run.

### How pages are read

- Pages are fetched over HTTP with the user agent `FrameskimFetch`. JavaScript is not run, so pages that only render in a browser may come back without text (and are not charged).
- `robots.txt` is respected for `FrameskimFetch` and for `*`, for the page and again on every redirect, with the `robots.txt` of the host it lands on. If a site's `robots.txt` cannot be read because of a server error or timeout, its pages are skipped.
- Each page gets 10 seconds, at most 5 redirects and 3 MB.
- Pages go through Apify's datacenter proxy, and a run without it stops instead of connecting directly. Before the page and each redirect, the addresses its host name resolves to are checked and private or local network addresses are refused. The proxy resolves the name again on its side, so through the proxy this check is best-effort, not a guarantee.

### Limits

- One results page per query, so at most 10 results per query; results come from the United States edition of Google.
- No crawling: only the result pages and the URLs you list.
- The search interface runs in English; which results come back is up to the search engine.

### Site owners

To keep your pages out, disallow `FrameskimFetch` in your `robots.txt`. Each run reads a host's `robots.txt` once, so a run already in progress may not see the change; runs that start after it read the new file. Rights holders and people whose personal data appears in a page can also write to support@frameskim.dev. Neither a `robots.txt` rule nor a request removes pages from datasets that users already hold.

### Terms and privacy

Web Search & Page Markdown for AI Agents is operated by Emttn LLC and is not affiliated with, endorsed by or sponsored by any search engine or by the sites it reads. You need a lawful basis to request, fetch and use each page, and you are responsible for following Google's terms, each site's terms, copyright and privacy law; a public page or a `robots.txt` allow is not a license to its content. The markdown is an automatic extraction and may be incomplete or inaccurate. Terms: https://frameskim.dev/terms. Privacy: https://frameskim.dev/privacy. Questions: support@frameskim.dev.

# Actor input Schema

## `queries` (type: `array`):

What to search for, one query per line, up to 20 per run. Each query with results is one search event.

## `query` (type: `string`):

One query as plain text, for agents and API calls; it runs before the list above.

## `maxResults` (type: `integer`):

How many of the top results to fetch and convert for each query, from 1 to 10.

## `urls` (type: `array`):

Pages to convert to markdown without searching, up to 50 per run (http or https).

## Actor input object example

```json
{
  "queries": [
    "how does robots.txt work",
    "markdown table syntax"
  ],
  "maxResults": 5,
  "urls": [
    "https://www.rfc-editor.org/rfc/rfc9309.html"
  ]
}
```

# Actor output Schema

## `results` (type: `string`):

Dataset items: source, query, rank, url, title, markdown and truncated; pages that could not be delivered carry an error and are not charged.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "queries": [
        "how does robots.txt work"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("emttn/web-search-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "queries": ["how does robots.txt work"] }

# Run the Actor and wait for it to finish
run = client.actor("emttn/web-search-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "queries": [
    "how does robots.txt work"
  ]
}' |
apify call emttn/web-search-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,emttn/web-search-markdown"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/S7nhScz7rPJbhMWlA/builds/iBG48iubO9aF1z1GV/openapi.json