# Search & Read - Web Search Results with Page Content (`leadproof/search-and-read`) Actor Search the web for a batch of queries, pinned to a country and language, and read the top result pages as clean text and markdown. Shared URLs are fetched once. Failed or unread pages keep their search snippet with an explicit status. - **URL**: https://apify.com/leadproof/search-and-read.md - **Developed by:** [Lead Proof](https://apify.com/leadproof) (community) - **Categories:** AI, Developer tools - **Stats:** 1 total users, 1 monthly users, 91.9% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $5.00 / 1,000 query searcheds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## Search & Read - Web Search Results with Page Content Search the web for a batch of queries, pinned to a country and language, and read the top result pages as clean text and markdown, in one run. One dataset record per query: its results, the content behind each result, and an explicit reason wherever there is no content. - **Country and language pinned per query.** Google through Apify's Google SERP proxy with the country Google domain, `gl`, `hl` and a proxy in that country, so results do not change language between runs. - **Pages read in the same run.** No child Actor per query or per page. A URL returned by several queries is fetched once; every query keeps its own rank and reference to it. - **Explicit statuses.** Search failures and page failures are separate. A page that could not be read keeps its search title and snippet with `contentSource: snippet_only` and null content, never an invented or partial claim. - **Bounded.** Limits on results, pages, bytes, time and concurrency. At the time limit, finished results are returned and unfinished steps say `deadline_exceeded`. - **Browser when the page needs it.** `renderMode: auto` reads over HTTP and switches to Chromium only for JavaScript shells, very thin pages and bot challenges; `browser` renders every page. Each record says which mode produced it. - **Honest coverage.** Every query record carries `search.coverage`: how many organic results there were, whether the engine showed local (map) listings that are not extracted, and whether the engine ran the query as sent. Built for lead research, market and competitor monitoring and RAG pipelines, where a list of search results is only useful once you have the text behind it. The page reader inside it is LeadProof's [Web Page Reader](https://apify.com/leadproof/web-page-reader); the search is our own, not a third-party Actor. ### Try it The prefilled input searches two Kenyan company names, pinned to Kenya and English, and reads up to four pages per query. One run, two queries, and the pages come back in the same dataset: a `query` record per query with its results, each result carrying the page record and the reason if the page could not be read. ### Input ```json { "queries": [ {"id": "eon", "query": "\"Eon Energy\" Kenya"}, {"id": "kavirondo", "query": "\"Kavirondo Chemist\" Homa Bay Kenya"}, {"id": "uganda-row", "query": "hardware store Kampala", "country": "UG"} ], "country": "KE", "language": "en", "maxResultsPerQuery": 4, "maxPagesPerQuery": 4, "contentScope": "page" } ``` | Field | Default | Notes | | --- | --- | --- | | `mode` | `search` | `search` needs `queries`; `urls` reads `urls` directly and never searches. | | `queries` | | Strings or `{id, query, country, language}`, up to 200. Missing ids become `q1`, `q2`... Identical queries keep their own ids; the search runs once and is shared (`search.sharedWithQueryId`). | | `urls` | | URL mode: strings or `{id, url}`, up to 500 public http(s) URLs. | | `country`, `language` | | Required in search mode, at the top level or per row. ISO code or English country name; language tag such as `en`, `sw`, `en-KE`. A row with an unknown country fails alone (`invalid_locale`). | | `maxResultsPerQuery` | 5 | 1-20 search results kept per query. | | `maxPagesPerQuery` | 5 | 0-20 top results whose pages are read. Others keep search metadata with page status `skipped` (`read_not_requested`). 0 = search only. | | `contentScope` | `auto` | `page` keeps headers and footers (contact details), `main` keeps the main content only. | | `renderMode` | `http` | `http` (served HTML only), `auto` (browser only when the page needs it), `browser` (always render). | | `browserConcurrency` | 2 | Pages rendered at once in browser and auto modes. Give the run 2 GB or more for browser work. | | `includeHtml` | false | Adds size-bounded HTML to each page. | | `searchTimeoutSecs`, `pageTimeoutSecs` | 30 | Per attempt. | | `searchConcurrency`, `pageConcurrency` | 3, 5 | Parallel searches and page reads. The reader also limits requests per host. | | `maxRetries` | 1 | Only failures marked retryable (timeouts, 429, 5xx). Never 4xx, unsupported content, login walls or bot blocks. | | `maxPagesTotal` | 200 | Unique page reads per run. | | `maxTotalMegabytes` | 50 | No new read starts after this much text, markdown and HTML. Reads already running finish. | | `maxPageKilobytes` | 2000 | Download limit per page. | | `maxRunSeconds` | 900 | Time limit; the run's own timeout is also respected (30 s kept for saving results). | | `saveFailedSearchPages` | false | Diagnostics: keep the raw result page of a search the provider could not parse (`markup_changed`, `unrecognized_page`) in this run's key-value store. | ### Output One dataset item per query (search mode) or per URL (URL mode), including failed and skipped rows. Every item follows the LeadProof contract v1: `schemaVersion: "1.0"`, `inputId`, `status` (`succeeded`, `partial`, `failed`, `skipped`), `warnings` and `error: null | {code, message, retryable}`. A query item (abbreviated from a real local run, 2026-09-19): ```json { "schemaVersion": "1.0", "recordType": "query", "inputId": "kavirondo", "queryId": "kavirondo", "query": "\"Kavirondo Chemist\" Homa Bay Kenya", "country": "KE", "language": "en", "provider": "google-serp-apify-proxy", "status": "partial", "warnings": ["some_pages_unreadable"], "error": null, "search": {"status": "succeeded", "resultCount": 4, "attempts": 1, "sharedWithQueryId": null, "appliedSettings": {"googleHost": "www.google.co.ke", "gl": "ke", "hl": "en", "proxyCountry": "KE"}, "usage": {"billableRequests": 1, "estimatedProviderCostUsd": 0.0025}}, "results": [ {"resultId": "kavirondo#1", "contentSource": "page", "pageKey": "p786dd5aead6089e3d9d1a195", "searchResult": {"rank": 1, "url": "https://keonline.biz/business/121561-kavirondo-chemist-ltd", "title": "KAVIRONDO CHEMIST LTD - Contacts, Career, Services ...", "snippet": "..."}, "page": {"schemaVersion": "1.0", "status": "succeeded", "httpStatus": 200, "title": "...", "language": "en", "text": "...", "markdown": "...", "contentHash": "...", "links": [], "metadata": {}, "structuredData": {"jsonLd": [], "jsonLdErrors": []}, "warnings": [], "error": null}}, {"resultId": "kavirondo#2", "contentSource": "snippet_only", "searchResult": {"rank": 2, "url": "https://www.facebook.com/groups/.../posts/...", "title": "Staff at Kavirondo Chemist and Agrovet in Homabay ...", "snippet": "..."}, "page": {"status": "failed", "httpStatus": 400, "text": null, "markdown": null, "error": {"code": "http_error", "message": "HTTP 400", "retryable": false}}} ], "stats": {"pagesRead": 3, "pagesFailed": 1, "pagesSkipped": 0}, "completedAt": "2026-09-19T19:46:32.517Z" } ``` - `searchResult` is SearchResult v1: `rank` is one-based within the returned results, not a global Google rank. Additive fields from the search library (`urlSource`, `page`, `displayUrl`) are kept. - `page` is PageRecord v1 from the page reader, always with every field present. Records we create ourselves (skipped or failed reads) have null content. - `contentSource`: `page` (readable content present), `snippet_only` (only the search title and snippet), `none`. - Status of a query: `failed` when the search failed, `succeeded` when every requested page was read, `partial` when some pages could not be read or the run stopped early. An empty result list from a successful search is `succeeded`, not a failure. - `search.coverage` and the warnings say what the search could not show: - `local_results_excluded` (and, with no organic results, `no_organic_results_local_listings_not_read`): the engine showed local business listings from its map product, which are not extracted. Zero organic results then means "we did not read what exists", not "the business has no web presence". - `exact_phrase_not_found`, `query_changed_by_engine`, `query_auto_corrected`: the engine ran another query (for example it dropped the quotes). The results are kept and `search.coverage.queryAsSent` is false. They are not evidence that the name matched; judge identity from the pages themselves. - `search.providerPages` and `search.providerAttempts` keep the search provider's own per-page and per-request diagnostics. - URL mode items are PageRecords with `recordType: "page"` and `pageKey`. The run's `SUMMARY` record has counts, stop reason, engine statistics (searches, shared searches, page requests, shared pages, bytes, retries, limits hit), backends and charged events. Views: `overview` (one line per query) and `pages` (URL mode). ### How it works and what it depends on | Part | What | Where it runs | Paid to | | --- | --- | --- | --- | | Web search | LeadProof's own `websearch` library, Google through Apify's Google SERP proxy | In this run | Covered by `query-searched` | | Page reader | LeadProof's [Web Page Reader](https://apify.com/leadproof/web-page-reader) engine, in a Node process next to the Python run | In this run | Covered by `page-read` / `page-read-rendered` | | Orchestration | The `search_and_read` package | In this run | Covered by the same events | Nothing is charged twice: the search proxy and the page reads are the platform usage of this run, and under pay-per-event that usage is ours to cover, not an extra line on your bill. No third-party Actor is called at runtime. Search requests and page reads are ordinary network traffic of this run. The page reader enforces the network policy (public destinations only, including redirects and browser subrequests); this Actor also rejects private, loopback, link-local, credentialed and non-HTTP(S) URLs in the input and in search results before anything is read. Robots rules for single result pages are the reader's policy; this Actor does not crawl sites. Page text is untrusted data. It is returned as data and never interpreted as instructions. ### What you pay for Pay per event, so an empty search and an unreadable page cost nothing: | Event | When it is charged | Price | | --- | --- | --- | | `query-searched` | One per query whose search completed, including one that found nothing | **$0.005** | | `page-read` | One per unique page read over HTTP with readable content | **$0.002** | | `page-read-rendered` | One per unique page that Chromium had to render (`renderMode: browser`, or `auto` falling back to it) | **$0.006** | | `apify-actor-start` | Apify's standard start event, once per GB of the run's memory | $0.00005 | What that means in practice: - **Each page is charged once, under one event.** A page read over HTTP is `page-read`. A page the browser rendered is `page-read-rendered` and **only** that: never both events for the same page. - **You are not charged for what you did not get.** A failed search, a query skipped by a limit, a page that returned 404, was blocked, held no text, or was withheld by your spending limit: free. A result that keeps only its search title and snippet (`contentSource: snippet_only`) is free too - it was not read, so it is not charged as a read. - **Repeats are charged once.** Two rows with the identical query share one search and one charge; a URL returned by five queries is read once and charged once, and every query still gets its own record pointing at it. - **You choose whether the browser can run.** `renderMode: http` (the default) never charges the rendered price. `auto` charges it only for the pages where the HTTP result was not the page; the pages it kept on HTTP stay at $0.002. `browser` renders everything. A rendered page costs us about four times an HTTP one, which is why it is priced apart rather than averaged into one number. - **Your spending limit is respected before the work starts.** No query begins unless the remaining limit covers its search and its page reads, and a restarted run never charges the same query or page twice. - Examples at 1 GB memory: - The example task "Read the pages behind a company-name search (Kenya)", a real run: 2 queries, 8 results, 7 readable pages, one result that kept its snippet because the site returned an error, 52 seconds: 2 x $0.005 + 7 x $0.002 + $0.00005 = **$0.02405**. - 10 queries, 3 readable pages each, HTTP: 10 x $0.005 + 30 x $0.002 + $0.00005 = **$0.11005**. - The same 10 queries where half the pages cannot be read: 10 x $0.005 + 15 x $0.002 = **$0.08005**. - 50 queries, search only (`maxPagesPerQuery: 0`): **$0.25005**. ### Measured performance Owner-account runs, September 2026 (run IDs in `DEPLOYMENT.md`): - 3 Kenyan queries, 3 results each: 9 pages read in 31 s, $0.0104 total usage (of which Google SERP proxy $0.0075). - 27-row company list searched through this Actor (8 runs, 37 queries including retries without the city, 79 page reads, 44 readable): **$0.1106, about $0.0041 per row** in `http` mode ($0.1310, $0.0049 per row in `auto`). 79% of it is the Google SERP proxy at $0.0025 per request, 13% compute, 7% key-value checkpoint writes. - A JavaScript page returns 67 characters over HTTP and 1,433 characters in the browser; in `auto` the fallback happened on its own and the static page stayed on HTTP. ### Limitations - The browser renders one page at a time per slot and costs more: on a 27-row business list `auto` cost 18% more than `http` and found nothing extra, because business directories are server-rendered. Use `auto` for sites that are JavaScript applications. - Local (map) listings are never extracted, only reported (`local_results_excluded`). - Only public pages. No login, CAPTCHA solving or block evasion. Directory and social sites often return 403 (`blocked_by_site`). - One Google results page per 10 results; very specific queries can return very few results. - The byte limit is checked when a read starts, so the total can exceed it by the reads already running. - Content fields are capped at 400,000 characters each and trimmed further if an item would exceed Apify's 9 MB item limit; `contentHash` always describes the full text before trimming. ### Development `python -m unittest discover -s test` (from this folder) covers batch order, locale propagation, shared URLs and searches, search versus page failures, retries, timeouts, the deadline, unsupported content, limits, resume from checkpoints, billing idempotency and budget exhaustion, input safety, contract schema compatibility, the Node bridge protocol (against a fake reader) and the web search adapter. `search_and_read/contracts/` holds the JSON schemas; `test/fixtures/contract/` holds contract fixtures. The engine package has no Apify dependency and can be imported by other Actors. *** *Built by [LeadProof](https://leadproof.co) - verified local-business lead lists, built to order.* # Actor input Schema ## `mode` (type: `string`): Search mode needs queries. URL mode needs urls and never runs a search. ## `queries` (type: `array`): Strings or objects {id, query, country, language}. Up to 200. Rows without an id get q1, q2, ... Identical queries keep their own ids; the search runs once and is shared. ## `urls` (type: `array`): Strings or objects {id, url}. Up to 500 public http(s) URLs. Duplicate URLs are fetched once and returned once per row. ## `country` (type: `string`): Required in search mode (here or per query row). ISO 3166-1 alpha-2 code (KE, US, GB) or an English country name. Pins the search to that country. ## `language` (type: `string`): Required in search mode (here or per query row). Language tag such as en, sw or en-KE. Sets the search interface language and the Accept-Language of page reads. ## `maxResultsPerQuery` (type: `integer`): Search results kept per query. Rank is one-based within the returned results. ## `maxPagesPerQuery` (type: `integer`): Top results whose pages are read. Other results keep search metadata only, with page status skipped. 0 = search only. ## `renderMode` (type: `string`): http reads the served HTML only. browser renders every page in Chromium (slower, more memory). auto reads over HTTP and switches to the browser only when the page is a JavaScript shell, too thin or a bot challenge. Each page record says which mode was used (diagnostics.renderModeUsed). ## `contentScope` (type: `string`): Which part of the page becomes text and markdown. Use page when you need contact details that sit in headers and footers. ## `browserConcurrency` (type: `integer`): Pages rendered in the browser at once (browser and auto modes). Give the run 2 GB or more for browser work. ## `saveFailedSearchPages` (type: `boolean`): When the search engine page cannot be parsed (markup\_changed, unrecognized\_page), keep the raw page in this run's key-value store (DIAG-SERP--) so the parser can be fixed. Off by default. ## `includeHtml` (type: `boolean`): Add the page HTML (size bounded) to each page record. Off by default. ## `searchTimeoutSecs` (type: `integer`): Per search attempt. ## `pageTimeoutSecs` (type: `integer`): Per page read attempt. ## `searchConcurrency` (type: `integer`): Maximum searches running at once. ## `pageConcurrency` (type: `integer`): Maximum page reads running at once across all queries. ## `maxRetries` (type: `integer`): Extra attempts for failures the search provider or reader marks retryable (timeouts, 429, 5xx). Never for 4xx, unsupported content or login walls. ## `maxPagesTotal` (type: `integer`): Unique page reads per run. Results past the limit keep search metadata only. ## `maxTotalMegabytes` (type: `integer`): No new page read starts once this much text, markdown and HTML has been returned. Reads already running finish. ## `maxPageKilobytes` (type: `integer`): Download limit per page, passed to the page reader. ## `maxRunSeconds` (type: `integer`): At this limit no new work starts; finished results are returned and unfinished steps are marked skipped with deadline\_exceeded. The run's own timeout is respected too. ## Actor input object example ```json { "mode": "search", "queries": [ { "id": "eon", "query": "\"Eon Energy\" Kenya" } ], "country": "ke", "language": "en", "maxResultsPerQuery": 5, "maxPagesPerQuery": 5, "renderMode": "http", "contentScope": "auto", "browserConcurrency": 2, "saveFailedSearchPages": false, "includeHtml": false, "searchTimeoutSecs": 30, "pageTimeoutSecs": 30, "searchConcurrency": 3, "pageConcurrency": 5, "maxRetries": 1, "maxPagesTotal": 200, "maxTotalMegabytes": 50, "maxPageKilobytes": 2000, "maxRunSeconds": 900 } ``` # Actor output Schema ## `results` (type: `string`): Query records with search metadata and {searchResult, page} pairs. ## `overview` (type: `string`): One line per query: status, result and page counts, warnings. ## `summary` (type: `string`): Counts, stop reason, limits, backends and charged events. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "queries": [ { "id": "eon", "query": "\"Eon Energy\" Kenya" } ], "country": "ke", "language": "en" }; // Run the Actor and wait for it to finish const run = await client.actor("leadproof/search-and-read").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "queries": [{ "id": "eon", "query": "\"Eon Energy\" Kenya", }], "country": "ke", "language": "en", } # Run the Actor and wait for it to finish run = client.actor("leadproof/search-and-read").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "queries": [ { "id": "eon", "query": "\\"Eon Energy\\" Kenya" } ], "country": "ke", "language": "en" }' | apify call leadproof/search-and-read --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,leadproof/search-and-read" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/FfAIydWqR5Ye2BekM/builds/pRHwkfVLuStF7vOsF/openapi.json