# Open Knowledge Search for AI Agents — Wikipedia, arXiv, HN (`leadsbrary/open-knowledge-search-for-agents`) Actor

Knowledge search for AI agents and RAG: one query searches Wikipedia, OpenAlex, Hacker News, Stack Exchange and arXiv. Returns deduplicated markdown with URL, date, authors, licence. Export JSON, CSV, Excel. From $0.80 per 1,000 results.

- **URL**: https://apify.com/leadsbrary/open-knowledge-search-for-agents.md
- **Developed by:** [Alexandre Manguis](https://apify.com/leadsbrary) (community)
- **Categories:** AI, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.80 / 1,000 results

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Open Knowledge Search for AI Agents — Wikipedia, arXiv, HN, Stack Exchange

**Knowledge search for AI agents** in one call: send a query and get ranked, deduplicated, citation-ready results from Wikipedia, OpenAlex, Hacker News, Stack Exchange and (opt-in) arXiv. Every result has the same schema: title, URL, date, authors, score, a clean markdown block, and an explicit licence and attribution line. It uses official public APIs only. No Google scraping, no browser, no login.

### What does it do?

You give it one query (or a batch of up to 50). It searches the selected open sources in parallel, merges duplicates (by DOI, arXiv id or URL), ranks the list, and pushes one dataset item per result. Each item includes a ready-to-use `markdown` field and an `attribution` string you can paste next to quoted text.

It complements generic web search: instead of arbitrary web pages, you get structured facts, research papers, developer Q\&A and tech discussions.

### Who is it for / use cases

- **AI agent and MCP builders** (LangChain, LlamaIndex, n8n, Make, Claude Desktop, Cursor): a "research this topic" tool that returns citable snippets.
- **RAG developers**: ground LLM answers, or build a small licensed corpus for a list of queries.
- **Research digests**: schedule a run on a topic with `dateFrom` to get new papers, HN threads and Q\&A.
- **Developer assistants**: look up Stack Overflow questions and related HN discussions for a library or error.

### Features

- Sources: Wikipedia (any language edition), OpenAlex (papers), Hacker News (Algolia search), Stack Exchange (any site), arXiv (opt-in).
- Stable output schema across all sources, with `rank`, `sourceRank`, `score`, `tags`, `doi`, `arxivId`, `pdfUrl` when the source provides them.
- Licence and attribution on every result (see the table below).
- Cross-source deduplication (`alsoFoundIn` lists the other sources).
- Optional fuller content (`fetchFullContent`), truncated to `maxContentChars`.
- Date filter (`dateFrom`) for sources that expose a date.
- Hard cap `maxResults` on results pushed and billed per run.
- Pacing: arXiv is opt-in and paced to respect its terms of use; Stack Exchange `backoff` responses are honoured.

| Source | Licence reported in `license` |
|---|---|
| Wikipedia | CC BY-SA 4.0 |
| OpenAlex | CC0 1.0 (metadata) |
| Stack Exchange | CC BY-SA |
| Hacker News | No open licence: cite and link only |
| arXiv | Metadata licence as reported by the Actor; PDFs are linked, not redistributed |

### Input

#### Input parameters

| Field | Type | Default | Description |
|---|---|---|---|
| `query` | string | — | Search text. Required unless `queries` is given. |
| `queries` | string\[] | — | Batch of queries (max 50). |
| `sources` | string\[] | wikipedia, openalex, hackernews, stackexchange | Allowed: `wikipedia`, `openalex`, `hackernews`, `stackexchange`, `arxiv`. |
| `maxResultsPerSource` | integer | 5 | 1–50 per source per query. |
| `maxResults` | integer | 20 | Hard cap on results pushed and billed. |
| `language` | string | `en` | Wikipedia language edition. |
| `stackExchangeSite` | string | `stackoverflow` | Stack Exchange site slug. |
| `stackExchangeKey` | string (secret) | — | Your own Stack Exchange app key to raise the quota. |
| `fetchFullContent` | boolean | false | Wikipedia: longer extract; Stack Exchange: question body. Open APIs only. |
| `maxContentChars` | integer | 2000 | Truncate `content` (200–20000). |
| `dateFrom` | string | — | ISO date, e.g. `2025-01-01`. |
| `dedupe` | boolean | true | Merge duplicates across sources. |
| `proxyConfiguration` | object | Apify datacenter proxy | Outgoing requests use the Apify datacenter pool by default; turn off to connect directly. |

#### Example input

```json
{
  "query": "retrieval augmented generation",
  "sources": ["wikipedia", "openalex", "hackernews", "stackexchange"],
  "maxResultsPerSource": 3,
  "maxResults": 12
}
```

### Output

#### Example output

Real sample (shortened) from a production run on the query above:

```json
{
  "query": "retrieval augmented generation",
  "rank": 2,
  "source": "openalex",
  "sourceRank": 1,
  "title": "Active Retrieval Augmented Generation",
  "url": "https://doi.org/10.18653/v1/2023.emnlp-main.495",
  "snippet": "Zhengbao Jiang, Frank Xu, ... Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.",
  "authors": ["Zhengbao Jiang", "Frank F. Xu", "Luyu Gao"],
  "publishedAt": "2023-01-01T00:00:00.000Z",
  "score": 456,
  "doi": "10.18653/v1/2023.emnlp-main.495",
  "pdfUrl": "https://aclanthology.org/2023.emnlp-main.495.pdf",
  "tags": ["Topic Modeling", "Natural Language Processing Techniques"],
  "license": "CC0 1.0",
  "attribution": "\"Active Retrieval Augmented Generation\" — OpenAlex, CC0 1.0, https://doi.org/10.18653/v1/2023.emnlp-main.495",
  "markdown": "### [Active Retrieval Augmented Generation](https://doi.org/10.18653/v1/2023.emnlp-main.495)\n\n*openalex · 2023-01-01 · ... · CC0 1.0*\n\n...",
  "alsoFoundIn": [],
  "fetchedAt": "2026-10-04T03:16:43.503Z"
}
```

#### Output fields & fill rates

Measured on a 40-item production run (default four sources, several queries). Fields depend on the source, so some are legitimately empty.

| Field | Fill rate | Note |
|---|---|---|
| `query`, `source`, `rank`, `sourceRank`, `title`, `url`, `license`, `attribution`, `markdown`, `fetchedAt` | 100% | Always set |
| `snippet` | 82.5% | |
| `publishedAt` | 77.5% | Wikipedia has no publication date |
| `authors` | 77.5% | Empty for Wikipedia |
| `content` | 67.5% | More complete with `fetchFullContent` |
| `score` | 60% | HN points, Stack Exchange score, OpenAlex citations |
| `updatedAt` | 60% | When the source provides it |
| `tags` | 47.5% | |
| `doi` | 37.5% | Papers only |
| `commentsCount` | 40% | HN / Stack Exchange |
| `pdfUrl` | 35% | Open-access link when available |
| `isAnswered` | 20% | Stack Exchange only |
| `arxivId` | 17.5% | arXiv papers only |
| `alsoFoundIn` | 0% | No duplicates occurred in this run |

### Configuration tips

- **Quick start**: set only `query`; the defaults search four sources, 5 results each, capped at 20 results per run.
- **Batch**: use `queries` (max 50). `maxResults` caps the whole run, not each query, so raise it for batches.
- **Papers only**: `"sources": ["openalex", "arxiv"]`. arXiv is opt-in and paced, so runs including it are slower.
- **Other languages / sites**: set `language` for Wikipedia and `stackExchangeSite` (for example `superuser`).
- **Fresh content**: set `dateFrom`; it applies only to sources that expose a date, and Wikipedia results are not filtered by it.
- **Cost control**: lower `maxResults`, since each pushed result is billed.

### How to use via API

Replace `YOUR_TOKEN` with your Apify API token.

**curl**

```bash
curl -X POST "https://api.apify.com/v2/acts/ACTOR_ID/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"query": "retrieval augmented generation", "maxResults": 12}'
```

**JavaScript**

```js
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_TOKEN' });
const run = await client.actor('ACTOR_ID').call({
  query: 'retrieval augmented generation',
  sources: ['wikipedia', 'openalex', 'hackernews'],
  maxResults: 12,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items.map((i) => i.markdown).join('\n\n'));
```

**Python**

```python
from apify_client import ApifyClient

client = ApifyClient("YOUR_TOKEN")
run = client.actor("ACTOR_ID").call(run_input={
    "query": "retrieval augmented generation",
    "sources": ["wikipedia", "openalex", "hackernews"],
    "fetchFullContent": False,
    "dedupe": True,
    "maxResults": 12,
})
items = client.dataset(run["defaultDatasetId"]).list_items().items
print("\n\n".join(i["markdown"] for i in items))
```

`ACTOR_ID` is the Actor's ID or `username~actor-name` shown on its Apify page. You can also call it from MCP clients through Apify's MCP server, or from n8n and Make with their Apify integrations.

### Pricing

Pay per result: one deduplicated result item pushed to the dataset is one billed event. Price per 1,000 results depends on your Apify plan:

| Plan | Per 1,000 results |
|---|---|
| FREE | $1.20 |
| BRONZE | $1.04 |
| SILVER | $0.92 |
| GOLD and above | $0.80 |

Use `maxResults` to cap spend. Apify platform usage is billed separately; in the measured runs it was very small (about $0.0007 of platform usage for a 40-item run).

### Troubleshooting

- **"Invalid input: Provide query or queries"**: the run fails when no non-empty search text is given. Set `query` or `queries`.
- **Fewer results than expected**: the cap `maxResults` applies to the whole run, and dedupe removes duplicates. Raise `maxResults` or `maxResultsPerSource`.
- **A source returns nothing**: public APIs can rate-limit shared IPs. Retry later, or for Stack Exchange add your own `stackExchangeKey`.
- **arXiv is slow**: it is opt-in and paced to respect its terms of use.
- **Empty `publishedAt` or `authors`**: Wikipedia entries do not carry them.

### FAQ

**Does it scrape Google or arbitrary web pages?** No. Only official public APIs of the listed sources.

**Can I republish the content?** Keep the `attribution` field. Wikipedia and Stack Exchange text is CC BY-SA (share-alike); Hacker News content has no open licence, so cite and link only. Check each source's terms for your use.

**Does it return personal data?** Only author or username strings as publicly displayed by each source, for attribution. No emails or contact details.

**Is it a replacement for web search?** No, it complements it. It covers open knowledge sources, not the whole web.

### Keywords

knowledge search for ai agents, web search for ai agents, rag search api, mcp research tool, wikipedia search api, arxiv search api, openalex api, hacker news search api, stack overflow search api, citation-ready search, llm grounding, research assistant api, academic paper search

### Hashtags

\#AIAgents #RAG #MCP #LLM #Wikipedia #arXiv #OpenAlex #HackerNews #StackOverflow #ResearchTools #KnowledgeSearch

# Changelog

This Actor's version history is a separate document: https://apify.com/leadsbrary/open-knowledge-search-for-agents/changelog.md

# Actor input Schema

## `query` (type: `string`):

Search text in natural language or keywords. Required unless 'Queries' is given.

## `queries` (type: `array`):

Optional batch of queries (max 50); each is searched like 'Query'.

## `sources` (type: `array`):

Sources to search: wikipedia, openalex, hackernews, stackexchange, arxiv. arXiv is opt-in (its terms limit clients to 1 request per 3 seconds, so the Actor paces itself).

## `maxResultsPerSource` (type: `integer`):

Results requested from each source for each query (1-50).

## `maxResults` (type: `integer`):

Hard cap on the total number of results pushed (and billed) in this run.

## `language` (type: `string`):

Wikipedia language edition code (ISO 639-1), e.g. en, fr, de.

## `stackExchangeSite` (type: `string`):

Stack Exchange site slug, e.g. stackoverflow, superuser, math.

## `stackExchangeKey` (type: `string`):

Your own Stack Exchange app key to raise the request quota. Not an access token; never logged.

## `fetchFullContent` (type: `boolean`):

Wikipedia: longer plain-text extract; Stack Exchange: question body. Uses open APIs only; never scrapes third-party pages.

## `maxContentChars` (type: `integer`):

Truncate the 'content' field to this many characters (200-20000).

## `dateFrom` (type: `string`):

ISO date such as 2025-01-01. Applied to sources that expose a date (OpenAlex, arXiv, Hacker News, Stack Exchange).

## `dedupe` (type: `boolean`):

Merge duplicates by DOI, arXiv id or URL; the other sources are listed in 'alsoFoundIn'.

## `proxyConfiguration` (type: `object`):

Outgoing requests use the default Apify datacenter proxy pool so the shared per-IP quotas of the public APIs are less likely to be exhausted. Turn off to connect directly.

## Actor input object example

```json
{
  "query": "retrieval augmented generation",
  "queries": [],
  "sources": [
    "wikipedia",
    "openalex",
    "hackernews",
    "stackexchange"
  ],
  "maxResultsPerSource": 3,
  "maxResults": 12,
  "language": "en",
  "stackExchangeSite": "stackoverflow",
  "fetchFullContent": false,
  "maxContentChars": 2000,
  "dedupe": true,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "query": "retrieval augmented generation",
    "queries": [],
    "sources": [
        "wikipedia",
        "openalex",
        "hackernews",
        "stackexchange"
    ],
    "maxResultsPerSource": 3,
    "maxResults": 12,
    "language": "en",
    "stackExchangeSite": "stackoverflow",
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("leadsbrary/open-knowledge-search-for-agents").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "query": "retrieval augmented generation",
    "queries": [],
    "sources": [
        "wikipedia",
        "openalex",
        "hackernews",
        "stackexchange",
    ],
    "maxResultsPerSource": 3,
    "maxResults": 12,
    "language": "en",
    "stackExchangeSite": "stackoverflow",
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("leadsbrary/open-knowledge-search-for-agents").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "query": "retrieval augmented generation",
  "queries": [],
  "sources": [
    "wikipedia",
    "openalex",
    "hackernews",
    "stackexchange"
  ],
  "maxResultsPerSource": 3,
  "maxResults": 12,
  "language": "en",
  "stackExchangeSite": "stackoverflow",
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call leadsbrary/open-knowledge-search-for-agents --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,leadsbrary/open-knowledge-search-for-agents"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/h9WlohM4YbOf8wzpj/builds/1C09Vq3JCj1LeI2g8/openapi.json