Open Knowledge Search for AI Agents — Wikipedia, arXiv, HN, Stack Exchange

Knowledge search for AI agents in one call: send a query and get ranked, deduplicated, citation-ready results from Wikipedia, OpenAlex, Hacker News, Stack Exchange and (opt-in) arXiv. Every result has the same schema: title, URL, date, authors, score, a clean markdown block, and an explicit licence and attribution line. It uses official public APIs only. No Google scraping, no browser, no login.

What does it do?

You give it one query (or a batch of up to 50). It searches the selected open sources in parallel, merges duplicates (by DOI, arXiv id or URL), ranks the list, and pushes one dataset item per result. Each item includes a ready-to-use markdown field and an attribution string you can paste next to quoted text.

It complements generic web search: instead of arbitrary web pages, you get structured facts, research papers, developer Q&A and tech discussions.

Who is it for / use cases

• AI agent and MCP builders (LangChain, LlamaIndex, n8n, Make, Claude Desktop, Cursor): a "research this topic" tool that returns citable snippets.
• RAG developers: ground LLM answers, or build a small licensed corpus for a list of queries.
• Research digests: schedule a run on a topic with dateFrom to get new papers, HN threads and Q&A.
• Developer assistants: look up Stack Overflow questions and related HN discussions for a library or error.

Features

• Sources: Wikipedia (any language edition), OpenAlex (papers), Hacker News (Algolia search), Stack Exchange (any site), arXiv (opt-in).
• Stable output schema across all sources, with rank, sourceRank, score, tags, doi, arxivId, pdfUrl when the source provides them.
• Licence and attribution on every result (see the table below).
• Cross-source deduplication (alsoFoundIn lists the other sources).
• Optional fuller content (fetchFullContent), truncated to maxContentChars.
• Date filter (dateFrom) for sources that expose a date.
• Hard cap maxResults on results pushed and billed per run.
• Pacing: arXiv is opt-in and paced to respect its terms of use; Stack Exchange backoff responses are honoured.

Source	Licence reported in license
Wikipedia	CC BY-SA 4.0
OpenAlex	CC0 1.0 (metadata)
Stack Exchange	CC BY-SA
Hacker News	No open licence: cite and link only
arXiv	Metadata licence as reported by the Actor; PDFs are linked, not redistributed

Input

Input parameters

Field	Type	Default	Description
query	string	—	Search text. Required unless queries is given.
queries	string[]	—	Batch of queries (max 50).
sources	string[]	wikipedia, openalex, hackernews, stackexchange	Allowed: wikipedia, openalex, hackernews, stackexchange, arxiv.
maxResultsPerSource	integer	5	1–50 per source per query.
maxResults	integer	20	Hard cap on results pushed and billed.
language	string	en	Wikipedia language edition.
stackExchangeSite	string	stackoverflow	Stack Exchange site slug.
stackExchangeKey	string (secret)	—	Your own Stack Exchange app key to raise the quota.
fetchFullContent	boolean	false	Wikipedia: longer extract; Stack Exchange: question body. Open APIs only.
maxContentChars	integer	2000	Truncate content (200–20000).
dateFrom	string	—	ISO date, e.g. 2025-01-01.
dedupe	boolean	true	Merge duplicates across sources.
proxyConfiguration	object	Apify datacenter proxy	Outgoing requests use the Apify datacenter pool by default; turn off to connect directly.

Example input

{
  "query": "retrieval augmented generation",
  "sources": ["wikipedia", "openalex", "hackernews", "stackexchange"],
  "maxResultsPerSource": 3,
  "maxResults": 12
}

Output

Example output

Real sample (shortened) from a production run on the query above:

{
  "query": "retrieval augmented generation",
  "rank": 2,
  "source": "openalex",
  "sourceRank": 1,
  "title": "Active Retrieval Augmented Generation",
  "url": "https://doi.org/10.18653/v1/2023.emnlp-main.495",
  "snippet": "Zhengbao Jiang, Frank Xu, ... Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.",
  "authors": ["Zhengbao Jiang", "Frank F. Xu", "Luyu Gao"],
  "publishedAt": "2023-01-01T00:00:00.000Z",
  "score": 456,
  "doi": "10.18653/v1/2023.emnlp-main.495",
  "pdfUrl": "https://aclanthology.org/2023.emnlp-main.495.pdf",
  "tags": ["Topic Modeling", "Natural Language Processing Techniques"],
  "license": "CC0 1.0",
  "attribution": "\"Active Retrieval Augmented Generation\" — OpenAlex, CC0 1.0, https://doi.org/10.18653/v1/2023.emnlp-main.495",
  "markdown": "### [Active Retrieval Augmented Generation](https://doi.org/10.18653/v1/2023.emnlp-main.495)\n\n*openalex · 2023-01-01 · ... · CC0 1.0*\n\n...",
  "alsoFoundIn": [],
  "fetchedAt": "2026-10-04T03:16:43.503Z"
}

Output fields & fill rates

Measured on a 40-item production run (default four sources, several queries). Fields depend on the source, so some are legitimately empty.

Field	Fill rate	Note
query, source, rank, sourceRank, title, url, license, attribution, markdown, fetchedAt	100%	Always set
snippet	82.5%
publishedAt	77.5%	Wikipedia has no publication date
authors	77.5%	Empty for Wikipedia
content	67.5%	More complete with fetchFullContent
score	60%	HN points, Stack Exchange score, OpenAlex citations
updatedAt	60%	When the source provides it
tags	47.5%
doi	37.5%	Papers only
commentsCount	40%	HN / Stack Exchange
pdfUrl	35%	Open-access link when available
isAnswered	20%	Stack Exchange only
arxivId	17.5%	arXiv papers only
alsoFoundIn	0%	No duplicates occurred in this run

Configuration tips

• Quick start: set only query; the defaults search four sources, 5 results each, capped at 20 results per run.
• Batch: use queries (max 50). maxResults caps the whole run, not each query, so raise it for batches.
• Papers only: "sources": ["openalex", "arxiv"]. arXiv is opt-in and paced, so runs including it are slower.
• Other languages / sites: set language for Wikipedia and stackExchangeSite (for example superuser).
• Fresh content: set dateFrom; it applies only to sources that expose a date, and Wikipedia results are not filtered by it.
• Cost control: lower maxResults, since each pushed result is billed.

How to use via API

Replace YOUR_TOKEN with your Apify API token.

curl

curl -X POST "https://api.apify.com/v2/acts/ACTOR_ID/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"query": "retrieval augmented generation", "maxResults": 12}'

JavaScript

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_TOKEN' });
const run = await client.actor('ACTOR_ID').call({
  query: 'retrieval augmented generation',
  sources: ['wikipedia', 'openalex', 'hackernews'],
  maxResults: 12,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items.map((i) => i.markdown).join('\n\n'));

Python

from apify_client import ApifyClient

client = ApifyClient("YOUR_TOKEN")
run = client.actor("ACTOR_ID").call(run_input={
    "query": "retrieval augmented generation",
    "sources": ["wikipedia", "openalex", "hackernews"],
    "fetchFullContent": False,
    "dedupe": True,
    "maxResults": 12,
})
items = client.dataset(run["defaultDatasetId"]).list_items().items
print("\n\n".join(i["markdown"] for i in items))

ACTOR_ID is the Actor's ID or username~actor-name shown on its Apify page. You can also call it from MCP clients through Apify's MCP server, or from n8n and Make with their Apify integrations.

Pricing

Pay per result: one deduplicated result item pushed to the dataset is one billed event. Price per 1,000 results depends on your Apify plan:

Plan	Per 1,000 results
FREE	$1.20
BRONZE	$1.04
SILVER	$0.92
GOLD and above	$0.80

Use maxResults to cap spend. Apify platform usage is billed separately; in the measured runs it was very small (about $0.0007 of platform usage for a 40-item run).

Troubleshooting

• "Invalid input: Provide query or queries": the run fails when no non-empty search text is given. Set query or queries.
• Fewer results than expected: the cap maxResults applies to the whole run, and dedupe removes duplicates. Raise maxResults or maxResultsPerSource.
• A source returns nothing: public APIs can rate-limit shared IPs. Retry later, or for Stack Exchange add your own stackExchangeKey.
• arXiv is slow: it is opt-in and paced to respect its terms of use.
• Empty publishedAt or authors: Wikipedia entries do not carry them.

FAQ

Does it scrape Google or arbitrary web pages? No. Only official public APIs of the listed sources.

Can I republish the content? Keep the attribution field. Wikipedia and Stack Exchange text is CC BY-SA (share-alike); Hacker News content has no open licence, so cite and link only. Check each source's terms for your use.

Does it return personal data? Only author or username strings as publicly displayed by each source, for attribution. No emails or contact details.

Is it a replacement for web search? No, it complements it. It covers open knowledge sources, not the whole web.

Keywords

knowledge search for ai agents, web search for ai agents, rag search api, mcp research tool, wikipedia search api, arxiv search api, openalex api, hacker news search api, stack overflow search api, citation-ready search, llm grounding, research assistant api, academic paper search

Hashtags

#AIAgents #RAG #MCP #LLM #Wikipedia #arXiv #OpenAlex #HackerNews #StackOverflow #ResearchTools #KnowledgeSearch