# Wikipedia Scraper (`automation-lab/wikipedia-scraper`) Actor

Search and extract Wikipedia articles — titles, summaries, full content, categories, and images. Uses the free MediaWiki API.

- **URL**: https://apify.com/automation-lab/wikipedia-scraper.md
- **Developed by:** [Automation Lab](https://apify.com/automation-lab) (community)
- **Categories:** Education, AI
- **Features:** Limited permissions
- **Stats:** 65 total users, 13 monthly users, 99.8% runs succeeded, 1 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.60 / 1,000 article extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Wikipedia Scraper

Search Wikipedia by keyword or fetch known pages by exact title. Export summaries, optional full plaintext, categories, Wikidata IDs, page descriptions, URLs, and input provenance from any of Wikipedia's 300+ language editions.

### What does Wikipedia Scraper do?

Wikipedia Scraper uses the official MediaWiki API to search by topic, fetch exact article titles, or combine both workflows. Every result includes the introductory `extract`, categories, Wikidata ID and short page description when available, page metadata, and clear provenance. Enable `includeFullContent` to also receive the full article as plain text in `fullContent`, ready for RAG and knowledge-base datasets.

Keyword results follow Wikipedia's relevance ranking and include `searchQuery` and `searchRank`. Exact-title results include `requestedTitle` and automatically resolve redirects. Mixed inputs are deduplicated by Wikipedia page ID, so an overlapping page is emitted and charged only once while `matchedSearchQueries` and `requestedTitles` retain all matching inputs.

### Who is it for?

- 🎓 **Academic researchers** — extracting structured knowledge from Wikipedia articles at scale
- 🤖 **NLP engineers** — building training datasets from Wikipedia text and metadata
- 📊 **Data analysts** — collecting factual data and statistics from Wikipedia pages
- 💻 **App developers** — enriching applications with Wikipedia content and summaries
- 📝 **Content creators** — gathering reference material and structured facts for writing

### Why scrape Wikipedia?

Wikipedia is the world's largest free encyclopedia with over 60 million articles across 300+ languages. It's a primary source for:

- **Knowledge base construction** — build reference datasets for AI training, chatbots, or research databases
- **LLM and RAG pipelines** — feed clean, structured article text into retrieval-augmented generation systems, fine-tuning datasets, or AI agent knowledge bases
- **Content enrichment** — add Wikipedia summaries to product catalogs, educational platforms, or content management systems
- **Research and analysis** — analyze article coverage, word counts, and edit patterns across topics
- **Multilingual data** — gather information in any language Wikipedia supports
- **SEO and content strategy** — understand topic coverage and find content gaps

### How much does it cost to scrape Wikipedia?

Wikipedia Scraper uses pay-per-event pricing:

| Event | Price |
|-------|-------|
| Run started | $0.001 |
| Article extracted | $0.001 per article |

**Example costs:**

- 10 articles on "machine learning": ~$0.011
- 100 articles on "history": ~$0.101
- 500 articles across 5 keywords: ~$0.506

Platform costs are minimal — a typical run uses under $0.001 in compute. Wikipedia's API is fast and does not require proxies.

### Input parameters

| Parameter | Type | Description | Default |
|-----------|------|-------------|---------|
| `searchQueries` | string\[] | Keywords to search on Wikipedia. Each keyword runs a separate ranked search. | Optional\* |
| `articleTitles` | string\[] | Exact Wikipedia article titles to fetch directly; redirects are resolved. | Optional\* |
| `language` | string | Wikipedia language code (e.g., `en`, `de`, `fr`, `es`, `ja`, `zh`) | `"en"` |
| `maxResultsPerSearch` | integer | Maximum articles per keyword (1–500); does not limit exact titles | `50` |
| `includeFullContent` | boolean | Add the full article plaintext in `fullContent`; `extract` remains the introductory summary | `false` |

\*Provide at least one non-blank value in `searchQueries` or `articleTitles`.

#### Mixed input example

```json
{
  "searchQueries": ["history of computing"],
  "articleTitles": ["Ada Lovelace", "Alan Turing"],
  "language": "en",
  "maxResultsPerSearch": 20,
  "includeFullContent": true
}
```

For exact pages only, omit `searchQueries` and provide `articleTitles`.

### Output example

Each article is returned as a JSON object:

```json
{
  "pageId": 1164,
  "title": "Artificial intelligence",
  "extract": "Artificial intelligence (AI) is the capability of computational systems to perform tasks typically associated with human intelligence...",
  "fullContent": "Artificial intelligence (AI), in its broadest sense, is intelligence exhibited by machines...",
  "url": "https://en.wikipedia.org/wiki/Artificial_intelligence",
  "wordCount": 26473,
  "size": 266568,
  "lastEdited": "2025-01-15T12:00:00Z",
  "thumbnail": "https://upload.wikimedia.org/wikipedia/commons/thumb/...",
  "categories": ["Artificial intelligence", "Computational fields of study"],
  "wikidataId": "Q11660",
  "description": "intelligence demonstrated by machines",
  "searchQuery": "artificial intelligence",
  "searchRank": 1,
  "matchedSearchQueries": ["artificial intelligence"],
  "scrapedAt": "2025-01-15T12:05:00.000Z"
}
```

#### Output fields

| Field | Type | Description |
|-------|------|-------------|
| `pageId` | number | Wikipedia internal page identifier |
| `title` | string | Article title |
| `extract` | string | Introductory summary (plain text, no HTML) |
| `fullContent` | string | Full article plaintext when `includeFullContent` is enabled; empty if Wikipedia has no full extract |
| `url` | string | Direct link to the Wikipedia article |
| `wordCount` | number | Total word count of the article |
| `size` | number | Article size in bytes |
| `lastEdited` | string | ISO timestamp of the last edit |
| `thumbnail` | string | URL to article thumbnail image (if available) |
| `categories` | string\[] | Wikipedia category labels associated with the page |
| `wikidataId` | string | Linked Wikidata entity ID, or an empty string when unavailable |
| `description` | string | Short page description, or an empty string when unavailable |
| `searchQuery` | string | First keyword that matched this page (search results only) |
| `searchRank` | number | One-based rank for the first matching keyword (search results only) |
| `requestedTitle` | string | First exact-title input that resolved to this page (exact results only) |
| `matchedSearchQueries` | string\[] | All keyword inputs that matched this deduplicated page |
| `requestedTitles` | string\[] | All exact-title inputs that resolved to this deduplicated page |
| `scrapedAt` | string | ISO timestamp when the data was extracted |

`wordCount` comes from Wikipedia's search response and is `0` for exact-title-only results. All other base fields retain the same types in both modes.

### Supported languages

Wikipedia Scraper supports all 300+ Wikipedia language editions. Use the standard language code:

| Code | Language | Articles |
|------|----------|----------|
| `en` | English | 6.9M+ |
| `de` | German | 2.9M+ |
| `fr` | French | 2.6M+ |
| `es` | Spanish | 2.0M+ |
| `ja` | Japanese | 1.4M+ |
| `ru` | Russian | 1.9M+ |
| `zh` | Chinese | 1.4M+ |
| `pt` | Portuguese | 1.1M+ |
| `it` | Italian | 1.8M+ |
| `ar` | Arabic | 1.2M+ |

Any valid Wikipedia language code works — see the [full list](https://meta.wikimedia.org/wiki/List_of_Wikipedias).

### How to scrape Wikipedia articles

1. Open [Wikipedia Scraper](https://apify.com/automation-lab/wikipedia-scraper) on Apify.
2. Enter search keywords in `searchQueries`, known page names in `articleTitles`, or both.
3. Set the `language` code (e.g., `en`, `de`, `fr`) for the Wikipedia edition you want.
4. Adjust `maxResultsPerSearch` to control how many articles each keyword returns (default: 50).
5. Optionally enable `includeFullContent` to add each full article in the `fullContent` field.
6. Click **Start** and wait for the scrape to finish.
7. Download the deduplicated articles as JSON, CSV, or Excel from the Dataset tab.

### API usage

#### Python

```python
from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")

run = client.actor("automation-lab/wikipedia-scraper").call(run_input={
    "searchQueries": ["climate change", "renewable energy"],
    "language": "en",
    "maxResultsPerSearch": 20,
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(f"{item['title']} — {item['wordCount']} words")
    print(f"  {item['url']}")
    print(f"  {item['extract'][:200]}...")
```

#### Node.js

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_API_TOKEN' });

const run = await client.actor('automation-lab/wikipedia-scraper').call({
    searchQueries: ['climate change', 'renewable energy'],
    language: 'en',
    maxResultsPerSearch: 20,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach(item => {
    console.log(`${item.title} — ${item.wordCount} words`);
    console.log(`  ${item.url}`);
});
```

#### REST API

```bash
curl -X POST "https://api.apify.com/v2/acts/automation-lab/wikipedia-scraper/runs?token=YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "searchQueries": ["artificial intelligence"],
    "language": "en",
    "maxResultsPerSearch": 10
  }'
```

### Integrations

Connect Wikipedia Scraper to hundreds of apps using built-in integrations:

- **Google Sheets** — export article data to spreadsheets
- **Slack / Microsoft Teams** — get notifications when scraping completes
- **Zapier / Make** — trigger workflows with scraped Wikipedia data
- **Amazon S3 / Google Cloud Storage** — store large datasets in cloud storage
- **Webhook** — send results to your own API endpoint

### Tips and best practices

1. **Use specific keywords** — more specific searches return more relevant results. "Quantum entanglement" is better than "quantum".
2. **Batch keywords efficiently** — combine related keywords in one run to save on startup costs.
3. **Language parameter** — set the language code to search non-English Wikipedias. Results, summaries, and URLs will all be in the selected language.
4. **Word count filtering** — use the `wordCount` field to filter out stub articles (typically < 500 words).
5. **Rate limits** — Wikipedia's API is generous but has rate limits. The scraper handles pagination and batching automatically.
6. **Choose the content depth** — the `extract` field always remains the short introductory summary. Enable `includeFullContent` when your workflow needs full article plaintext in `fullContent`; this opt-in mode makes one additional batched Wikipedia API request.
7. **Metadata scope** — the Actor returns categories, Wikidata IDs, and short descriptions when Wikipedia provides them. It does not return citations, external links, coordinates, HTML, or category pagination beyond the API's per-request limit.
8. **Exact titles and redirects** — use the page title, not a full URL. Wikipedia redirects are resolved automatically; missing titles are logged and skipped without discarding valid pages.
9. **Max 500 results per keyword** — this is a Wikipedia API limit. For broader coverage, use multiple related keywords.

### Data source, privacy, and support

- The Actor sends your search keywords, exact titles, language, and ordinary request metadata to Wikimedia's public MediaWiki API. It does not use AI, paid APIs, proxies, source accounts, or cookies.
- Results can include public biographical information when you request pages about people. Collect only data you need and follow applicable privacy and data-protection rules.
- The Actor stores no external cache or private copy. Run inputs, logs, and datasets remain in Apify storage under your account's retention and deletion settings.
- Wikipedia and Wikimedia are named only to identify the public data source. This Actor is not affiliated with or endorsed by the Wikimedia Foundation.
- For help, use the Actor's **Issues** tab on Apify and include a sanitized input and run ID.

### Legality

Scraping publicly available data is generally legal according to the [US Court of Appeals ruling](https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn) (HiQ Labs v. LinkedIn). This actor only accesses publicly available information and does not require authentication. Always review and comply with the target website's Terms of Service before scraping. For personal data, ensure compliance with GDPR, CCPA, and other applicable privacy regulations.

### FAQ

**Q: Does this scraper get the full article text?**
A: Yes, as an opt-in feature. Set `includeFullContent` to `true` and read the full article plaintext from `fullContent`. The existing `extract` field still contains only the introductory section. If Wikipedia returns no full extract for an article, `fullContent` is an empty string and the base article record is retained.

**Q: How fast is it?**
A: Very fast. Wikipedia's API is highly optimized. A typical run extracting 50 articles completes in under 5 seconds.

**Q: Does it need proxies?**
A: No. Wikipedia's API is open and does not block automated requests. The scraper identifies itself with a proper User-Agent header.

**Q: Can I search in multiple languages at once?**
A: Each run uses one language. To search multiple languages, run the scraper once per language.

### Use with Claude AI (MCP)

This actor is available as a tool in Claude AI through the Model Context Protocol (MCP). Add it to Claude Desktop, Cursor, Windsurf, or any MCP-compatible client.

#### Setup for Claude Code

```bash
claude mcp add --transport http apify "https://mcp.apify.com?tools=automation-lab/wikipedia-scraper"
```

#### Setup for Claude Desktop, Cursor, or VS Code

Add this to your MCP config file:

```json
{
    "mcpServers": {
        "apify": {
            "url": "https://mcp.apify.com?tools=automation-lab/wikipedia-scraper"
        }
    }
}
```

#### Example prompts

- "Search Wikipedia for articles about quantum computing and give me the summaries"
- "Fetch Wikipedia articles on these 5 historical events and compare their word counts"
- "Look up Wikipedia articles on machine learning in both English and German and extract the introductions"

Learn more in the [Apify MCP documentation](https://docs.apify.com/platform/integrations/mcp).

**The extract is truncated or too short.**
The `extract` field intentionally contains only the article's introductory section. Set `includeFullContent` to `true` to receive the full article plaintext in `fullContent` while preserving the concise introductory `extract`.

**I'm getting irrelevant results for my search query.**
Wikipedia's search API ranks by relevance, which may include loosely related articles. Use more specific keywords (e.g., "quantum entanglement" instead of "quantum") and reduce `maxResultsPerSearch` to get only the top matches. If you already know the page, put its title in `articleTitles` to fetch it directly.

### Other research and news scrapers on Apify

- [ArXiv Scraper](https://apify.com/automation-lab/arxiv-scraper) -- search and extract academic papers from ArXiv
- [CrossRef Scraper](https://apify.com/automation-lab/crossref-scraper) -- extract scholarly article metadata from CrossRef
- [OpenAlex Scraper](https://apify.com/automation-lab/openalex-scraper) -- search and extract academic research data from OpenAlex

# Changelog

This Actor's version history is a separate document: https://apify.com/automation-lab/wikipedia-scraper/changelog.md

# Actor input Schema

## `searchQueries` (type: `array`):

Keywords to search on Wikipedia. Each keyword runs a separate relevance-ranked search. Provide search keywords, exact article titles, or both.

## `articleTitles` (type: `array`):

Wikipedia article titles to fetch directly, such as Ada Lovelace or Analytical Engine. Redirects are resolved automatically.

## `language` (type: `string`):

Wikipedia language code (e.g. en, de, fr, es, ja).

## `maxResultsPerSearch` (type: `integer`):

Maximum articles per search keyword (1–500). This does not limit exact article titles.

## `includeFullContent` (type: `boolean`):

Fetch the full article as plain text in the fullContent field. The extract field remains the introductory summary.

## Actor input object example

```json
{
  "searchQueries": [
    "artificial intelligence"
  ],
  "language": "en",
  "maxResultsPerSearch": 20,
  "includeFullContent": false
}
```

# Actor output Schema

## `overview` (type: `string`):

Dataset containing the extracted Wikipedia articles.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "searchQueries": [
        "artificial intelligence"
    ],
    "language": "en",
    "maxResultsPerSearch": 20
};

// Run the Actor and wait for it to finish
const run = await client.actor("automation-lab/wikipedia-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "searchQueries": ["artificial intelligence"],
    "language": "en",
    "maxResultsPerSearch": 20,
}

# Run the Actor and wait for it to finish
run = client.actor("automation-lab/wikipedia-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "searchQueries": [
    "artificial intelligence"
  ],
  "language": "en",
  "maxResultsPerSearch": 20
}' |
apify call automation-lab/wikipedia-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,automation-lab/wikipedia-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Nka97qJs0YHXosATX/builds/baIP0CPgHALdtGOba/openapi.json
