# Wikipedia Scraper \[$0.7💰 / 1k] (`ahmed_jasarevic/wikipedia-scraper`) Actor

Extract structured data from Wikipedia articles only $0.7/1k: infobox fields, section headings, tables, and raw wikitext. Uses Wikipedia REST API for structured data. Supports all language editions. Ideal for researchers, educators, AI trainers, and knowledge graph builders

- **URL**: https://apify.com/ahmed\_jasarevic/wikipedia-scraper.md
- **Developed by:** [Ahmed Jasarevic](https://apify.com/ahmed_jasarevic) (community)
- **Categories:** Developer tools, AI, Other
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.70 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Wikipedia Scraper

Extract structured data from Wikipedia articles including **full article text**, **infobox fields**, **section headings**, **tables**, and **raw wikitext**. Built for researchers, educators, AI trainers, and knowledge graph builders who need clean, structured Wikipedia data at scale.

[![Try on Apify](https://apify.com/actor-badge.svg)](https://console.apify.com/actors/wikipedia-scraper)

### What does Wikipedia Scraper do?

Wikipedia Scraper extracts structured data from any Wikipedia article using a **hybrid approach**: it leverages the official **Wikipedia REST API** for structured data (infoboxes from wikitext templates, section hierarchy from TOC) and **CheerioCrawler** for HTML-based extraction (article text, tables, fallback parsing). This dual strategy ensures maximum data completeness and reliability.

**Key features:**

- 📄 **Full article text** – Clean plain text of the whole article plus per-section texts with TOC-style numbering
- 📋 **Infobox extraction** – Parses template parameters from raw wikitext for clean key-value pairs (supports both `Infobox` and `Sidebar` templates)
- 📑 **Section headings** – Full hierarchy (h2–h6) with anchor IDs and section numbers from API TOC
- 📊 **Tables** – Extracts wikitable and infobox tables with headers and typed rows
- 📝 **Raw wikitext** – Optional full article wikimarkup for custom processing
- 🌐 **Multi-language** – Supports all Wikipedia language editions (en, de, fr, es, etc.)
- ⚡ **Fast & cheap** – CheerioCrawler (no browser) + API calls = minimal compute units
- 🔒 **Proxy rotation** – Routes through Apify proxies when available

### Why use Wikipedia Scraper?

| Use Case                 | Benefit                                                                         |
| ------------------------ | ------------------------------------------------------------------------------- |
| **AI/ML training data**  | Clean, structured JSON ready for fine-tuning or RAG pipelines                   |
| **Knowledge graphs**     | Infobox fields map directly to entity attributes (birth date, occupation, etc.) |
| **Research & education** | Batch-extract thousands of articles for literature reviews or curricula         |
| **Data enrichment**      | Augment your datasets with authoritative Wikipedia metadata                     |
| **Content analysis**     | Section hierarchy enables document structure analysis                           |

### How to use Wikipedia Scraper

1. **Add URLs** – Paste Wikipedia article URLs (any language) into the **Start URLs** field
2. **Choose fields** – Toggle extraction options: Article Text, Infobox, Sections, Tables, Wikitext
3. **Set limits** – Configure max items per field (infobox fields, section depth, tables)
4. **Run** – Click **Start** and download results as JSON, CSV, Excel, or via API

#### Input Example

```json
{
    "startUrls": [
        { "url": "https://en.wikipedia.org/wiki/Albert_Einstein" },
        { "url": "https://en.wikipedia.org/wiki/Quantum_mechanics" }
    ],
    "extractMainText": true,
    "extractInfobox": true,
    "extractSectionHeadings": true,
    "extractTables": true,
    "extractWikitext": false,
    "maxInfoboxFields": 50,
    "maxSectionDepth": 3,
    "maxSectionTextLength": 50000,
    "maxTables": 20,
    "language": "en"
}
```

### Output

Each article produces one dataset item with this structure:

```json
{
  "url": "https://en.wikipedia.org/wiki/Albert_Einstein",
  "title": "Albert Einstein",
  "pageTitle": "Albert_Einstein",
  "language": "en",
  "infobox": {
    "birth_date": "14 March 1879",
    "birth_place": "Ulm, Kingdom of Württemberg, German Empire",
    "death_date": "18 April 1955",
    "fields": "Physics",
    "known_for": "General relativity, Special relativity, Photoelectric effect, E=mc²..."
  },
  "infoboxTemplates": [
    {
      "templateName": "scientist",
      "fields": { "birth_date": "...", "death_date": "...", ... }
    }
  ],
  "sectionHeadings": [
    { "level": 1, "title": "Life and career", "id": "Life_and_career", "number": "1" },
    { "level": 2, "title": "Childhood, youth and education", "id": "Childhood,_youth_and_education", "number": "1.1" }
  ],
  "sections": [
    { "level": 0, "title": "", "number": "", "text": "Albert Einstein (14 March 1879 – 18 April 1955) was..." },
    { "level": 1, "title": "Life and career", "number": "1", "text": "..." }
  ],
  "mainText": "Albert Einstein (14 March 1879 – 18 April 1955) was... \n\n Life and career...",
  "textStats": { "words": 8377, "chars": 52723 },
  "tables": [
    {
      "caption": "Nobel Prize laureates in Physics",
      "headers": ["Year", "Laureate", "Country", "Rationale"],
      "rows": [
        { "Year": "1921", "Laureate": "Albert Einstein", "Country": "Germany", "Rationale": "..." }
      ]
    }
  ],
  "wikitext": "{{Short description|...}}\n{{Infobox scientist\n| image = ...\n...",
  "userType": "PAID",
  "freeLimit": 10,
  "crawledAt": "2026-08-22T10:30:00.000Z"
}
```

### Data Table

| Field              | Type     | Description                                              |
| ------------------ | -------- | -------------------------------------------------------- |
| `url`              | string   | Source article URL                                       |
| `title`            | string   | Display title from HTML                                  |
| `pageTitle`        | string   | URL-encoded page title                                   |
| `language`         | string   | Wikipedia language code                                  |
| `infobox`          | object   | Flattened key-value pairs from all infobox templates     |
| `infoboxTemplates` | array    | Individual templates with names and fields               |
| `sectionHeadings`  | array\[]  | Hierarchical sections with level, title, anchor, number  |
| `sections`         | array\[]  | Per-section plain text with level, title, and TOC number |
| `mainText`         | string   | Full article text (lead + all sections joined)           |
| `textStats`        | object   | Word and character counts of `mainText`                  |
| `tables`           | array\[]  | Tables with caption, headers, and row objects            |
| `wikitext`         | string   | Raw wikimarkup (optional)                                |
| `crawledAt`        | ISO date | Extraction timestamp                                     |

### Pricing / Cost Estimation

| Plan                       | Cost per 1,000 articles\*       |
| -------------------------- | ------------------------------ |
| **Free tier**              | 10 articles free, then upgrade |
| **Apify Starter ($49/mo)** | ~$0.50 (≈2,000 CU)             |
| **Apify Scale ($499/mo)**  | ~$0.10 (included CU)           |

\*Estimates based on typical article (~50 KB HTML + 2 API calls). Actual cost depends on article size, table count, and proxy usage. Free users limited to 10 articles per run.

### Tips & Advanced Options

- **Use API mode** (default) – Faster and more reliable for infoboxes/sections
- **Increase `maxSectionDepth`** to 4–5 for deep articles (biographies, histories)
- **Enable `extractWikitext`** only when needed – adds ~20 KB per article
- **Batch URLs** – Submit 100+ URLs per run for best throughput
- **Schedule runs** – Use Apify Scheduler for daily/weekly knowledge base updates

### FAQ, Disclaimers & Support

⚠️ **Legal**: Wikipedia content is licensed under CC-BY-SA 3.0. You may scrape and reuse data with attribution. See [Wikimedia Terms of Use](https://wikimediafoundation.org/wiki/Terms_of_Use).

🔧 **Limitations**:

- Sitemap access is restricted by Wikimedia (no bulk discovery via sitemap)
- Very long articles (>2 MB) may hit request timeouts
- Some infobox templates use complex nested structures not fully flattened
- Overview-style articles (e.g. "Artificial intelligence") have no true infobox – their navigation sidebar is extracted as `title` / `content1`…`contentN` fields instead

🐛 **Issues & Feature Requests**: [GitHub Issues](https://github.com/your-org/wikipedia-scraper/issues) or Apify Console **Issues** tab

💡 **Custom Solutions**: Need Wikidata integration, category crawling, or revision history? [Contact us](https://apify.com/contact) for custom Actor development.

***

**Built with** [Crawlee](https://crawlee.dev/) + [Apify SDK](https://docs.apify.com/sdk/js) • **Runs on** Apify Platform with residential proxy rotation

# Actor input Schema

## `startUrls` (type: `array`):

URLs to start scraping from (Wikipedia article URLs).

## `extractInfobox` (type: `boolean`):

Whether to extract infobox key-value pairs from the top-right infobox.

## `extractSectionHeadings` (type: `boolean`):

Whether to extract section headings (h2, h3) and their hierarchy.

## `extractMainText` (type: `boolean`):

Whether to extract the full article text plus per-section text of each page.

## `maxSectionTextLength` (type: `integer`):

Maximum number of characters stored per section's text (0 = unlimited).

## `extractTables` (type: `boolean`):

Whether to extract wikitext tables from the article content.

## `extractWikitext` (type: `boolean`):

Whether to extract the raw wikitext of the article.

## `maxInfoboxFields` (type: `integer`):

Maximum number of infobox fields to extract (0 = all).

## `maxSectionDepth` (type: `integer`):

Maximum section heading depth to extract (1 = h2 only, 2 = h2+h3, etc.).

## `maxTables` (type: `integer`):

Maximum number of tables to extract (0 = all).

## `useApiForInfobox` (type: `boolean`):

Use Wikipedia REST API to parse infobox from wikitext (more accurate). Disable to use HTML parsing only.

## `useApiForSections` (type: `boolean`):

Use Wikipedia REST API to get section hierarchy from TOC (more accurate). Disable to use HTML parsing only.

## `language` (type: `string`):

Wikipedia language edition code (e.g., 'en', 'de', 'fr', 'es', 'ja', 'zh').

## `maxRequestsPerCrawl` (type: `integer`):

Maximum number of pages to scrape (0 = unlimited).

## `maxConcurrency` (type: `integer`):

Maximum concurrent requests (higher = faster but more likely to hit rate limits).

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"
    }
  ],
  "extractInfobox": true,
  "extractSectionHeadings": true,
  "extractMainText": true,
  "maxSectionTextLength": 50000,
  "extractTables": true,
  "extractWikitext": false,
  "maxInfoboxFields": 50,
  "maxSectionDepth": 3,
  "maxTables": 20,
  "useApiForInfobox": true,
  "useApiForSections": true,
  "language": "en",
  "maxRequestsPerCrawl": 100,
  "maxConcurrency": 4
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("ahmed_jasarevic/wikipedia-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://en.wikipedia.org/wiki/Artificial_intelligence" }] }

# Run the Actor and wait for it to finish
run = client.actor("ahmed_jasarevic/wikipedia-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"
    }
  ]
}' |
apify call ahmed_jasarevic/wikipedia-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,ahmed_jasarevic/wikipedia-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Fg22T5bw1kCv0LJ9W/builds/bkTLTzrtdayXdEq67/openapi.json