Wikipedia Scraper

Extract structured data from Wikipedia articles including full article text, infobox fields, section headings, tables, and raw wikitext. Built for researchers, educators, AI trainers, and knowledge graph builders who need clean, structured Wikipedia data at scale.

What does Wikipedia Scraper do?

Wikipedia Scraper extracts structured data from any Wikipedia article using a hybrid approach: it leverages the official Wikipedia REST API for structured data (infoboxes from wikitext templates, section hierarchy from TOC) and CheerioCrawler for HTML-based extraction (article text, tables, fallback parsing). This dual strategy ensures maximum data completeness and reliability.

Key features:

• 📄 Full article text – Clean plain text of the whole article plus per-section texts with TOC-style numbering
• 📋 Infobox extraction – Parses template parameters from raw wikitext for clean key-value pairs (supports both Infobox and Sidebar templates)
• 📑 Section headings – Full hierarchy (h2–h6) with anchor IDs and section numbers from API TOC
• 📊 Tables – Extracts wikitable and infobox tables with headers and typed rows
• 📝 Raw wikitext – Optional full article wikimarkup for custom processing
• 🌐 Multi-language – Supports all Wikipedia language editions (en, de, fr, es, etc.)
• ⚡ Fast & cheap – CheerioCrawler (no browser) + API calls = minimal compute units
• 🔒 Proxy rotation – Routes through Apify proxies when available

Why use Wikipedia Scraper?

Use Case	Benefit
AI/ML training data	Clean, structured JSON ready for fine-tuning or RAG pipelines
Knowledge graphs	Infobox fields map directly to entity attributes (birth date, occupation, etc.)
Research & education	Batch-extract thousands of articles for literature reviews or curricula
Data enrichment	Augment your datasets with authoritative Wikipedia metadata
Content analysis	Section hierarchy enables document structure analysis

How to use Wikipedia Scraper

1. Add URLs – Paste Wikipedia article URLs (any language) into the Start URLs field
2. Choose fields – Toggle extraction options: Article Text, Infobox, Sections, Tables, Wikitext
3. Set limits – Configure max items per field (infobox fields, section depth, tables)
4. Run – Click Start and download results as JSON, CSV, Excel, or via API

Input Example

{
    "startUrls": [
        { "url": "https://en.wikipedia.org/wiki/Albert_Einstein" },
        { "url": "https://en.wikipedia.org/wiki/Quantum_mechanics" }
    ],
    "extractMainText": true,
    "extractInfobox": true,
    "extractSectionHeadings": true,
    "extractTables": true,
    "extractWikitext": false,
    "maxInfoboxFields": 50,
    "maxSectionDepth": 3,
    "maxSectionTextLength": 50000,
    "maxTables": 20,
    "language": "en"
}

Output

Each article produces one dataset item with this structure:

{
  "url": "https://en.wikipedia.org/wiki/Albert_Einstein",
  "title": "Albert Einstein",
  "pageTitle": "Albert_Einstein",
  "language": "en",
  "infobox": {
    "birth_date": "14 March 1879",
    "birth_place": "Ulm, Kingdom of Württemberg, German Empire",
    "death_date": "18 April 1955",
    "fields": "Physics",
    "known_for": "General relativity, Special relativity, Photoelectric effect, E=mc²..."
  },
  "infoboxTemplates": [
    {
      "templateName": "scientist",
      "fields": { "birth_date": "...", "death_date": "...", ... }
    }
  ],
  "sectionHeadings": [
    { "level": 1, "title": "Life and career", "id": "Life_and_career", "number": "1" },
    { "level": 2, "title": "Childhood, youth and education", "id": "Childhood,_youth_and_education", "number": "1.1" }
  ],
  "sections": [
    { "level": 0, "title": "", "number": "", "text": "Albert Einstein (14 March 1879 – 18 April 1955) was..." },
    { "level": 1, "title": "Life and career", "number": "1", "text": "..." }
  ],
  "mainText": "Albert Einstein (14 March 1879 – 18 April 1955) was... \n\n Life and career...",
  "textStats": { "words": 8377, "chars": 52723 },
  "tables": [
    {
      "caption": "Nobel Prize laureates in Physics",
      "headers": ["Year", "Laureate", "Country", "Rationale"],
      "rows": [
        { "Year": "1921", "Laureate": "Albert Einstein", "Country": "Germany", "Rationale": "..." }
      ]
    }
  ],
  "wikitext": "{{Short description|...}}\n{{Infobox scientist\n| image = ...\n...",
  "userType": "PAID",
  "freeLimit": 10,
  "crawledAt": "2026-08-22T10:30:00.000Z"
}

Data Table

Field	Type	Description
url	string	Source article URL
title	string	Display title from HTML
pageTitle	string	URL-encoded page title
language	string	Wikipedia language code
infobox	object	Flattened key-value pairs from all infobox templates
infoboxTemplates	array	Individual templates with names and fields
sectionHeadings	array[]	Hierarchical sections with level, title, anchor, number
sections	array[]	Per-section plain text with level, title, and TOC number
mainText	string	Full article text (lead + all sections joined)
textStats	object	Word and character counts of mainText
tables	array[]	Tables with caption, headers, and row objects
wikitext	string	Raw wikimarkup (optional)
crawledAt	ISO date	Extraction timestamp

Pricing / Cost Estimation

Plan	Cost per 1,000 articles*
Free tier	10 articles free, then upgrade
Apify Starter ($49/mo)	~$0.50 (≈2,000 CU)
Apify Scale ($499/mo)	~$0.10 (included CU)

*Estimates based on typical article (~50 KB HTML + 2 API calls). Actual cost depends on article size, table count, and proxy usage. Free users limited to 10 articles per run.

Tips & Advanced Options

• Use API mode (default) – Faster and more reliable for infoboxes/sections
• Increase maxSectionDepth to 4–5 for deep articles (biographies, histories)
• Enable extractWikitext only when needed – adds ~20 KB per article
• Batch URLs – Submit 100+ URLs per run for best throughput
• Schedule runs – Use Apify Scheduler for daily/weekly knowledge base updates

FAQ, Disclaimers & Support

⚠️ Legal: Wikipedia content is licensed under CC-BY-SA 3.0. You may scrape and reuse data with attribution. See Wikimedia Terms of Use.

🔧 Limitations:

• Sitemap access is restricted by Wikimedia (no bulk discovery via sitemap)
• Very long articles (>2 MB) may hit request timeouts
• Some infobox templates use complex nested structures not fully flattened
• Overview-style articles (e.g. "Artificial intelligence") have no true infobox – their navigation sidebar is extracted as title / content1…contentN fields instead

🐛 Issues & Feature Requests: GitHub Issues or Apify Console Issues tab

💡 Custom Solutions: Need Wikidata integration, category crawling, or revision history? Contact us for custom Actor development.

---

Built with Crawlee + Apify SDK • Runs on Apify Platform with residential proxy rotation