Wikipedia Scraper

A powerful Wikipedia scraper that extracts structured data, infoboxes, tables, and article text from any Wikipedia page — optimized for RAG pipelines and AI training datasets.

Use Cases

• AI Training Data Collection — Extract clean Wikipedia text for LLM fine-tuning, embeddings, and knowledge distillation.
• Knowledge Graph Construction — Pull structured infoboxes and entity relationships for graph databases and ontologies.
• RAG Pipeline Development — Chunk and retrieve Wikipedia content for retrieval-augmented generation systems.
• Content Repurposing — Scrape articles in multiple languages for localization, translation, or multi-lingual NLP.
• Research Automation — Bulk-extract medical, legal, or technical Wikipedia content for domain-specific analysis.
• Structured Data Extraction — Get tabular data, metadata, and section hierarchies for data warehousing.

How It Works

1. Provide URLs — Input one or more Wikipedia article URLs to start.
2. Configure Extraction — Toggle infoboxes, section headings, main text, tables, and wikitext.
3. Run the Actor — The scraper fetches pages via the Wikipedia API or direct HTML parsing.
4. Get Structured Output — Results are delivered as clean JSON with article metadata, content, and structured fields.

Input Fields

Field	Type	Default	Description
startUrls	array	—	List of Wikipedia article URLs to scrape
extractInfobox	boolean	true	Extract structured infobox data
extractSectionHeadings	boolean	true	Extract section headings hierarchy
extractMainText	boolean	true	Extract article body text
maxSectionTextLength	integer	50000	Maximum characters per section
extractTables	boolean	false	Extract HTML tables from articles
extractWikitext	boolean	false	Extract raw wikitext source
maxInfoboxFields	integer	50	Maximum infobox fields to return
maxSectionDepth	integer	3	Maximum section nesting depth
maxTables	integer	20	Maximum tables to extract per page
useApiForInfobox	boolean	true	Use Wikipedia API for infobox parsing
useApiForSections	boolean	true	Use Wikipedia API for section parsing
language	string	"en"	Wikipedia language code (en, de, fr, etc.)
maxRequestsPerCrawl	integer	100	Maximum HTTP requests per crawl
maxConcurrency	integer	4	Maximum concurrent requests

Output

Each dataset item contains:

{
  "url": "https://en.wikipedia.org/wiki/Artificial_intelligence",
  "title": "Artificial intelligence",
  "pageId": 233488,
  "language": "en",
  "lastEdited": "2026-08-29T12:00:00Z",
  "infobox": {
    "type": "Field",
    "fields": {
      "Field of": "Computer science, Philosophy, Linguistics",
      "Key thinkers": "Alan Turing, John McCarthy, Marvin Minsky"
    }
  },
  "sectionHeadings": [
    { "level": 2, "title": "History", "text": "..." },
    { "level": 3, "title": "Early history", "text": "..." }
  ],
  "mainText": "Artificial intelligence (AI) is the intelligence of machines or software...",
  "tables": [
    { "headers": ["Year", "Event"], "rows": [["1950", "Turing Test"]] }
  ],
  "wikitext": "{{Infobox field|...}}",
  "categories": ["Artificial intelligence", "Emerging technologies"],
  "wordCount": 45230
}

Example Input

{
  "startUrls": [
    { "url": "https://en.wikipedia.org/wiki/Artificial_intelligence" },
    { "url": "https://en.wikipedia.org/wiki/Machine_learning" }
  ],
  "extractInfobox": true,
  "extractSectionHeadings": true,
  "extractMainText": true,
  "extractTables": false,
  "extractWikitext": false,
  "maxSectionTextLength": 50000,
  "maxInfoboxFields": 50,
  "maxSectionDepth": 3,
  "language": "en",
  "maxRequestsPerCrawl": 100,
  "maxConcurrency": 4
}

Integrations

• Google Sheets — Export scraped data directly to a spreadsheet.
• Airtable — Store structured Wikipedia data in Airtable.
• LangChain / LlamaIndex — Feed extracted text into RAG pipelines.
• Pinecone / Weaviate — Vectorize article content for semantic search.
• Zapier / Make — Automate workflows with scraped Wikipedia data.
• REST API — Access results via the Apify API for custom integrations.

Related Actors

• Web Scraper — General-purpose web scraping for any website.
• RAG Web Browser — Web search and content extraction for AI agents.
• Wikipedia API — Official Wikipedia API documentation.

FAQ

Can I scrape multiple Wikipedia languages?

Yes. Set the language field to any valid Wikipedia language code (e.g., de, fr, es, ja).

Is Wikipedia content free to use?

Yes. Wikipedia content is licensed under Creative Commons Attribution-ShareAlike 4.0.

Can I extract raw wikitext?

Yes. Enable extractWikitext: true in the input to get the raw MediaWiki source.

What's the difference between API and HTML parsing?

API mode uses the official Wikipedia API for faster, more structured extraction. HTML mode parses the rendered page directly.

How do I limit output size?

Use maxSectionTextLength, maxInfoboxFields, maxSectionDepth, and maxTables to control output volume.

Can I use this for RAG pipelines?

Absolutely. The actor is designed for RAG use cases — extract clean, chunked text ready for vectorization.

For AI Agents

{
  "actor": "Fg22T5bw1kCv0LJ9W",
  "input": {
    "startUrls": [{ "url": "https://en.wikipedia.org/wiki/YourTopic" }],
    "extractInfobox": true,
    "extractMainText": true,
    "language": "en"
  }
}

Pass startUrls as an array of objects with url key. Set extractInfobox: true for structured entity data. Use language code for multilingual extraction.

SEO Keywords

wikipedia scraper, wikipedia api, wikipedia dataset, wikipedia infobox scraper, wikipedia table scraper, wikipedia for RAG, wiki text extraction, wikipedia ai training data, wikipedia knowledge graph, wikipedia mult-l lingua scraper, wikipedia article extractor, wikipedia api alternative, scrape wikipedia data, wikipedia data mining, wikipedia wikipedia corpus, wikipedia embeddings data, wiki dump alternative, wikipedia structured data, wikipedia content downloader, wikipedia for llm