Wikipedia to RAG — Bulk Article Scraper for AI & Vector Databases

Turn any Wikipedia topic into clean, chunked Markdown ready for RAG pipelines, vector databases, and LLM fine-tuning — no API key required.

Why use this Actor?

Wikipedia is the world's largest free knowledge base with 60+ million articles. This Actor automates:

• 🔍 Multi-query search — find relevant articles by keyword
• 📄 Clean text extraction — removes infoboxes, references, navbars
• ✂️ Smart chunking — overlapping chunks optimized for RAG retrieval
• 📦 Vector DB ready — outputs JSONL with text + metadata fields

Use Cases

• Building domain-specific RAG knowledge bases
• Creating AI training datasets
• Populating Pinecone / Weaviate / Chroma / Qdrant
• Research automation and summarization pipelines

Input

{
  "searchQueries": ["machine learning", "neural networks", "transformer model"],
  "articleUrls": ["https://en.wikipedia.org/wiki/BERT_(language_model)"],
  "language": "en",
  "maxArticles": 20,
  "chunkSize": 400,
  "chunkOverlap": 40,
  "includeIntroOnly": false,
  "outputMarkdown": true
}

Parameter	Type	Default	Description
searchQueries	string[]	[]	Keywords to search Wikipedia
articleUrls	string[]	[]	Direct Wikipedia article URLs
language	string	"en"	Wikipedia language code (en, zh, de, fr, ja, ...)
maxArticles	number	20	Maximum number of articles to process
chunkSize	number	400	Words per chunk
chunkOverlap	number	40	Overlapping words between chunks
includeIntroOnly	boolean	false	Only extract the article introduction
outputMarkdown	boolean	true	Save full markdown to Key-Value store

Output

Dataset (JSONL chunks)

Each row contains a text chunk ready for embedding:

{
  "text": "A transformer is a deep learning architecture...",
  "metadata": {
    "title": "Transformer (deep learning architecture)",
    "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning_architecture)",
    "chunkIndex": 0,
    "totalChunks": 12,
    "language": "en",
    "source": "wikipedia",
    "scrapedAt": "2025-01-01T00:00:00.000Z"
  }
}

Key-Value Store

• wiki-{title}.md — full article as clean Markdown
• output.jsonl — all chunks as downloadable JSONL file

Integration Examples

Pinecone

import json
from pinecone import Pinecone
import openai

pc = Pinecone(api_key="YOUR_KEY")
index = pc.Index("wikipedia-knowledge")

# Download output.jsonl from Actor run
with open("output.jsonl") as f:
    for line in f:
        item = json.loads(line)
        embedding = openai.embeddings.create(
            input=item["text"], model="text-embedding-3-small"
        ).data[0].embedding
        index.upsert([(item["metadata"]["url"] + str(item["metadata"]["chunkIndex"]),
                       embedding, item["metadata"])])

LangChain

from langchain.vectorstores import Chroma
from langchain.embeddings import OpenAIEmbeddings
from langchain.schema import Document
import json

docs = []
with open("output.jsonl") as f:
    for line in f:
        item = json.loads(line)
        docs.append(Document(page_content=item["text"], metadata=item["metadata"]))

vectorstore = Chroma.from_documents(docs, OpenAIEmbeddings())

Multilingual Support

Set the language parameter to scrape any Wikipedia language edition:

Code	Language
en	English
zh	Chinese
de	German
fr	French
ja	Japanese
es	Spanish
ko	Korean

FAQ

Is this legal? Yes. Wikipedia content is published under the Creative Commons Attribution-ShareAlike license and explicitly allows programmatic access. The Actor respects rate limits with polite delays.

Does it require an API key? No. The Wikipedia REST API is completely free and requires no authentication.

How many articles can I scrape? The FREE Apify plan supports up to hundreds of articles per run. For bulk scraping of thousands of articles, consider upgrading.