Wikipedia Scraper

Ask MediaWiki for an article and it hands back the intro paragraph, not the article. The flag that changes this is presence-based, so sending exintro=0 still gets you the intro; the only way off is to omit it entirely, which is what the fullText checkbox does here.

The rest comes through the same official MediaWiki Action API, as JSON instead of HTML. Two modes: search by keyword, or fetch exact page titles. Any language edition works. Wikipedia publishes this API for anyone to use, so there's no key to get and no anti-bot to fight.

The two modes

Search — set searchQuery. Returns matching articles with title, pageid, url, a plain-text snippet (the API's HTML markup is stripped for you), wordcount, size and timestamp. Pages 50 at a time up to maxItems.

Page data — set pageTitles to a list of exact article titles. Returns title, pageid, url, a plain-text extract, a thumbnail URL and categories. Titles go 50 per batch. Turn on fullText and extract holds the whole article instead of just the intro.

Set both and search wins. Pick one.

Input

Field	Default	Notes
searchQuery	—	Keywords, e.g. machine learning.
pageTitles	[]	Exact titles, e.g. ["Apify", "Web scraping"]. Titles have to match — Wikipedia's API doesn't guess.
fullText	false	Page-titles mode only. Whole article vs. just the intro.
language	en	Edition code: en, fr, de, es, ja. Picks the host {lang}.wikipedia.org.
maxItems	50	Up to 5000.
proxyConfiguration	off	Wikipedia has no anti-bot, so this is genuinely optional.

{ "searchQuery": "machine learning", "language": "en", "maxItems": 30 }

{ "pageTitles": ["Apify", "Web scraping"], "fullText": true, "language": "en" }

What each mode returns

Field	Mode	Notes
title	both	Article title.
pageid	both	Stable page id. Used to deduplicate, and the right key to join on.
url	both	Canonical article URL.
snippet	search	Plain-text match snippet, HTML stripped.
wordcount, size, timestamp	search	Word count, byte size, last-edit time.
extract	page	Plain-text body — intro by default, whole article with fullText.
thumbnail	page	Lead image URL up to 400px, when the page has one.
categories	page	Visible category names. Hidden maintenance categories are excluded.

Rows are deduplicated by pageid. A search that matches nothing, or a title Wikipedia doesn't have, produces an uncharged diagnostic row with an errorCode rather than a silent empty dataset.

Limits

• Search mode has no extract. The two modes return different field sets because the API does. If you want article bodies, you need titles, not a keyword search.
• Page-titles mode needs exact titles. Web scraping works, web scraping tools doesn't. Run a search first to resolve names into titles, then feed those titles back in.
• categories excludes hidden categories — the maintenance ones like "Articles with dead external links". You get the ones a reader sees.
• No infoboxes, no references, no wikitext. extract is flattened plain text. If you need structured infobox values, Wikidata is the right source, not this.

Billing

$1.00 per 1,000 pages ($0.001 each), and nothing charged for starting a run. Same price on the free plan as on a paid one.

A row is one article, in either mode, intro or whole body. A 40,000-word article costs the same $0.001 as a two-line stub. And with no start fee, a run that finds nothing costs you nothing.

Two patterns worth knowing

Corpus building. pageTitles with fullText: true hands back plain text, 50 titles per batch. The markup, the reference markers and the infobox are already gone. There's no parsing step left on your side.

Name resolution. Push a messy list of companies, people or places through search mode and keep the title and pageid that come back. Whatever spelling you had is now Wikipedia's own. Feed those titles back in as pageTitles and the second pass hands you a thumbnail as well.

Two smaller things. categories gives you a subject graph without crawling category pages yourself, and search mode returns timestamp and size on every hit, so running the same query a week apart and diffing shows which articles actually moved. Changing language points the same title list at the French, German or Japanese edition.

Notes

Calls https://{language}.wikipedia.org/w/api.php. Per Wikimedia's user-agent policy the actor always sends a descriptive User-Agent with a contact address.