Wikipedia Articles Scraper

Wikipedia articles with daily traffic and Wikidata facts attached — search the encyclopedia or look up titles directly, in any language edition. HTTP-only, no API key, no login, no browser.

What makes this different from a plain article dump: every row can carry the article's pageview history (how much attention it actually gets) and its Wikidata claims, joined automatically through the wikibase_item the summary endpoint already returns.

Four upstream quirks it corrects

1. Redirects are silent — and your own input is the only evidence

/api/rest_v1/page/summary/AI
  -> title "Artificial intelligence", pageid 1164, type "standard"

Nothing in that response says a redirect happened. It is shaped identically to a direct hit. So a run over 500 titles comes back with some rows describing a different article than the one requested, and nothing distinguishes them unless you compare your own input to the output.

Every row therefore carries requestedTitle, resolvedTitle and a computed wasRedirected, and the summary counts them. AI → Artificial intelligence is flagged; Machine learning → Machine learning is not.

2. Not every 200 is an article

/summary/ML returns HTTP 200 with type: "disambiguation" — a list of things called ML, not a subject. Its extract reads exactly like a real summary and will be treated as one unless you check. Published as pageType and flagged via isDisambiguation.

3. totalhits is honest about matches, not about reach

srsearch="machine learning"  ->  totalhits 57,800
sroffset 0 / 100 / 1,000 / 5,000  ->  results
sroffset 10,000  ->  HTTP 200, zero results, and a NAMED error:
    {"code": "cirrussearch-offset-too-large",
     "info": "Could not retrieve results. Up to 10000 search results are ..."}

Retrieval is hard-capped at 10,000 no matter what the hit count says. Wikipedia is unusually honest here — it names the limit instead of returning an empty page and letting you guess — so the actor stops on that code explicitly and reports searchDepthCapped. A query claiming 57,800 hits never looks like it was walked to the end.

4. Search pages overlap, and the live index shifts underneath you

Two runs of the same 120-row walk produced offsets [0,50,100,120,122] with 6 duplicates and [0,50,100,121,124] with 2, ending at 120 and 119 rows. Advancing the offset by the number of results actually returned is still not enough — the same article comes back on two pages.

So dedup by title is mandatory (reported as duplicateArticlesDropped), and maxResults is a ceiling rather than a promise: a walk can finish a row or two short when overlap consumes a page.

Bonus: a pageviews 404 blames the dates

Asking for views of an article that does not exist returns 404 with "The date(s) you used are valid, but we either do not have data for those date(s), or the project you asked for is not loaded yet." The dates were fine and the project was loaded — the title was not real. Reported as "no data for this article and range", and it degrades that one field rather than the row.

Output

One SEARCH_SUMMARY per run, one ARTICLE per article, one ERROR per title that could not be fetched.

ARTICLE: requestedTitle, resolvedTitle, wasRedirected, pageId, pageType, isDisambiguation, description, extract, articleUrl, thumbnailUrl, lastRevisionAt, wikidataId, resultRank, plus — when enabled — pageviewsTotal, pageviewsDailyMean, pageviewsDaily, wikidataLabel, wikidataClaimCount and wikidataInstanceOf.

A note on politeness

Wikimedia asks automated clients to send a descriptive User-Agent identifying the tool and a way to reach its operator, and reserves the right to rate-limit generic browser strings. This actor sends one because they ask — not because anything blocks without it. For the same reason the proxy is offered but off by default: routing this traffic through a shared residential pool would make the actor harder for Wikimedia to attribute and rate-limit fairly, which is precisely what their policy asks clients not to do.

Limits

• Search retrieval stops at 10,000 results however many matched.
• maxResults is a ceiling; overlap can end a walk slightly short.
• Pageview history is capped at 365 days per run and lags by one day.
• Titles are spelling-sensitive past the first letter — but a title that merely redirects resolves silently, which is quirk 1.