# Wikipedia Articles Scraper (`scrapyx/wikipedia-articles-scraper`) Actor

Wikipedia articles with daily pageview history and Wikidata facts attached. Flags the silent redirects Wikipedia never reports - asking for 'AI' returns 'Artificial intelligence' with nothing saying so - and reports the 10,000-result retrieval cap that sits under a far larger hit count.

- **URL**: https://apify.com/scrapyx/wikipedia-articles-scraper.md
- **Developed by:** [Ibnu Adzim](https://apify.com/scrapyx) (community)
- **Categories:** AI, Education, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.40 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Wikipedia Articles Scraper

Wikipedia articles with **daily traffic** and **Wikidata facts** attached —
search the encyclopedia or look up titles directly, in any language edition.
HTTP-only, no API key, no login, no browser.

What makes this different from a plain article dump: every row can carry the
article's **pageview history** (how much attention it actually gets) and its
**Wikidata claims**, joined automatically through the `wikibase_item` the
summary endpoint already returns.

### Four upstream quirks it corrects

#### 1. Redirects are **silent** — and your own input is the only evidence

```
/api/rest_v1/page/summary/AI
  -> title "Artificial intelligence", pageid 1164, type "standard"
```

Nothing in that response says a redirect happened. It is shaped **identically**
to a direct hit. So a run over 500 titles comes back with some rows describing
a *different* article than the one requested, and nothing distinguishes them
unless you compare your own input to the output.

Every row therefore carries `requestedTitle`, `resolvedTitle` and a computed
`wasRedirected`, and the summary counts them. `AI` → `Artificial intelligence`
is flagged; `Machine learning` → `Machine learning` is not.

#### 2. Not every 200 is an article

`/summary/ML` returns HTTP 200 with `type: "disambiguation"` — a list of
things called ML, not a subject. Its `extract` reads exactly like a real
summary and will be treated as one unless you check. Published as `pageType`
and flagged via `isDisambiguation`.

#### 3. `totalhits` is honest about **matches**, not about **reach**

```
srsearch="machine learning"  ->  totalhits 57,800
sroffset 0 / 100 / 1,000 / 5,000  ->  results
sroffset 10,000  ->  HTTP 200, zero results, and a NAMED error:
    {"code": "cirrussearch-offset-too-large",
     "info": "Could not retrieve results. Up to 10000 search results are ..."}
```

Retrieval is hard-capped at 10,000 no matter what the hit count says.
Wikipedia is unusually honest here — it *names* the limit instead of returning
an empty page and letting you guess — so the actor stops on that code
explicitly and reports `searchDepthCapped`. A query claiming 57,800 hits never
looks like it was walked to the end.

#### 4. Search pages **overlap**, and the live index shifts underneath you

Two runs of the same 120-row walk produced offsets `[0,50,100,120,122]` with
**6 duplicates** and `[0,50,100,121,124]` with **2**, ending at 120 and 119
rows. Advancing the offset by the number of results actually returned is still
not enough — the same article comes back on two pages.

So dedup by title is mandatory (reported as `duplicateArticlesDropped`), and
`maxResults` is a **ceiling rather than a promise**: a walk can finish a row or
two short when overlap consumes a page.

#### Bonus: a pageviews 404 blames the dates

Asking for views of an article that does not exist returns 404 with *"The
date(s) you used are valid, but we either do not have data for those date(s),
or the project you asked for is not loaded yet."* The dates were fine and the
project was loaded — the title was not real. Reported as "no data for this
article and range", and it degrades that one field rather than the row.

### Output

One `SEARCH_SUMMARY` per run, one `ARTICLE` per article, one `ERROR` per title
that could not be fetched.

`ARTICLE`: `requestedTitle`, `resolvedTitle`, `wasRedirected`, `pageId`,
`pageType`, `isDisambiguation`, `description`, `extract`, `articleUrl`,
`thumbnailUrl`, `lastRevisionAt`, `wikidataId`, `resultRank`, plus — when
enabled — `pageviewsTotal`, `pageviewsDailyMean`, `pageviewsDaily`,
`wikidataLabel`, `wikidataClaimCount` and `wikidataInstanceOf`.

### A note on politeness

Wikimedia asks automated clients to send a **descriptive User-Agent**
identifying the tool and a way to reach its operator, and reserves the right to
rate-limit generic browser strings. This actor sends one because they ask —
not because anything blocks without it. For the same reason the proxy is
offered but **off by default**: routing this traffic through a shared
residential pool would make the actor harder for Wikimedia to attribute and
rate-limit fairly, which is precisely what their policy asks clients not to do.

### Limits

- Search retrieval stops at 10,000 results however many matched.
- `maxResults` is a ceiling; overlap can end a walk slightly short.
- Pageview history is capped at 365 days per run and lags by one day.
- Titles are spelling-sensitive past the first letter — but a title that merely
  *redirects* resolves silently, which is quirk 1.

# Actor input Schema

## `mode` (type: `string`):

search = search the encyclopedia. articles = look up titles you name.

## `searchTerm` (type: `string`):

For mode='search'. Note Wikipedia reports the true match count but caps RETRIEVAL at 10,000 results — a query matching 57,800 articles can only ever hand back 10,000 of them, and it says so with a named error rather than an empty page.

## `titles` (type: `array`):

For mode='articles'. A title like `Machine learning` or a full wikipedia.org/wiki/ URL. NOTE that a title which merely redirects resolves SILENTLY — asking for 'AI' returns 'Artificial intelligence' with nothing in the response saying so. Every row carries requestedTitle, resolvedTitle and wasRedirected.

## `language` (type: `string`):

Wikipedia language code — en, de, fr, id, es and so on.

## `includePageviews` (type: `boolean`):

Adds how much traffic each article actually gets — daily series, total and mean — at one extra request per article. This is the field that turns an article dump into an attention dataset.

## `includeWikidata` (type: `boolean`):

Adds the linked Wikidata entity's label, description, claim count and instance-of types. The join key is free — the summary response already carries it — but the lookup costs one extra request per article.

## `pageviewDays` (type: `integer`):

Up to 365. Data lags by one day.

## `maxResults` (type: `integer`):

A CEILING, not a promise: search pages overlap and Wikipedia's live index shifts between requests, so a capped walk can finish a row or two short. Set 0 for unlimited (still bounded by the 10,000 retrieval cap).

## `maxConcurrency` (type: `integer`):

Article lookups in flight at once. Keep it modest — Wikimedia asks automated clients to be identifiable and well-behaved.

## `minRequestInterval` (type: `integer`):

Politeness pacing shared across all workers. 0 uses the built-in default.

## `proxyConfiguration` (type: `object`):

OFF by default, deliberately. These are public APIs with no WAF, and Wikimedia's policy asks automated clients to be identifiable — routing this traffic through a shared residential pool would make it harder for them to attribute and rate-limit fairly.

## Actor input object example

```json
{
  "mode": "search",
  "searchTerm": "machine learning",
  "titles": [
    "Machine learning",
    "https://en.wikipedia.org/wiki/Artificial_intelligence"
  ],
  "language": "en",
  "includePageviews": false,
  "includeWikidata": false,
  "pageviewDays": 30,
  "maxResults": 50,
  "maxConcurrency": 4,
  "minRequestInterval": 0,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `items` (type: `string`):

One row per scraped record. See the dataset's default view for field definitions.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "titles": [
        "Machine learning"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("scrapyx/wikipedia-articles-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "titles": ["Machine learning"] }

# Run the Actor and wait for it to finish
run = client.actor("scrapyx/wikipedia-articles-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "titles": [
    "Machine learning"
  ]
}' |
apify call scrapyx/wikipedia-articles-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,scrapyx/wikipedia-articles-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/ETxoGmMzQxdsDmBld/builds/307Hb0bNz0angVZa3/openapi.json