# World Bank Indicators Scraper (`scrapyx/worldbank-indicators-scraper`) Actor

Macroeconomic and development data from the World Bank: 29,544 indicators across 217 countries. Flags the 78 aggregate rows the API mixes in with real countries, which otherwise overstate a global total roughly tenfold.

- **URL**: https://apify.com/scrapyx/worldbank-indicators-scraper.md
- **Developed by:** [Ibnu Adzim](https://apify.com/scrapyx) (community)
- **Categories:** AI, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.84 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## World Bank Indicators Scraper

Macroeconomic and development data from the **World Bank**: 29,544 indicators
across 217 countries and 78 aggregates, back to 1960.

Public data. No login, no API key, no WAF.

### Input

| Field | Type | Default | Meaning |
|---|---|---|---|
| `indicators` | string\[] | — | e.g. `NY.GDP.MKTP.CD`, `SP.POP.TOTL`. One request each. |
| `countries` | string\[] | all | ISO codes, e.g. `USA`, `IDN`. |
| `includeAggregates` | boolean | **`false`** | See below — the setting that matters most. |
| `yearFrom` / `yearTo` | integer | — | Year range. |
| `mostRecentValues` | integer | — | Latest N per country. Mutually exclusive with a year range. |
| `skipMissingValues` | boolean | `false` | Gaps stay visible by default. |

```json
{ "indicators": ["NY.GDP.MKTP.CD"], "countries": ["IDN","USA"], "mostRecentValues": 5 }
```

### Four things this actor will not let you get wrong

#### 1. 78 of the API's 295 "countries" are aggregates — summing is 10× wrong

`/country` returns "Arab World", "Africa", "High income", "Euro area" and
"World" alongside Argentina and Albania. Measured on 2023 population over
`country/all`:

| | total | vs the truth |
|---|---|---|
| naive sum of every row | 86,466,868,911 | **10.72×** |
| filtering aggregates by **ISO3 code** | 16,102,473,551 | **2.00×** |
| filtering on **`isAggregate`** | 8,039,550,134 | **0.997×** |
| the `World` row itself | 8,062,923,417 | 1.00× |

The obvious fix — drop rows whose ISO3 code is a known aggregate — **still
double-counts, exactly 2×**. The five income-group aggregates ("High income",
"Low income", "Lower middle income", "Upper middle income", "Not classified")
come back with a **blank `countryiso3code`**, so they slip straight through.

The reliable key is the two-letter `country.id`, matched against `iso2Code` in
`/country` where `region.id == "NA"` marks an aggregate. Every row carries
`isAggregate`; aggregates are excluded by default and counted in the summary
when included.

#### 2. Errors arrive as HTTP 200, in an array that is one element short

The response is a two-element array — and it has **three** shapes:

```
normal          [meta, [rows…]]
valid but empty [meta, null]          <- null, not []
INVALID         [{"message": […]}]    <- length 1, and HTTP 200
```

A bogus indicator or country code is a **200**. `payload[1]` raises
`IndexError`, and guarding with `len(payload) > 1` is worse — it turns a
rejected query into a silent empty success. One place in this actor
(`parse_envelope`) knows all three, and a rejection becomes an
`invalid_parameter` error row that says the rejection came inside a 200.

#### 3. `mostRecentValues` is silently dropped by the CDN, intermittently

Asking for `mrv=3` on a 66-row series returned 3 rows at almost every page
size — but **10 rows at `per_page=10`**, with `total` reporting 66. The failing
responses carried an `Age` header of several hundred seconds: a **cached
response computed without `mrv`**. A cache-busting parameter made it apply
again, and the same request against other countries worked.

So it is not a rule — it is intermittent, host-dependent and silent, which is
worse. `mrv` is therefore enforced client-side as well as requested, with
`mrvHonouredUpstream` and `rowsTrimmedClientSide` reporting what happened.

#### 4. The `;` multi-indicator syntax silently returns only the first series

`/country/USA/indicator/NY.GDP.MKTP.CD;SP.POP.TOTL&source=2` answers **HTTP
200 with plenty of rows — all of them GDP**. Half the request is discarded with
nothing to say so. This actor issues one request per indicator.

#### Also: missing observations are common, and are not zeros

Tuvalu's GDP series has 10 gaps in 21 years. `hasValue` is explicit and
`value` stays `null`, so a gap is never averaged as a zero.

### Output

Envelope on every row: `_input`, `_source`, `_scrapedAt`, `recordType`.

- **`OBSERVATION`** — indicator, country (with region, income level, capital,
  coordinates), year, value, `hasValue`, `isAggregate`, `seriesLastUpdated`
- **`SEARCH_SUMMARY`** — `aggregateRowsIncluded` vs `countryRowsIncluded`,
  `rowsWithoutValue`, `mrvHonouredUpstream`, `cachedResponses`
- **`ERROR`** — `_error` + `_errorDetail`

If the country/aggregate lookup cannot be loaded, the run **stops** rather than
emitting data that cannot safely be summed.

# Actor input Schema

## `indicators` (type: `array`):

World Bank indicator codes, e.g. NY.GDP.MKTP.CD (GDP, current US$), SP.POP.TOTL (population), FP.CPI.TOTL.ZG (inflation). 29,544 exist. Each runs as its own request — the API's ';' multi-indicator syntax silently returns only the first series.

## `countries` (type: `array`):

ISO codes, e.g. USA, IDN, BRA. Leave empty for every country AND every aggregate — see 'Include aggregates' below, which is the setting that matters most here.

## `includeAggregates` (type: `boolean`):

OFF by default. 78 of the API's 295 'countries' are actually aggregates — 'Arab World', 'Africa', 'High income', 'Euro area', 'World'. Summing a full series with them left in overstates a global total by roughly 10x. Every row carries isAggregate either way.

## `yearFrom` (type: `integer`):

e.g. 2000. Mutually exclusive with 'Most recent values'.

## `yearTo` (type: `integer`):

e.g. 2024.

## `mostRecentValues` (type: `integer`):

Return only the N latest observations per country. Enforced here as well as requested upstream — the World Bank CDN intermittently serves a cached response computed without it, returning more rows than asked for with nothing to say so.

## `skipMissingValues` (type: `boolean`):

Missing observations are common in this data and are NOT zeros. Off by default so gaps stay visible; every row carries hasValue.

## `maxResultsPerIndicator` (type: `integer`):

0 = unlimited.

## `pageSize` (type: `integer`):

The API accepted 20,000 in testing, but 32,768 took over 90 seconds to answer — capped at 10,000 here because a page that large is a hang risk.

## `maxConcurrency` (type: `integer`):

Requests in flight at once across all indicators.

## `minRequestInterval` (type: `number`):

Politeness pacing shared across all workers.

## `proxyConfiguration` (type: `object`):

Optional and OFF by default. No WAF was observed on api.worldbank.org.

## Actor input object example

```json
{
  "indicators": [
    "SP.POP.TOTL",
    "FP.CPI.TOTL.ZG"
  ],
  "countries": [
    "USA",
    "IDN",
    "BRA"
  ],
  "includeAggregates": false,
  "skipMissingValues": false,
  "maxResultsPerIndicator": 1000,
  "pageSize": 1000,
  "maxConcurrency": 3,
  "minRequestInterval": 0.2,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `items` (type: `string`):

One row per scraped record. See the dataset's default view for field definitions.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "indicators": [
        "NY.GDP.MKTP.CD"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("scrapyx/worldbank-indicators-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "indicators": ["NY.GDP.MKTP.CD"] }

# Run the Actor and wait for it to finish
run = client.actor("scrapyx/worldbank-indicators-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "indicators": [
    "NY.GDP.MKTP.CD"
  ]
}' |
apify call scrapyx/worldbank-indicators-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,scrapyx/worldbank-indicators-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/PFofCLublDjsQGfY2/builds/dbmqWCIIU5uk1cgXz/openapi.json