World Bank Indicators Scraper

Macroeconomic and development data from the World Bank: 29,544 indicators across 217 countries and 78 aggregates, back to 1960.

Public data. No login, no API key, no WAF.

Input

Field	Type	Default	Meaning
indicators	string[]	—	e.g. NY.GDP.MKTP.CD, SP.POP.TOTL. One request each.
countries	string[]	all	ISO codes, e.g. USA, IDN.
includeAggregates	boolean	false	See below — the setting that matters most.
yearFrom / yearTo	integer	—	Year range.
mostRecentValues	integer	—	Latest N per country. Mutually exclusive with a year range.
skipMissingValues	boolean	false	Gaps stay visible by default.

{ "indicators": ["NY.GDP.MKTP.CD"], "countries": ["IDN","USA"], "mostRecentValues": 5 }

Four things this actor will not let you get wrong

1. 78 of the API's 295 "countries" are aggregates — summing is 10× wrong

/country returns "Arab World", "Africa", "High income", "Euro area" and "World" alongside Argentina and Albania. Measured on 2023 population over country/all:

	total	vs the truth
naive sum of every row	86,466,868,911	10.72×
filtering aggregates by ISO3 code	16,102,473,551	2.00×
filtering on isAggregate	8,039,550,134	0.997×
the World row itself	8,062,923,417	1.00×

The obvious fix — drop rows whose ISO3 code is a known aggregate — still double-counts, exactly 2×. The five income-group aggregates ("High income", "Low income", "Lower middle income", "Upper middle income", "Not classified") come back with a blank countryiso3code, so they slip straight through.

The reliable key is the two-letter country.id, matched against iso2Code in /country where region.id == "NA" marks an aggregate. Every row carries isAggregate; aggregates are excluded by default and counted in the summary when included.

2. Errors arrive as HTTP 200, in an array that is one element short

The response is a two-element array — and it has three shapes:

normal          [meta, [rows…]]
valid but empty [meta, null]          <- null, not []
INVALID         [{"message": […]}]    <- length 1, and HTTP 200

A bogus indicator or country code is a 200. payload[1] raises IndexError, and guarding with len(payload) > 1 is worse — it turns a rejected query into a silent empty success. One place in this actor (parse_envelope) knows all three, and a rejection becomes an invalid_parameter error row that says the rejection came inside a 200.

3. mostRecentValues is silently dropped by the CDN, intermittently

Asking for mrv=3 on a 66-row series returned 3 rows at almost every page size — but 10 rows at per_page=10, with total reporting 66. The failing responses carried an Age header of several hundred seconds: a cached response computed without mrv. A cache-busting parameter made it apply again, and the same request against other countries worked.

So it is not a rule — it is intermittent, host-dependent and silent, which is worse. mrv is therefore enforced client-side as well as requested, with mrvHonouredUpstream and rowsTrimmedClientSide reporting what happened.

4. The ; multi-indicator syntax silently returns only the first series

/country/USA/indicator/NY.GDP.MKTP.CD;SP.POP.TOTL&source=2 answers HTTP 200 with plenty of rows — all of them GDP. Half the request is discarded with nothing to say so. This actor issues one request per indicator.

Also: missing observations are common, and are not zeros

Tuvalu's GDP series has 10 gaps in 21 years. hasValue is explicit and value stays null, so a gap is never averaged as a zero.

Output

Envelope on every row: _input, _source, _scrapedAt, recordType.

• OBSERVATION — indicator, country (with region, income level, capital, coordinates), year, value, hasValue, isAggregate, seriesLastUpdated
• SEARCH_SUMMARY — aggregateRowsIncluded vs countryRowsIncluded, rowsWithoutValue, mrvHonouredUpstream, cachedResponses
• ERROR — _error + _errorDetail

If the country/aggregate lookup cannot be loaded, the run stops rather than emitting data that cannot safely be summed.