All notable changes to Wikipedia Scraper · Articles, Extracts, Categories & Links. Dates are release dates of this Actor.
Features
- Extracts are now requested in batches of 20 page ids with
exlimit=20, MediaWiki's ceiling. The API defaults exlimit to 1, so previously only the first article of a run carried an extract; now every row does.
- Categories are followed through the API's
continue token instead of being cut off at the first batch, and hidden maintenance categories are filtered out at the source with clshow=!hidden.
- New output fields:
namespace, revisionId, wikidataId, thumbnailUrl, languageCount, pageviews30d, linkCount, links.
- New input fields:
namespace, searchSort, extractScope, extractChars, includeHiddenCategories, includeWikidataId, includePageImage, includeLanguageCount, includePageviews, includeLinks.
- Dataset schema raised to three named views: Overview, Text & Categories, Entity IDs & Provenance.
- The run now names its own default dataset, so the storage is kept past the retention window that deletes unnamed ones.
- Localised category prefixes (
Kategorie:, Categoría:) are stripped, not just the English one.
Features
- Initial release.
- Ships an input schema with field-level
pattern and dateType validation, a dataset schema with named views, and an output schema.
- Every row is pushed to the dataset as it is built rather than buffered to the end, so a run that hits its time limit still returns everything it collected.
- Build tag
v1-0, version 1.0.
