# Smithsonian Open Access Scraper (`crawlerbros/smithsonian-open-access-scraper`) Actor Scrape the Smithsonian Institution's Open Access CC0 public-domain collection - 2.8M+ images and object records across 30+ museums and research units. Search by keyword, browse by museum/unit, or filter by object type. - **URL**: https://apify.com/crawlerbros/smithsonian-open-access-scraper.md - **Developed by:** [Crawler Bros](https://apify.com/crawlerbros) (community) - **Categories:** Developer tools, Other, Automation - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $3.00 / 1,000 results This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Smithsonian Open Access Scraper Search and browse the Smithsonian Institution's **Open Access** collection — over 2.8 million CC0 public-domain images and object records spanning 30+ Smithsonian museums, archives, and research units (National Portrait Gallery, National Air and Space Museum, Smithsonian American Art Museum, National Museum of Natural History departments, Cooper Hewitt, Hirshhorn, and more). Every record returned is public domain (CC0) — free to use, remix, and republish with no permission needed. ### What this actor does - **Three modes:** `search` (keyword), `byUnit` (browse a specific museum/unit), `byObjectType` (browse by kind of object) - **30+ museums and units** covered, from fine art to natural history specimens - **Real CC0 images**: thumbnail and high-resolution download URLs for every item that has one - **Rich metadata**: maker/artist, date, medium, dimensions, credit line, topics, object number - **Empty fields are omitted** — every record only contains what the Smithsonian actually published for that item ### Output per item - `id`, `recordUrl`, `guid` — identifiers and canonical Smithsonian page link - `title` - `unitCode`, `unitName` — owning museum/unit - `makers[]`, `primaryMaker` — artist / maker / sitter names - `date`, `datePeriods[]` — display date and decade/period buckets - `objectType`, `objectTypes[]` — kind of object (e.g. Photograph, Painting, Specimen) - `topics[]` — subject/topic tags - `medium`, `dimensions[]` - `creditLine`, `rights` (almost always `CC0`), `objectNumber`, `collection` - `dataSource`, `notes` (collection/curatorial description), `place` (geographic origin), `publisher` (related publication, when applicable) - `imageUrls[]`, `thumbnailUrl`, `highResImageUrl`, `mediaCount` - `recordType: "object"`, `scrapedAt` ### Input | Field | Type | Default | Description | |---|---|---|---| | `mode` | string | `search` | `search` / `byUnit` / `byObjectType` | | `searchQuery` | string | `portrait` | Keyword matched against title, maker names, and topics | | `unitCode` | string (select) | `npg` | Museum/unit to browse or scope to (mode=byUnit required) | | `objectType` | string | `Photograph` | Object type match, e.g. Painting, Sculpture, Coin, Specimen (mode=byObjectType required) | | `onlyWithImages` | boolean | `true` | Only emit items that have a downloadable image | | `maxItems` | int | `25` | Hard cap (1–500) | #### Example: search across all units ```json { "mode": "search", "searchQuery": "moon landing", "maxItems": 20 } ```` #### Example: browse the National Air and Space Museum ```json { "mode": "byUnit", "unitCode": "nasm", "maxItems": 50 } ``` #### Example: find paintings anywhere in the collection ```json { "mode": "byObjectType", "objectType": "painting", "maxItems": 30 } ``` ### Use cases - **Digital archives & research** — bulk-collect CC0 images and metadata for a topic or museum - **Education** — source public-domain historical images for course materials - **Creative reuse** — royalty-free imagery for design, publishing, or generative-AI training sets - **Museum data analysis** — compare object types, makers, and eras across Smithsonian units - **Content discovery** — find high-resolution scans of specific artifacts, artworks, or specimens ### Data source / limitations This actor reads the Smithsonian's public **Open Access bulk metadata** (announced via the [Smithsonian/OpenAccess GitHub repository](https://github.com/Smithsonian/OpenAccess)), which is mirrored as line-delimited JSON files on a public AWS S3 bucket, organized by owning unit and content hash. There is no interactive search index over the full 2.8M+ records without downloading the entire ~47 GB dataset, so: - `search` and `byObjectType` scan a bounded, round-robined sample of shard files across the relevant unit(s) rather than the full corpus. Broad or common terms (e.g. "portrait", "airplane") reliably return results; very obscure terms may return fewer matches than a true full-text index would find. Scoping to a specific `unitCode` speeds this up and improves relevance. - `byUnit` walks a unit's shard files sequentially and is the most exhaustive, reliable mode. - Archive-focused units (Archives of American Art, National Anthropological Archives, Human Studies Film Archives, Eliot Elisofon Photographic Archives, Freer and Sackler Archives, Anacostia Community Museum Archives) primarily hold archival finding-aid metadata rather than individually digitized media records, so they may return fewer image-bearing items than art or history museums. - `recordUrl` links to the item's page on its owning museum's own website (e.g. `npg.si.edu`, `airandspace.si.edu`). Image URLs (`imageUrls`, `thumbnailUrl`, `highResImageUrl`) are served from the Smithsonian's shared `ids.si.edu` media CDN and are always publicly reachable. A handful of individual museum websites (observed: National Portrait Gallery's `npg.si.edu`) run their own bot-protection in front of the human-facing object page, which can reject fully automated/scripted requests even though the same link opens normally in a real browser — this only affects clicking through to that one museum's page, never the metadata or images themselves. - No login, API key, or paid proxy is required — the bucket and all image CDN links are fully public. ### FAQ **Is this data really public domain?** Yes — the Smithsonian released this collection under [CC0](https://creativecommons.org/publicdomain/zero/1.0/) in 2020. No attribution is legally required, though crediting the Smithsonian is appreciated. **Why do some items have no image?** Not every cataloged object has been digitized yet, and some records (especially in archival units) describe a folder or document rather than a single photographed object. Set `onlyWithImages: true` (the default) to skip these. **What's the difference between `objectType` and `objectTypes`?** `objectType` is the primary/first type Smithsonian assigned to the item; `objectTypes` is the full list when an item has more than one classification. **Can I get every record for a museum?** Run `mode=byUnit` with a high `maxItems`. Very large units may require multiple runs to page through fully, since each run samples a bounded number of shard files. **Are the image URLs safe to hotlink?** Yes — they're served from the Smithsonian's own `ids.si.edu` media delivery service and are publicly accessible without cookies or a referer header. # Actor input Schema ## `mode` (type: `string`): What to fetch. `search` looks for a keyword across sampled records; `byUnit` browses a specific museum/unit; `byObjectType` looks for a kind of object (e.g. painting, specimen). ## `searchQuery` (type: `string`): Keyword matched against title, maker/artist names, and topics (mode=search; also narrows byUnit/byObjectType if set). ## `unitCode` (type: `string`): Smithsonian owning unit (mode=byUnit required; optional scope filter for search/byObjectType). ## `objectType` (type: `string`): Free-text object/type match (mode=byObjectType required; optional filter otherwise). Examples: Photograph, Painting, Sculpture, Coin, Stamp, Specimen, Fossil, Costume, Furniture, Print, Drawing. Matched case-insensitively as a substring against the record's object type. ## `onlyWithImages` (type: `boolean`): Only emit records that include at least one downloadable image. ## `maxItems` (type: `integer`): Hard cap on emitted records. ## Actor input object example ```json { "mode": "search", "searchQuery": "portrait", "unitCode": "npg", "objectType": "Photograph", "onlyWithImages": true, "maxItems": 25 } ``` # Actor output Schema ## `items` (type: `string`): Dataset containing all scraped Smithsonian Open Access items. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "mode": "search", "searchQuery": "portrait", "unitCode": "npg", "objectType": "Photograph", "onlyWithImages": true, "maxItems": 25 }; // Run the Actor and wait for it to finish const run = await client.actor("crawlerbros/smithsonian-open-access-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "mode": "search", "searchQuery": "portrait", "unitCode": "npg", "objectType": "Photograph", "onlyWithImages": True, "maxItems": 25, } # Run the Actor and wait for it to finish run = client.actor("crawlerbros/smithsonian-open-access-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "mode": "search", "searchQuery": "portrait", "unitCode": "npg", "objectType": "Photograph", "onlyWithImages": true, "maxItems": 25 }' | apify call crawlerbros/smithsonian-open-access-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=crawlerbros/smithsonian-open-access-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Smithsonian Open Access Scraper", "description": "Scrape the Smithsonian Institution's Open Access CC0 public-domain collection - 2.8M+ images and object records across 30+ museums and research units. Search by keyword, browse by museum/unit, or filter by object type.", "version": "1.0", "x-build-id": "ixb2hHeNnPOGCFfAd" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/crawlerbros~smithsonian-open-access-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-crawlerbros-smithsonian-open-access-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/crawlerbros~smithsonian-open-access-scraper/runs": { "post": { "operationId": "runs-sync-crawlerbros-smithsonian-open-access-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/crawlerbros~smithsonian-open-access-scraper/run-sync": { "post": { "operationId": "run-sync-crawlerbros-smithsonian-open-access-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "mode" ], "properties": { "mode": { "title": "Mode", "enum": [ "search", "byUnit", "byObjectType" ], "type": "string", "description": "What to fetch. `search` looks for a keyword across sampled records; `byUnit` browses a specific museum/unit; `byObjectType` looks for a kind of object (e.g. painting, specimen).", "default": "search" }, "searchQuery": { "title": "Search query", "type": "string", "description": "Keyword matched against title, maker/artist names, and topics (mode=search; also narrows byUnit/byObjectType if set).", "default": "portrait" }, "unitCode": { "title": "Museum / unit", "enum": [ "", "aaa", "acah", "acm", "cfchfolklife", "chndm", "eepa", "fbr", "fsa", "fsg", "hac", "hmsg", "hsfa", "naa", "nasm", "nmaahc", "nmafa", "nmah", "nmai", "nmnhanthro", "nmnhbirds", "nmnhbotany", "nmnheducation", "nmnhento", "nmnhfishes", "nmnhherps", "nmnhinv", "nmnhmammals", "nmnhminsci", "nmnhpaleo", "npg", "npm", "nzp", "saam", "si", "sia", "sil" ], "type": "string", "description": "Smithsonian owning unit (mode=byUnit required; optional scope filter for search/byObjectType).", "default": "npg" }, "objectType": { "title": "Object type", "type": "string", "description": "Free-text object/type match (mode=byObjectType required; optional filter otherwise). Examples: Photograph, Painting, Sculpture, Coin, Stamp, Specimen, Fossil, Costume, Furniture, Print, Drawing. Matched case-insensitively as a substring against the record's object type." }, "onlyWithImages": { "title": "Only items with images", "type": "boolean", "description": "Only emit records that include at least one downloadable image.", "default": true }, "maxItems": { "title": "Max items", "minimum": 1, "maximum": 500, "type": "integer", "description": "Hard cap on emitted records.", "default": 25 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```