# McMaster-Carr Product Scraper (`automation-lab/mcmaster-carr-product-scraper`) Actor Extract public McMaster-Carr product families, catalog categories, specification filters, product counts, and source URLs for procurement intelligence. - **URL**: https://apify.com/automation-lab/mcmaster-carr-product-scraper.md - **Developed by:** [Stas Persiianenko](https://apify.com/automation-lab) (community) - **Categories:** E-commerce - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing Pay per event This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## McMaster-Carr Product Scraper Extract public McMaster-Carr catalog intelligence from category, search, and product-family pages. This actor captures product-family metadata, visible specification filters, category links, source URLs, and scrape timestamps from the dynamic McMaster-Carr catalog. ### What does McMaster-Carr Product Scraper do? McMaster-Carr Product Scraper opens McMaster-Carr catalog pages in a browser and listens for the first-party catalog data that powers the page. It can collect: - Product-family names - Product counts - Presentation counts - Catalog category links - Visible specification filters - Filter values and product counts - Source URLs - Login-wall detection - Scrape timestamps ### Who is it for? This scraper is designed for industrial and MRO workflows. - Procurement teams comparing catalog coverage - Ecommerce analysts monitoring industrial supply categories - Engineering sourcing teams mapping part families - Catalog data teams building product taxonomies - Price and assortment intelligence teams - Operations teams researching replacement parts ### Why use this actor? McMaster-Carr is a large dynamic catalog. Manual catalog review is slow when you need to compare many product families or understand category structure. This actor turns public catalog pages into structured dataset rows that are easy to export, filter, and enrich. ### What data can you extract? | Field | Description | | --- | --- | | `entityType` | `product_family`, `spec_filter`, or `catalog_category` | | `title` | Human-readable record title | | `productFamily` | McMaster product-family name when available | | `productCount` | Public product count when exposed | | `presentationCount` | Presentation count when exposed | | `attributeName` | Specification/filter name | | `attributeValues` | Visible values with counts and links | | `requestedUrl` | URL or keyword-derived URL requested by the actor | | `sourceUrl` | Final source URL for the row | | `loginWallDetected` | Whether McMaster displayed a login wall during browsing | | `scrapeTimestamp` | ISO timestamp for the scrape | ### How much does it cost to scrape McMaster-Carr products? The actor uses pay-per-event pricing. - A start event is charged once per run: **$0.005**. - Each saved dataset item is charged with tiered pricing. - Tier discounts apply automatically based on your Apify plan. | Apify plan tier | Price per saved item | | --- | ---: | | FREE | $0.00020828 | | BRONZE | $0.00018111 | | SILVER | $0.00014127 | | GOLD | $0.00010867 | | PLATINUM | $0.000072445 | | DIAMOND | $0.000050711 | **Cost examples:** Calculated as the $0.005 start event + saved items × the applicable item rate. | Run size | FREE plan estimate | BRONZE plan estimate | What it covers | | --- | ---: | ---: | --- | | 25 items | about $0.0103 | about $0.0095 | Quick category test | | 50 items | about $0.0154 | about $0.0141 | Typical validation run | | 500 items | about $0.1091 | about $0.0956 | Larger catalog-mapping batch | Free-plan example: a 50-item run costs about **$0.0154** in actor charges (`$0.005` start + `50 × $0.00020828`). Use a low `maxItems` value for first tests, then increase once the output matches your workflow. ### Quick start 1. Open the actor on Apify. 2. Add one or more McMaster-Carr category or search URLs. 3. Optionally add keywords such as `socket head screws` or `bearings`. 4. Set `maxItems`. 5. Run the actor. 6. Export results from the dataset as JSON, CSV, Excel, or via API. ### Input options #### Category or search URLs Use McMaster URLs such as: ```text https://www.mcmaster.com/products/screws/ https://www.mcmaster.com/products/bearings/ ```` #### Keywords Keywords are converted into McMaster URL slugs. Example: ```json { "keywords": ["socket head screws", "bearings"] } ``` #### Maximum records `maxItems` controls the maximum number of rows saved. Start small for testing: ```json { "maxItems": 25 } ``` #### Include spec filters Set `includeDetails` to `true` to collect visible public specification filters and values. ```json { "includeDetails": true } ``` ### Example input ```json { "startUrls": [ { "url": "https://www.mcmaster.com/products/screws/" } ], "keywords": ["bearings"], "maxItems": 50, "includeDetails": true, "proxyConfiguration": { "useApifyProxy": false } } ``` ### Example output ```json { "entityType": "product_family", "requestedUrl": "https://www.mcmaster.com/products/screws/", "sourceUrl": "https://www.mcmaster.com/products/screws/", "title": "Socket Head Screws", "productFamily": "Socket Head Screws", "productCount": 7979, "presentationCount": 39, "pageType": "product_family_landing_page", "loginWallDetected": true, "scrapeTimestamp": "2026-07-09T04:41:49.387Z" } ``` ### Working with login-wall detection McMaster-Carr may show a message such as `To continue browsing, please log in` after initial public catalog hydration. The actor does not collect private account data and does not require McMaster credentials. When this happens, `loginWallDetected` is set to `true` so downstream workflows can separate public metadata from pages that were not fully browsable. ### Tips for reliable runs - Use category URLs for the most stable results. - Keep `maxItems` low while testing. - Use `includeDetails: false` if you only need category discovery. - Avoid very large keyword batches in one run. - Use datacenter proxy first if a proxy is needed. - Use residential proxy only if datacenter is blocked. ### Integrations You can connect the dataset to concrete industrial workflows: - **Procurement dashboards**: refresh screw, bearing, fastener, or raw-material category coverage each week and chart product-family counts. - **Google Sheets**: export CSV rows for sourcing analysts who need a quick sortable catalog map without writing code. - **Airtable**: maintain a lightweight product-family tracker with owner, sourcing status, and notes per category. - **Snowflake or BigQuery**: append scheduled runs into a warehouse table and compare category/filter changes over time. - **Internal PIM systems**: seed taxonomy enrichment queues with public McMaster product-family names and spec attributes. - **Price-monitoring jobs**: use category and filter rows to identify which product families should be checked manually or by an account-authorized internal process. - **Catalog taxonomy enrichment**: merge McMaster public attribute names with your internal MRO vocabulary for search and faceted navigation. ### API usage with Node.js ```js import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('automation-lab/mcmaster-carr-product-scraper').call({ startUrls: [{ url: 'https://www.mcmaster.com/products/screws/' }], maxItems: 25, includeDetails: true, }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); console.log(items); ``` ### API usage with Python ```python from apify_client import ApifyClient import os client = ApifyClient(os.environ['APIFY_TOKEN']) run = client.actor('automation-lab/mcmaster-carr-product-scraper').call(run_input={ 'startUrls': [{'url': 'https://www.mcmaster.com/products/screws/'}], 'maxItems': 25, 'includeDetails': True, }) items = client.dataset(run['defaultDatasetId']).list_items().items print(items) ``` ### API usage with cURL ```bash curl -X POST "https://api.apify.com/v2/acts/automation-lab~mcmaster-carr-product-scraper/runs?token=$APIFY_TOKEN" \ -H 'Content-Type: application/json' \ -d '{"startUrls":[{"url":"https://www.mcmaster.com/products/screws/"}],"maxItems":25,"includeDetails":true}' ``` ### Use with MCP Connect this actor to AI tools through Apify MCP. MCP URL: ```text https://mcp.apify.com/?tools=automation-lab/mcmaster-carr-product-scraper ``` Claude Code setup command: ```bash claude mcp add apify-mcmaster-carr \ --transport http \ "https://mcp.apify.com/?tools=automation-lab/mcmaster-carr-product-scraper" ``` Claude Desktop, Cursor, or VS Code MCP JSON configuration: ```json { "mcpServers": { "apify-mcmaster-carr": { "url": "https://mcp.apify.com/?tools=automation-lab/mcmaster-carr-product-scraper", "headers": { "Authorization": "Bearer ${APIFY_TOKEN}" } } } } ``` If your MCP client does not expand `${APIFY_TOKEN}`, paste your Apify API token into the Authorization header locally. Keep the token private and do not commit it to a repository. Example prompts: - "Scrape public McMaster-Carr screw category metadata and summarize the main product families." - "Extract spec filters for bearings from McMaster-Carr and group them by attribute." - "Compare category counts across several McMaster-Carr product URLs." ### Data quality notes The actor extracts public catalog metadata exposed during page hydration. Some part-level pricing, inventory, CAD downloads, or account-specific data may require a logged-in McMaster session and is not collected. ### Legality This actor extracts publicly accessible catalog metadata. Make sure your use complies with McMaster-Carr terms, applicable laws, and your internal data governance rules. ### FAQ #### Why do I see `loginWallDetected: true`? McMaster-Carr sometimes displays a login wall after the public page data loads. The actor records this flag instead of pretending that the page was fully available. #### Why did I get category rows instead of product-family rows? Some McMaster pages expose only browse-category links before showing a login wall. Try a more specific product URL or keyword. #### Why are part numbers missing? Part-level tables are not always exposed to anonymous public sessions. The actor saves product-family and filter metadata when part numbers are not available. ### Related scrapers Related automation-lab actors that can support catalog intelligence workflows: - https://apify.com/automation-lab/uline-scraper - https://apify.com/automation-lab/toolstation-scraper - https://apify.com/automation-lab/costco-business-center-scraper ### Changelog #### 0.1 Initial private build for McMaster-Carr public catalog metadata extraction. ### Support If a McMaster-Carr URL produces fewer records than expected, include the run ID, input JSON, and the target URL in your support request. ### Limitations - No private McMaster account login. - No bypassing credential walls. - Part-level price and stock may not be available anonymously. - Dynamic catalog data can vary by page and session. ### Best practices - Use specific product-family URLs. - Combine URLs and keywords for broader coverage. - Store run inputs with your reports for reproducibility. - Monitor `loginWallDetected` in downstream pipelines. - Keep first test runs small. ### Output formats Apify datasets can be exported as: - JSON - JSONL - CSV - Excel - XML - RSS ### Procurement workflow example 1. Scrape screw and bearing category URLs. 2. Export product-family rows to CSV. 3. Group by `productFamily` and `productCount`. 4. Review spec filters for key attributes. 5. Feed selected families into internal sourcing research. ### Catalog intelligence workflow example 1. Run keywords for a list of industrial part families. 2. Store the dataset in BigQuery. 3. Track product counts over time. 4. Alert if a product family appears, disappears, or changes count significantly. ### Final notes McMaster-Carr Product Scraper is built for stable public catalog discovery. It is intentionally conservative around account-only data and records login-wall signals clearly for transparent downstream analysis. # Actor input Schema ## `startUrls` (type: `array`): McMaster-Carr product/category/search URLs to scrape, for example https://www.mcmaster.com/products/screws/. ## `keywords` (type: `array`): Optional McMaster-Carr search terms. Each keyword is converted to a public McMaster URL such as /socket-head-screws/. ## `maxItems` (type: `integer`): Maximum dataset records to save across product families, spec filters, and catalog category links. ## `includeDetails` (type: `boolean`): Capture visible McMaster specification filters and value counts when available from the public page data. ## `proxyConfiguration` (type: `object`): Optional Apify proxy configuration. Start without proxy or with datacenter; use residential only if your target pages are blocked. ## Actor input object example ```json { "startUrls": [ { "url": "https://www.mcmaster.com/products/screws/" } ], "keywords": [ "socket head screws", "bearings" ], "maxItems": 20, "includeDetails": true, "proxyConfiguration": { "useApifyProxy": false } } ``` # Actor output Schema ## `overview` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ { "url": "https://www.mcmaster.com/products/screws/" } ], "keywords": [ "socket head screws", "bearings" ], "maxItems": 20, "includeDetails": true, "proxyConfiguration": { "useApifyProxy": false } }; // Run the Actor and wait for it to finish const run = await client.actor("automation-lab/mcmaster-carr-product-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": [{ "url": "https://www.mcmaster.com/products/screws/" }], "keywords": [ "socket head screws", "bearings", ], "maxItems": 20, "includeDetails": True, "proxyConfiguration": { "useApifyProxy": False }, } # Run the Actor and wait for it to finish run = client.actor("automation-lab/mcmaster-carr-product-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ { "url": "https://www.mcmaster.com/products/screws/" } ], "keywords": [ "socket head screws", "bearings" ], "maxItems": 20, "includeDetails": true, "proxyConfiguration": { "useApifyProxy": false } }' | apify call automation-lab/mcmaster-carr-product-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=automation-lab/mcmaster-carr-product-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "McMaster-Carr Product Scraper", "description": "Extract public McMaster-Carr product families, catalog categories, specification filters, product counts, and source URLs for procurement intelligence.", "version": "0.1", "x-build-id": "h92dPPGqnAJyd4zQj" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/automation-lab~mcmaster-carr-product-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-automation-lab-mcmaster-carr-product-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/automation-lab~mcmaster-carr-product-scraper/runs": { "post": { "operationId": "runs-sync-automation-lab-mcmaster-carr-product-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/automation-lab~mcmaster-carr-product-scraper/run-sync": { "post": { "operationId": "run-sync-automation-lab-mcmaster-carr-product-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "startUrls": { "title": "Category or search URLs", "type": "array", "description": "McMaster-Carr product/category/search URLs to scrape, for example https://www.mcmaster.com/products/screws/.", "items": { "type": "object", "required": [ "url" ], "properties": { "url": { "type": "string", "title": "URL of a web page", "format": "uri" } } } }, "keywords": { "title": "Search keywords", "type": "array", "description": "Optional McMaster-Carr search terms. Each keyword is converted to a public McMaster URL such as /socket-head-screws/.", "items": { "type": "string" } }, "maxItems": { "title": "Maximum records", "minimum": 1, "maximum": 10000, "type": "integer", "description": "Maximum dataset records to save across product families, spec filters, and catalog category links.", "default": 20 }, "includeDetails": { "title": "Include spec filters", "type": "boolean", "description": "Capture visible McMaster specification filters and value counts when available from the public page data.", "default": true }, "proxyConfiguration": { "title": "Proxy configuration", "type": "object", "description": "Optional Apify proxy configuration. Start without proxy or with datacenter; use residential only if your target pages are blocked." } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```