# Siemens Catalog Crawler — MPN Discovery (`crawloop/siemens-sieportal-product-catalog-crawler`) Actor
Search Siemens SiePortal live by keywords (6ES, SIMATIC, S7-1200) via OneSearch. Export unique MPNs, titles, category paths & PDP links. Deduplicated output with resume for large catalog builds.
- **URL**: https://apify.com/crawloop/siemens-sieportal-product-catalog-crawler.md
- **Developed by:** [Andrej Kiva](https://apify.com/crawloop) (community)
- **Categories:** Developer tools, Integrations, Automation
- **Stats:** 2 total users, 1 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet
## Pricing
from $0.20 / 1,000 catalog product exporteds
This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event
## What's an Apify Actor?
Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".
## How to integrate an Actor?
If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.
In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md):
```bash
npm install apify-client
```
In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md):
```bash
pip install apify-client
```
In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md):
````bash
# MacOS / Linux
curl -fsSL https://apify.com/install-cli.sh | bash
# Windows
irm https://apify.com/install-cli.ps1 | iex
```bash
In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md).
If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md).
For usage examples, see the [API](#api) section below.
For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).
# README
## Siemens Catalog Crawler — MPN Discovery
> **Crawloop Siemens Automation Suite** — Structured data extraction for Siemens SiePortal (Industry Mall), SIOS, and TED product datasheets. Built for procurement teams, system integrators, and BOM engineering workflows.
>
> Suite hub: [github.com/PLCSPS-DEV/siemens-sieportal-automation](https://github.com/PLCSPS-DEV/siemens-sieportal-automation)
>
> Product site: [crawloop.com/siemens-automation](https://crawloop.com/siemens-automation.html)
| Discovery | Enrichment | SIOS documents | TED datasheets |
| :--- | :--- | :--- | :--- |
| [Catalog Crawler](https://apify.com/crawloop/siemens-sieportal-product-catalog-crawler?fpr=guboir) | [SiePortal Scraper](https://apify.com/crawloop/siemens-sieportal-scraper?fpr=guboir) | [Document Downloader](https://apify.com/crawloop/siemens-sieportal-document-downloader?fpr=guboir) | [TED Datasheet Downloader](https://apify.com/crawloop/siemens-datasheet-downloader?fpr=guboir) |
| | [Lifecycle Tracker](https://apify.com/crawloop/siemens-sieportal-product-lifecycle-tracker?fpr=guboir) | [Document PDF Parser](https://apify.com/crawloop/siemens-sieportal-product-document-pdf-crawler?fpr=guboir) | [TED Datasheet Parser](https://apify.com/crawloop/siemens-datasheet-pdf-crawler?fpr=guboir) |
> **Disclaimer:** This is an unofficial integration developed independently of Siemens AG. It is not affiliated with, sponsored by, or endorsed by Siemens AG or any of its subsidiaries.
>
> **Siemens**, **SiePortal**, **SIMATIC**, and related names are trademarks of Siemens AG. Product data is read from publicly accessible Siemens web sources only; no proprietary databases are redistributed.
>
> This Actor is provided **for informational and research purposes only** (e.g. procurement research, BOM audits, internal engineering workflows). You are solely responsible for ensuring your use complies with applicable laws, Siemens website terms of use, and your organization's policies.
>
> No warranty is given as to accuracy, completeness, or continued availability of third-party data. Use at your own risk.
Search the official Siemens SiePortal catalog by keyword and export live product part numbers (MPNs). Each result is fetched from Siemens OneSearch inside a stealth browser session — no static MPN lists, no stale exports.
Use this Actor to build MPN inventories for product families (`6ES`, `SIMATIC`, `S7-1200`), procurement databases, or as input for downstream enrichment with the [SiePortal Scraper](https://apify.com/crawloop/siemens-sieportal-scraper?fpr=guboir) and [Lifecycle Tracker](https://apify.com/crawloop/siemens-sieportal-product-lifecycle-tracker?fpr=guboir).
### When to use this Actor
Use the **Catalog Crawler** when you need to discover Siemens part numbers by search terms or crawl the full catalog index (~350k+ MPNs) via the authenticated OneSearch API.
For full PDP specifications on a known MPN list, use the [SiePortal Scraper](https://apify.com/crawloop/siemens-sieportal-scraper?fpr=guboir).
### Siemens Automation Pipeline
````
Phase 1 — Discover MPNs Phase 2 — Screen & enrich Phase 3 — Documents & specs
───────────────────────── ───────────────────────── ─────────────────────────────
Catalog Crawler ◄── you are here
│
▼
MPN list ──► Lifecycle Tracker ──► SiePortal Scraper
│
│
┌────────────────────────────┴────────────────────────────┐
│ │
▼ ▼
Document Downloader (SIOS) TED Datasheet Downloader
certificates, manuals, CAD compact catalog PDFs
│ │
▼ ▼
Document PDF Parser TED Datasheet Parser
specs from SIOS PDFs specs from TED PDFs
````
### Key Features
- **Keyword search** — Provide product families, prefixes, or terms; the Actor queries Siemens OneSearch live.
- **Multi-sort crawl** — Each keyword is queried with multiple `sortingOptions` to maximize coverage; deduplication removes overlap.
- **Unique output** — Each MPN emitted once (`deduplicateOutput`, default `true`).
- **Full catalog mode** — Optional crawl of the entire Siemens index with adaptive shard subdivision and resume checkpoints.
- **Akamai bypass** — CloakBrowser stealth fingerprinting and session bootstrap for authenticated OneSearch API access.
- **Resume support** — Skip completed search terms on interrupted runs.
### How It Works
1. Bootstrap SiePortal search page (Akamai WAF).
2. Capture Bearer JWT for `POST /api/onesearch/search?api-version=2.0`.
3. For each `searchKeywords` term, query OneSearch and paginate results.
4. Auto-subdivide large result sets into child shards.
5. Push unique MPN records to the dataset with category breadcrumbs.
### Input Parameters
| Parameter | Description | Default |
| :--- | :--- | :--- |
| `searchKeywords` | **Required** for `keywords` mode — product families, prefixes, or terms. | `[]` |
| `sortingOptions` | OneSearch sort orders per keyword (multiple = more coverage). | Relevance + 3 more |
| `useMultipleSorts` | Crawl every keyword with all sorting options. | `true` |
| `discoveryMode` | `keywords`, `full_catalog`, or `catalog_tree`. | `keywords` |
| `locale` | SiePortal locale (`en-ww`, `en-nl`, …). | `en-ww` |
| `maxSearchShards` | Cap terms processed (0 = unlimited; use for testing). | `0` |
| `concurrencyLimit` | Parallel OneSearch workers. | `3` |
| `deduplicateOutput` | Emit each MPN only once. | `true` |
| `resumeFromCheckpoint` | Resume interrupted runs. | `true` |
| `proxyConfiguration` | Apify proxy settings. Residential proxies recommended. | — |
#### Input Example — keyword search (recommended)
```json
{
"searchKeywords": ["6ES", "SIMATIC", "S7-1200", "3RT", "contactor"],
"discoveryMode": "keywords",
"concurrencyLimit": 3,
"deduplicateOutput": true,
"proxyConfiguration": {
"useApifyProxy": true,
"apifyProxyGroups": ["RESIDENTIAL"]
}
}
````
#### Input Example — smoke test
```json
{
"searchKeywords": ["6ES"],
"maxSearchShards": 3,
"concurrencyLimit": 2,
"resumeFromCheckpoint": false
}
```
#### Input Example — full catalog (~350k MPNs)
```json
{
"discoveryMode": "full_catalog",
"concurrencyLimit": 3,
"resumeFromCheckpoint": true
}
```
### Output Format
Each dataset record:
| Field | Description |
| :--- | :--- |
| `partNumber` | Siemens article number (MPN) |
| `title` | Short product title |
| `description` | Product description |
| `pdpUrl` | Product detail page URL |
| `categoryId` | Parent category node ID |
| `categoryPath` | Breadcrumb trail |
| `source` | Always `onesearch` (live Siemens API) |
| `searchShard` | Keyword/shard that found this product |
| `sortingOption` | Sort order that surfaced this product |
### Typical Workflow
```
Keywords (6ES, SIMATIC, …)
│
▼
Catalog Crawler → deduplicated MPN list with PDP URLs
│
▼
Lifecycle Tracker → screen for obsolete parts
│
▼
SiePortal Scraper → full specs on selected MPNs
```
### Actor Comparison
| Task | Catalog Crawler | SiePortal Scraper |
| :--- | :---: | :---: |
| Discover MPNs by keyword | Yes | No |
| Full PDP specifications | No | Yes |
| Full catalog index (~350k) | Yes | No |
### Pricing
Pay-per-event billing. You are charged only for unique MPNs pushed to the dataset when deduplication is enabled.
| Event | Price |
| :--- | :--- |
| Actor start | $0.05 per run |
| Catalog product | $0.20 / 1,000 MPNs ($0.0002 per unique product) |
Configure these events in the Apify Store Publication tab: `apify-actor-start` and `catalog-product`. Disable `apify-default-dataset-item` to avoid double billing.
Residential proxies are strongly recommended. `full_catalog` mode can take many hours; all output is scraped live from Siemens.
***
**Learn more:** [Product page](https://crawloop.com/products/siemens-catalog-crawler.html) · [Suite hub](https://crawloop.com/siemens-automation.html) · [GitHub docs](https://plcsps-dev.github.io/siemens-sieportal-automation/)
**Also from Crawloop Industrial:** [Rockwell Automation Suite](https://crawloop.com/rockwell-automation.html) · [GitHub docs](https://plcsps-dev.github.io/rockwell-automation/)
# Actor input Schema
## `searchKeywords` (type: `array`):
What to look for on Siemens website. One term per line.
Examples: 6ES, SIMATIC, S7-1200, contactor
## `discoveryMode` (type: `string`):
Search my keywords — recommended. Uses the list above.
Full Siemens catalog — all products, very slow.
Category pages — experimental.
## `locale` (type: `string`):
Siemens website region. Default: Worldwide (English).
## `proxyConfiguration` (type: `object`):
Keep Apify residential proxy enabled (recommended).
## `maxSearchShards` (type: `integer`):
For a quick test use 3. For a full run use 0 (no limit).
## `concurrencyLimit` (type: `integer`):
How many searches run in parallel. Default 3. Use 1 if errors appear.
## `deduplicateOutput` (type: `boolean`):
Each article number appears only once in results.
## `resumeFromCheckpoint` (type: `boolean`):
Skip keywords already finished in a previous run.
## Actor input object example
```json
{
"searchKeywords": [
"6ES",
"SIMATIC",
"S7-1200"
],
"discoveryMode": "keywords",
"locale": "en-ww",
"proxyConfiguration": {
"useApifyProxy": true,
"apifyProxyGroups": [
"RESIDENTIAL"
]
},
"maxSearchShards": 3,
"concurrencyLimit": 3,
"deduplicateOutput": true,
"resumeFromCheckpoint": false
}
```
# API
You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.
## JavaScript example
```javascript
import { ApifyClient } from 'apify-client';
// Initialize the ApifyClient with your Apify API token
// Replace the '' with your token
const client = new ApifyClient({
token: '',
});
// Prepare Actor input
const input = {
"searchKeywords": [
"6ES",
"SIMATIC",
"S7-1200"
],
"discoveryMode": "keywords",
"locale": "en-ww",
"proxyConfiguration": {
"useApifyProxy": true,
"apifyProxyGroups": [
"RESIDENTIAL"
]
},
"maxSearchShards": 3,
"concurrencyLimit": 3,
"deduplicateOutput": true,
"resumeFromCheckpoint": false
};
// Run the Actor and wait for it to finish
const run = await client.actor("crawloop/siemens-sieportal-product-catalog-crawler").call(input);
// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
console.dir(item);
});
// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs
```
## Python example
```python
from apify_client import ApifyClient
# Initialize the ApifyClient with your Apify API token
# Replace '' with your token.
client = ApifyClient("")
# Prepare the Actor input
run_input = {
"searchKeywords": [
"6ES",
"SIMATIC",
"S7-1200",
],
"discoveryMode": "keywords",
"locale": "en-ww",
"proxyConfiguration": {
"useApifyProxy": True,
"apifyProxyGroups": ["RESIDENTIAL"],
},
"maxSearchShards": 3,
"concurrencyLimit": 3,
"deduplicateOutput": True,
"resumeFromCheckpoint": False,
}
# Run the Actor and wait for it to finish
run = client.actor("crawloop/siemens-sieportal-product-catalog-crawler").call(run_input=run_input)
# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
print(item)
# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start
```
## CLI example
```bash
echo '{
"searchKeywords": [
"6ES",
"SIMATIC",
"S7-1200"
],
"discoveryMode": "keywords",
"locale": "en-ww",
"proxyConfiguration": {
"useApifyProxy": true,
"apifyProxyGroups": [
"RESIDENTIAL"
]
},
"maxSearchShards": 3,
"concurrencyLimit": 3,
"deduplicateOutput": true,
"resumeFromCheckpoint": false
}' |
apify call crawloop/siemens-sieportal-product-catalog-crawler --silent --output-dataset
```
## MCP server setup
```json
{
"mcpServers": {
"apify": {
"command": "npx",
"args": [
"mcp-remote",
"https://mcp.apify.com/?tools=crawloop/siemens-sieportal-product-catalog-crawler",
"--header",
"Authorization: Bearer "
]
}
}
}
```
## OpenAPI specification
```json
{
"openapi": "3.0.1",
"info": {
"title": "Siemens Catalog Crawler — MPN Discovery",
"description": "Search Siemens SiePortal live by keywords (6ES, SIMATIC, S7-1200) via OneSearch. Export unique MPNs, titles, category paths & PDP links. Deduplicated output with resume for large catalog builds.",
"version": "2.2",
"x-build-id": "hFfn3NVzUYbLKnIax"
},
"servers": [
{
"url": "https://api.apify.com/v2"
}
],
"paths": {
"/acts/crawloop~siemens-sieportal-product-catalog-crawler/run-sync-get-dataset-items": {
"post": {
"operationId": "run-sync-get-dataset-items-crawloop-siemens-sieportal-product-catalog-crawler",
"x-openai-isConsequential": false,
"summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
"tags": [
"Run Actor"
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/inputSchema"
}
}
}
},
"parameters": [
{
"name": "token",
"in": "query",
"required": true,
"schema": {
"type": "string"
},
"description": "Enter your Apify token here"
}
],
"responses": {
"200": {
"description": "OK"
}
}
}
},
"/acts/crawloop~siemens-sieportal-product-catalog-crawler/runs": {
"post": {
"operationId": "runs-sync-crawloop-siemens-sieportal-product-catalog-crawler",
"x-openai-isConsequential": false,
"summary": "Executes an Actor and returns information about the initiated run in response.",
"tags": [
"Run Actor"
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/inputSchema"
}
}
}
},
"parameters": [
{
"name": "token",
"in": "query",
"required": true,
"schema": {
"type": "string"
},
"description": "Enter your Apify token here"
}
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/runsResponseSchema"
}
}
}
}
}
}
},
"/acts/crawloop~siemens-sieportal-product-catalog-crawler/run-sync": {
"post": {
"operationId": "run-sync-crawloop-siemens-sieportal-product-catalog-crawler",
"x-openai-isConsequential": false,
"summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
"tags": [
"Run Actor"
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/inputSchema"
}
}
}
},
"parameters": [
{
"name": "token",
"in": "query",
"required": true,
"schema": {
"type": "string"
},
"description": "Enter your Apify token here"
}
],
"responses": {
"200": {
"description": "OK"
}
}
}
}
},
"components": {
"schemas": {
"inputSchema": {
"type": "object",
"properties": {
"searchKeywords": {
"title": "Search keywords",
"type": "array",
"description": "What to look for on Siemens website. One term per line.
Examples: 6ES, SIMATIC, S7-1200, contactor",
"default": [
"6ES",
"SIMATIC",
"S7-1200"
],
"items": {
"type": "string"
}
},
"discoveryMode": {
"title": "What to crawl",
"enum": [
"keywords",
"full_catalog",
"catalog_tree"
],
"type": "string",
"description": "Search my keywords — recommended. Uses the list above.
Full Siemens catalog — all products, very slow.
Category pages — experimental.",
"default": "keywords"
},
"locale": {
"title": "Catalog region",
"enum": [
"en-ww",
"en-nl",
"de-de",
"en-us",
"fr-fr",
"en-gb"
],
"type": "string",
"description": "Siemens website region. Default: Worldwide (English).",
"default": "en-ww"
},
"proxyConfiguration": {
"title": "Proxy",
"type": "object",
"description": "Keep Apify residential proxy enabled (recommended).",
"default": {
"useApifyProxy": true,
"apifyProxyGroups": [
"RESIDENTIAL"
]
}
},
"maxSearchShards": {
"title": "Test limit (number of searches)",
"minimum": 0,
"type": "integer",
"description": "For a quick test use 3. For a full run use 0 (no limit).",
"default": 3
},
"concurrencyLimit": {
"title": "Parallel speed",
"minimum": 1,
"maximum": 10,
"type": "integer",
"description": "How many searches run in parallel. Default 3. Use 1 if errors appear.",
"default": 3
},
"deduplicateOutput": {
"title": "Remove duplicates",
"type": "boolean",
"description": "Each article number appears only once in results.",
"default": true
},
"resumeFromCheckpoint": {
"title": "Continue previous run",
"type": "boolean",
"description": "Skip keywords already finished in a previous run.",
"default": false
}
}
},
"runsResponseSchema": {
"type": "object",
"properties": {
"data": {
"type": "object",
"properties": {
"id": {
"type": "string"
},
"actId": {
"type": "string"
},
"userId": {
"type": "string"
},
"startedAt": {
"type": "string",
"format": "date-time",
"example": "2025-01-08T00:00:00.000Z"
},
"finishedAt": {
"type": "string",
"format": "date-time",
"example": "2025-01-08T00:00:00.000Z"
},
"status": {
"type": "string",
"example": "READY"
},
"meta": {
"type": "object",
"properties": {
"origin": {
"type": "string",
"example": "API"
},
"userAgent": {
"type": "string"
}
}
},
"stats": {
"type": "object",
"properties": {
"inputBodyLen": {
"type": "integer",
"example": 2000
},
"rebootCount": {
"type": "integer",
"example": 0
},
"restartCount": {
"type": "integer",
"example": 0
},
"resurrectCount": {
"type": "integer",
"example": 0
},
"computeUnits": {
"type": "integer",
"example": 0
}
}
},
"options": {
"type": "object",
"properties": {
"build": {
"type": "string",
"example": "latest"
},
"timeoutSecs": {
"type": "integer",
"example": 300
},
"memoryMbytes": {
"type": "integer",
"example": 1024
},
"diskMbytes": {
"type": "integer",
"example": 2048
}
}
},
"buildId": {
"type": "string"
},
"defaultKeyValueStoreId": {
"type": "string"
},
"defaultDatasetId": {
"type": "string"
},
"defaultRequestQueueId": {
"type": "string"
},
"buildNumber": {
"type": "string",
"example": "1.0.0"
},
"containerUrl": {
"type": "string"
},
"usage": {
"type": "object",
"properties": {
"ACTOR_COMPUTE_UNITS": {
"type": "integer",
"example": 0
},
"DATASET_READS": {
"type": "integer",
"example": 0
},
"DATASET_WRITES": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_READS": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_WRITES": {
"type": "integer",
"example": 1
},
"KEY_VALUE_STORE_LISTS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_READS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_WRITES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_INTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_EXTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_SERPS": {
"type": "integer",
"example": 0
}
}
},
"usageTotalUsd": {
"type": "number",
"example": 0.00005
},
"usageUsd": {
"type": "object",
"properties": {
"ACTOR_COMPUTE_UNITS": {
"type": "integer",
"example": 0
},
"DATASET_READS": {
"type": "integer",
"example": 0
},
"DATASET_WRITES": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_READS": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_WRITES": {
"type": "number",
"example": 0.00005
},
"KEY_VALUE_STORE_LISTS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_READS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_WRITES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_INTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_EXTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_SERPS": {
"type": "integer",
"example": 0
}
}
}
}
}
}
}
}
}
}
```