# Poland Public Tenders Scraper (`toughtcrime/poland-public-tenders-scraper`) Actor Extract and normalize Polish public procurement notices from BZP and TED into one structured dataset. Filter by source, date, keywords, CPV codes, and supported fields, then export to JSON or CSV or use through the Apify API. - **URL**: https://apify.com/toughtcrime/poland-public-tenders-scraper.md - **Developed by:** [Vladyslav Hlovatskyi](https://apify.com/toughtcrime) (community) - **Categories:** Developer tools, Automation - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.10 / 1,000 normalized tenders This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. - **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use). - **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`. - **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`. - **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`. - **Any other language** — the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Poland Public Tenders API — BZP + TED **Poland Public Tenders Scraper · Public Beta · version 1.0.0 · Apify Actor source version 1.0** One normalized data source for Polish national and EU procurement notices. Collect BZP/e-Zamówienia and Poland-scoped TED procurement data through one Actor, one normalized schema, and one integration. Run bounded historical exports or scheduled incremental monitoring, apply supported procurement filters, and deliver structured records to JSON, CSV, Apify API integrations, data pipelines, CRMs, and analytics tools. Most Polish procurement integrations need separate handling for national BZP notices and EU-level TED notices. Their identifiers, formats, pagination, schemas, and incremental state differ. This Actor provides one integration for both, while keeping source provenance visible. ### Why use this Actor? - **Stop integrating BZP and TED separately.** Select either source or both in the same Actor input. - **Build on normalized data.** TED-only and combined runs use the source-independent `pl-procurement-v1` contract; BZP-only runs preserve the established Tender Schema v1 compatibility path. - **Run history or monitoring.** Use bounded date ranges for exports and durable source-specific cursors for recurring incremental collection. - **Trust incomplete runs less, not more.** Partial or limited processing does not incorrectly advance durable state. - **Operate with evidence.** Stable source identifiers, official links, conservative candidate links, and machine-readable run summaries remain available to downstream systems. ### BZP and TED in one integration | Capability | Typical BZP-only integration | This Actor | | --- | ---: | ---: | | BZP notices | Yes | Yes | | Poland-scoped TED notices | No | Yes | | Unified cross-source schema | No | Yes | | Historical bounded runs | Varies | Yes | | Incremental monitoring | Varies | Yes | | Independent source state | No | Yes | | Partial-run cursor protection | Varies | Yes | | Conservative BZP/TED candidate linking | No | Yes | | Machine-readable run summaries | Varies | Yes | | JSON and CSV access | Usually | Yes (qualified) | **CSV note:** JSON is the recommended unified output. The documented fixed 42-column CSV is available for BZP-only compatibility; Apify can also export Dataset items to CSV, but nested unified fields are best consumed as JSON. ### Key features - **Two official public source interfaces:** BZP/e-Zamówienia and the TED Search API for notices involving a Polish buyer, a place of performance in Poland, or either condition. - **Unified procurement records:** one `pl-procurement-v1` contract for TED-only and combined BZP + TED runs, without breaking the BZP-only Tender Schema v1 default. - **Stable identifiers:** notice and procedure IDs derive from official source identity or documented immutable fallbacks, never mutable titles or deadlines. - **Historical and incremental modes:** bounded exports and recurring feeds use the same validated input and normalization pipeline. - **Durable TED state:** temporary TED iteration tokens remain client-local; persistent state uses a semantic cursor based on publication and notice identity. - **Partial-run protection:** failed, incomplete, unsupported-format, and delivery-limited TED scans do not incorrectly commit cursor or item state. - **Deterministic change detection:** normalized business changes produce stable events; retry noise, retrieval timestamps, ordering, and URL query noise do not. - **Conservative source linking:** possible BZP/TED representations can be marked as candidates for review, but weak similarity never merges records. - **Structured run summaries:** inspect fetched, normalized, filtered, emitted, duplicate and unsupported counts, API requests, retries, partial status, cursor decisions, warnings, and charged-event totals. ### Supported sources - **BZP only:** preserves the established Tender Schema v1 and its stable CSV projection. - **TED only:** returns the source-independent `pl-procurement-v1` JSON schema. - **BZP + TED:** returns both sources through `pl-procurement-v1`; conservative candidate links may be added for review without merging source records. Source-specific limitations remain explicit. For example, BZP buyer-region and city filters do not have an equivalent verified TED mapping and are rejected when TED is selected. ### Use cases - **Tender monitoring:** schedule collection of newly published or updated procurement records. - **Procurement consulting:** serve clients that require national BZP and Poland-relevant TED data through one maintained integration. - **CRM and sales workflows:** use Apify Tasks, schedules, webhooks, the API, or Dataset exports to build delivery into a CRM, Slack, Microsoft Teams, Make, n8n, Zapier, or an internal service. These are integration possibilities, not bundled ready-made connectors. - **Market intelligence:** analyze available buyer, CPV, source, location, publication, and procedure metadata. - **Data products and internal APIs:** supply procurement dashboards, SaaS products, search tools, reports, ERP integrations, and internal services. - **Historical exports:** extract a bounded date range without changing durable TED incremental state by default. The Actor is designed for developers, automation agencies, procurement consultants, tender and bid teams, data analysts, market-intelligence teams, and companies integrating procurement information into CRMs, ERPs, dashboards, or internal services. It is a data component, not a tender eligibility, legal-advice, or bid-recommendation service. ### Quick start 1. Open the Actor's **Input** tab in Apify Console. 2. Paste one of the small examples below. 3. Select **Start**. 4. Open **Output** for the Dataset and **Run summary** for completeness, warnings, cursor safety, and links. Date ranges can validly return zero records. Start with a narrow range and a small `maxItems`; source acquisition work can be larger than the delivered result count. ### Input examples #### Small combined BZP + TED run This inexpensive snapshot uses `pl-procurement-v1` for both sources. Snapshot runs do not advance incremental state. ```json { "mode": "snapshot", "sources": ["BZP", "TED"], "publishedFrom": "2025-07-31", "publishedTo": "2025-07-31", "maxItems": 5, "includeRawData": false } ``` #### Small TED-only bounded run This replay does not advance durable TED state because `advanceBoundedTedState` remains false. ```json { "mode": "incremental", "sources": ["TED"], "incrementalFrom": "2025-07-31T00:00:00Z", "incrementalTo": "2025-08-01T00:00:00Z", "advanceBoundedTedState": false, "stateKey": "ted-bounded-sample", "maxItems": 5, "includeRawData": false } ``` The output uses `pl-procurement-v1`. Change the date window if the official TED historical window no longer returns records. #### Small BZP-only run This compatibility snapshot uses Tender Schema v1 and exposes the stable 42-column CSV projection. It does not advance incremental state. ```json { "mode": "snapshot", "sources": ["BZP"], "publishedFrom": "2025-07-31", "publishedTo": "2025-07-31", "maxItems": 5, "includeRawData": false } ``` ### Output example When TED is selected, each Dataset item uses the unified contract. This concise example is based on the repository's sanitized official TED fixture: ```json { "schemaVersion": "pl-procurement-v1", "recordType": "procedure", "source": { "system": "TED", "sourceNoticeId": "e2c98c9a-af8b-44a8-9be1-4476f6ac55ad", "sourceProcedureId": "b4dabe7c-5b9b-4a08-94f9-6fd398ba0319", "sourceVersion": "2", "publicationId": "499972-2025", "sourceUrl": "https://ted.europa.eu/pl/notice/-/detail/499972-2025", "xmlUrl": "https://ted.europa.eu/en/notice/499972-2025/xml", "retrievedAt": "2025-08-01T10:00:00Z" }, "canonicalNoticeId": "ted:notice:e2c98c9a-af8b-44a8-9be1-4476f6ac55ad", "procedureId": "ted:procedure:b4dabe7c-5b9b-4a08-94f9-6fd398ba0319", "notices": [ { "canonicalNoticeId": "ted:notice:e2c98c9a-af8b-44a8-9be1-4476f6ac55ad", "sourceNoticeId": "e2c98c9a-af8b-44a8-9be1-4476f6ac55ad", "sourceVersion": "2", "publicationId": "499972-2025", "publicationDate": "2025-07-31" } ], "noticeVersion": 2, "noticeType": "can-standard", "formType": "result", "procedureStatus": "awarded", "procurementType": null, "procedureType": null, "publicationDate": "2025-07-31", "dispatchDate": "2025-07-30", "titles": [ {"language": "pol", "text": "Zakup średniego samochodu ratowniczo-gaśniczego"} ], "descriptions": [], "buyers": [ { "name": "OCHOTNICZA STRAŻ POŻARNA W ŚWIESZYNIE", "identifiers": ["NIP: 4990332417", "REGON: 331251384"], "country": "POL", "roles": ["buyer"] } ], "suppliers": [], "lots": [], "cpvCodes": ["34144210"], "placesOfPerformance": [ {"country": "POL", "nutsCode": "PL426", "description": null} ], "estimatedValue": {"amount": 1338900, "currency": "PLN", "valueType": "estimated"}, "awardedValue": {"amount": 1382000, "currency": "PLN", "valueType": "awarded"}, "submissionDeadline": null, "awards": [], "contracts": [], "documents": [], "links": [ {"linkType": "unlinked", "confidence": 0, "targetProcedureId": null, "evidence": []} ], "quality": { "noticeFormat": "eforms", "sourceSchemaVersion": "1.12", "supported": true, "warnings": [] }, "events": [], "firstSeenAt": "2025-08-01T10:00:00Z", "lastSeenAt": "2025-08-01T10:00:00Z" } ``` The default Dataset contains only public procurement records. The machine- readable `OUTPUT` Key-Value Store record contains `run-summary-v1`, counters, warnings, source outcomes, Dataset links, cursor decisions, and charged-event count. It never contains credentials or complete raw notices. ### Supported filters Common filters include publication/update/deadline dates, keywords, CPV codes, buyer names, statuses, procurement types, procedure types, and value bounds. Filter groups use AND; values inside a group generally use OR according to the field's documented matching option. `regions` and `cities` depend on the verified BZP buyer-address mapping and are rejected when TED is selected because no equivalent verified TED field exists. Unsupported combinations fail input validation rather than being silently ignored. ### Historical and incremental runs Use snapshot mode for bounded publication-date exports. Use incremental mode with a stable `stateKey` for a recurring feed. BZP and TED keep independent source state so one source's cursor is not mistaken for the other's. TED's temporary `ITERATION` pagination token describes only one point-in-time scan and is never stored as the durable cursor. A complete scan may advance the semantic source cursor; partial, failed, unsupported-format, or delivery-limited processing does not. Bounded `incrementalFrom`/`incrementalTo` replays leave TED state unchanged unless `advanceBoundedTedState` is explicitly enabled for an isolated verification partition. ### Change detection Incremental mode delivers newly observed records and, when `includeUpdates` is true, records whose normalized business content changed. Unified records can carry deterministic events such as notice creation/update, deadline, document, status, cancellation, award, contract, supplier, or value changes when the normalized source evidence supports them. This is not a simplified guaranteed `new`/`updated`/`awarded`/`cancelled` classification across every source record. Unchanged records, repeated TED notice versions, source-page overlap, and retry duplicates are suppressed. Candidate BZP/TED links are evidence for review; they do not silently merge procedures. ### API and automation Results are available in the default Dataset as JSON and through the standard Apify Dataset API. BZP-only runs also expose a run-scoped stable CSV projection in the documented 42-column order. Unified BZP/TED runs should use JSON because nested multilingual and provenance fields cannot be represented faithfully in the legacy flat CSV contract. Save an input as an Apify Task for schedules, connect a webhook for terminal run statuses, or start the Actor through the official Apify API/client. Keep `APIFY_TOKEN` in secrets and send it in an Authorization header, never a shared URL. ### Pricing Public Beta uses **Pay per event + platform usage**: - **$0.0001 per `normalized-tender`** — $0.10 per 1,000 normalized tender records; - a small one-time Actor-start event for each run; - Apify platform usage billed separately under the user's Apify plan. A `normalized-tender` event is charged only after one unique normalized procurement record is successfully delivered to the public default Dataset. There is no custom tender-event charge for run summaries, warnings, filtered, duplicate, rejected or unsupported records, retries, API requests, candidate links, or failed Dataset delivery. The maximum run charge selected by the user is checked before delivery and respected. Platform pricing and allowances can change; review the current run estimate and [Apify pricing](https://apify.com/pricing). Implementation details are in [docs/monetization.md](docs/monetization.md). ### Data quality and limitations - This is an unofficial, community-maintained Actor, not an authority-operated procurement service. - Coverage is limited to the verified public BZP and TED interfaces; it does not include every Polish commercial or private tender. - Upstream availability, publication timing, and source metadata depend on the official systems. Data can be incomplete, delayed, corrected, multilingual, or internally inconsistent. - BZP-specific buyer-region and city filters do not have exact verified TED equivalents. - Legacy non-eForms TED notices can be detected and reported as unsupported but are not normalized using guessed eForms assumptions. - Candidate cross-source links are suggestions for review, not confirmed BZP/TED merges. - `partial: true` or source warnings mean the delivered subset must not be treated as an exhaustive scan. - Metadata normalization does not replace the original notice. Verify legally significant deadlines, requirements, values, documents, and decisions at the linked source. - The Actor does not determine eligibility or suitability for a procurement procedure and provides no legal advice or completeness guarantee. - Raw payloads are unstable and opt-in. Attachments are not downloaded or searched. ### Responsible usage Both source flows use verified official public procurement interfaces with bounded concurrency, finite retries, and backoff for transient failures. The Actor uses no browser automation, HTML scraping, or proxy requirement. It does not assume that an upstream API is unrestricted or will remain available on unchanged terms. ### Troubleshooting - **Input rejected:** check required snapshot dates, paired incremental bounds, source-specific filters, and unknown fields. - **Zero records:** broaden the date range or remove filters; zero can be a valid successful result. - **Partial run:** inspect `OUTPUT.sources`, warnings, and `cursorNotAdvancedReason`, then rerun without manually moving state. - **Private export says insufficient permissions:** download from Console or send `Authorization: Bearer …`; do not put tokens in the URL. - **Budget/limit reached:** raise the run maximum only if intended, or reduce the date range and `maxItems`. ### FAQ #### Does the Actor merge BZP and TED records automatically? No. It may emit conservative candidate links when exact normalized title or buyer-identifier evidence overlaps. Candidate links never merge source state or claim an authoritative relationship. #### Can I monitor tenders on a schedule? Yes. Save a tested incremental input as an Apify Task and attach an Apify Schedule. Avoid overlapping runs that use the same `stateKey`, and inspect the run summary before treating a scan as complete. #### Which output format should I use? Use JSON for TED-only and combined BZP + TED runs because it preserves nested provenance, multilingual text, organizations, lots, and locations. BZP-only runs can also use the documented stable 42-column CSV projection. #### Does a bounded TED replay change my monitoring cursor? Not by default. Keep `advanceBoundedTedState: false`. Enabling it is intended only for deliberate verification with an isolated `stateKey`. #### Are summaries, retries, and rejected records charged as tenders? No custom `normalized-tender` event is charged for them. A run still has its Actor-start event and separate Apify platform usage. ### Support When reporting an issue, include the Actor build number, run ID, sanitized input, source, status, and warning/error codes. Never include API tokens, Authorization headers, or full confidential payloads. Source-contract defects should include the smallest reproducible official request. ### Unofficial-product disclaimer This independent product is not affiliated with or endorsed by BZP, UZP, e-Zamówienia, TED, the European Union, or any Polish or EU procurement authority. Official source records remain authoritative. Verify legally significant information in the linked original notice. Normalized data is not legal advice and cannot guarantee legal completeness or procurement eligibility. ### Technical documentation The remainder of this README is the detailed product and data contract. Start with the [schema guide](docs/schema.md), [TED integration notes](docs/TED_INTEGRATION.md), [operations guide](docs/operations.md), [source audit](docs/source-audit.md), and [architecture decisions](docs/architecture.md). The original BZP-first sections remain authoritative for BZP-only backward compatibility; section 31 adds TED without weakening that contract. ### 1. Product summary Poland Public Tenders Scraper is an Apify Actor that retrieves Polish public-procurement notices from the official BZP and TED APIs, aggregates related notices into stable records, filters them, and writes structured procedures to the run's default Apify Dataset. The Public Beta covers two verified official source families: - **Source:** Biuletyn Zamówień Publicznych (BZP) on the Polish e-Zamówienia platform. - **Official integration page:** `https://ezamowienia.gov.pl/pl/integracja/` - **Current endpoint advertised by the official integration page at specification time:** `https://ezamowienia.gov.pl/mo-board/api/v1/notice` - **TED:** official EU TED Search API v3, documented in section 31 and `docs/TED_INTEGRATION.md`. The official integration page states that reading BZP notices and notice statistics is available through the BZP WebService and does not require the integration procedure required by the platform's write APIs. The implementation must still verify the current contract, parameters, endpoint family, and terms before coding the client. This project is not an official Polish government product and must not be presented as one. ### 2. MVP goals The MVP must provide: - official BZP and TED procurement sources; - two run modes: `snapshot` and `incremental`; - filters by keywords, CPV codes, dates, buyers, geography, status, procurement type, procedure type, and value; - stable structured output with schema versioning; - JSON and CSV access through Apify Dataset exports; - one Dataset item per unique procurement procedure; - optional detection and delivery of updated procedures; - Apify pay-per-event monetization aligned to records delivered; - an English user-facing README and examples; - production-grade validation, retries, tests, CI, and deployment configuration. ### 3. Explicit non-goals Do not implement these in the MVP: - nationwide multi-source monitoring; - additional procurement sources beyond BZP and TED; - HTML scraping or browser automation fallback; - Playwright, Selenium, Puppeteer, or Crawlee; - proxy rotation or rate-limit circumvention; - FastAPI or another custom public API server; - PostgreSQL or another external database; - automatic translation; - fuzzy search, stemming, typo correction, or semantic search; - downloading or parsing PDF, ZIP, or other tender attachments; - keyword search inside attachments; - external enrichment of buyers or suppliers; - KRS, VAT, financial, employee, or contact enrichment; - coordinates, radius search, or “near me” features; - complete historical version storage; - procurement plans, general statistics, or records that cannot be tied reliably to a procedure; - a custom scheduler inside the Actor. ### 4. Core record definition #### 4.1 One paid item equals one procurement procedure A Dataset item represents one unique procurement procedure, not one technical notice version. Related BZP notices must be aggregated into the current state of that procedure, including, where available: - procurement notices; - changes and corrections; - cancellations; - procedure results; - contract notices; - contract-execution notices. One procedure containing multiple lots is still one Dataset item and one paid result. #### 4.2 Procedure identity Output must contain: - `tenderId`: the scraper's stable identifier; - `sourceTenderId`: the official stable BZP procedure identifier when available; - `notices`: metadata for all reliably linked notices. Identity priority: 1. Use the official stable procedure identifier, such as the verified BZP/OCDS procedure ID, as the source identity. 2. If unavailable, use immutable source links between notices. 3. Only then use a deterministic composite key based on stable source fields. 4. Never use mutable fields such as title, deadline, status, or buyer contact information as the sole identity. 5. Never merge uncertain matches automatically. Keep them separate and emit a warning. `tenderId` must not change when the title, deadline, notice version, status, or award information changes. ### 5. Run modes #### 5.1 `snapshot` A one-time retrieval for a publication-date range. Requirements: - `publishedFrom` and `publishedTo` are required. - Both boundaries are inclusive. - The maximum requested range is 365 calendar days. - Future dates are valid and may return zero records. - The Actor does not read or mutate incremental state. - Every delivered record has `changeType: "new"` because no persistent comparison is performed. - Reaching `maxItems` is an intentional limit, not a partial-source failure. #### 5.2 `incremental` A persistent monitoring run that returns new and, optionally, updated procedures. Requirements: - State is isolated by `stateKey`. - First run without a cursor uses `initialLookbackDays`. - Later runs use the last successful cursor with `overlapHours`. - Optional `incrementalFrom` and `incrementalTo` boundaries support a finite, replayable source window. The cursor is clamped to that range and a complete run commits its exclusive ending instant. - Cursor advancement happens only after a complete candidate scan. - BZP may persist successfully handled item observations during a partial run, but its cursor never advances. - TED stages item observations in memory and persists them only with a complete cursor commit; limited, warning-bearing, or failed TED runs leave its durable cursor and item observations unchanged. - `includeUpdates` controls whether changed procedures are delivered. - Unchanged procedures are never written to the Dataset. - Different `stateKey` values may run concurrently. - Concurrent runs using the same `stateKey` must be rejected using the strongest lock/lease mechanism available without an external database. - `resetState` deletes only the selected state partition before the run. Large historical backfills must use `snapshot`, not a very large incremental lookback. ### 6. Input contract The Actor input is a JSON object. Unknown properties are forbidden. Validate the complete input before the first BZP request. #### 6.1 Input fields | Field | Type | Default | Rules | |---|---:|---:|---| | `mode` | `"snapshot" \| "incremental"` | required | Selects run mode. | | `publishedFrom` | `YYYY-MM-DD \| null` | `null` | Required in snapshot. Optional business filter in incremental. | | `publishedTo` | `YYYY-MM-DD \| null` | `null` | Required in snapshot. Inclusive full day. | | `incrementalFrom` | RFC 3339 timestamp or `null` | `null` | Inclusive incremental acquisition instant. Must be paired with `incrementalTo`. | | `incrementalTo` | RFC 3339 timestamp or `null` | `null` | Exclusive incremental acquisition instant. Maximum paired range: 365 days. | | `updatedFrom` | `YYYY-MM-DD \| null` | `null` | Filters by `sourceUpdatedAt`, falling back to `latestNoticeDate`. | | `updatedTo` | `YYYY-MM-DD \| null` | `null` | Inclusive full day. | | `deadlineFrom` | `YYYY-MM-DD \| null` | `null` | Filters the current submission deadline. | | `deadlineTo` | `YYYY-MM-DD \| null` | `null` | Inclusive full day. | | `keywords` | `string[]` | `[]` | Phrases searched in `title` and `description`. | | `keywordMatch` | `"any" \| "all"` | `"any"` | Match any phrase or all phrases. | | `cpvCodes` | `string[]` | `[]` | Match primary and additional CPV codes. | | `cpvMatch` | `"exact" \| "prefix"` | `"prefix"` | Exact normalized code or CPV hierarchy prefix. | | `buyerNames` | `string[]` | `[]` | Official buyer names only. | | `buyerNameMatch` | `"contains" \| "exact"` | `"contains"` | OR between buyer names. | | `buyerNationalIds` | `string[]` | `[]` | Exact match against official IDs such as NIP/REGON when present. | | `regions` | `string[]` | `[]` | Polish voivodeships; OR within the array. | | `cities` | `string[]` | `[]` | Buyer cities; OR within the array. | | `statuses` | `TenderStatus[]` | `[]` | OR within the array. | | `procurementTypes` | `ProcurementType[]` | `[]` | OR within the array. | | `procedureTypes` | `string[]` | `[]` | Verified normalized procedure codes; OR within the array. | | `minValue` | non-negative number or `null` | `null` | See value-filter semantics. | | `maxValue` | non-negative number or `null` | `null` | Must be greater than or equal to `minValue`. | | `maxItems` | integer | `100` | Inclusive range `1..100000`; counts unique delivered procedures. Values above 1000 can materially increase time and cost. | | `includeUpdates` | boolean | `true` | Relevant to incremental mode. | | `includeRawData` | boolean | `false` | Adds unstable `rawData` to JSON records. | | `initialLookbackDays` | integer | `7` | Incremental only; range `1..365`. | | `overlapHours` | integer | `24` | Incremental only; range `1..168`. | | `stateKey` | string | `"default"` | Incremental state partition. Pattern `[A-Za-z0-9_-]{1,64}`. | | `resetState` | boolean | `false` | Allowed only in incremental mode. | #### 6.2 Date validation - `snapshot` requires both `publishedFrom` and `publishedTo`. - `publishedFrom <= publishedTo`. - Snapshot publication range may not exceed 365 days. - `incrementalFrom` and `incrementalTo` are incremental-only aware timestamps, must be supplied together with `from < to`, and may span at most 365 days. - For every optional business-date pair, if both values exist, `from <= to`. - A single lower or upper boundary is allowed for optional business-date filters, but not for the bounded incremental acquisition pair. - Business-filter and snapshot date inputs are calendar dates. Bounded incremental acquisition inputs are UTC-normalized aware timestamps at whole second precision. - Interpret inclusive day boundaries in `Europe/Warsaw`, then compare using UTC instants. In ordinary incremental mode, acquisition is driven by cursor/lookback and overlap. `incrementalFrom` and `incrementalTo` instead clamp that cursor-driven acquisition to a finite historic range. The other date fields remain business filters applied after aggregation and never replace cursor logic. #### 6.3 String and array normalization Before validation/fingerprinting: - trim leading and trailing whitespace; - Unicode-normalize strings to NFC; - collapse repeated internal whitespace where matching requires it; - reject empty strings; - remove duplicate array values while preserving first-seen order; - compare text using Unicode-aware `casefold()`; - preserve Polish diacritics and never transliterate them for matching. #### 6.4 Keyword matching - Search only `title` and `description`. - Each element of `keywords` is one phrase. - Do not split a phrase into independent words. - `keywordMatch: "any"` means at least one phrase is present. - `keywordMatch: "all"` means every phrase is present. - Empty `keywords` disables the keyword filter. - No fuzzy matching, stemming, typo correction, translation, or attachment search. #### 6.5 CPV matching Accept canonical CPV input as eight digits, optionally followed by the official check-digit suffix, for example `72000000` or `72000000-5`. Normalize matching to the eight-digit base code. - `exact`: normalized eight-digit equality. - `prefix`: hierarchy match using the meaningful prefix obtained by removing trailing zero placeholders from the requested eight-digit code. Example: `72000000` represents the `72` hierarchy prefix. - Match against both primary and additional CPV codes. - Multiple requested codes use OR logic. - Invalid CPV formats fail input validation. The source audit must verify whether the API exposes check digits and how they should be retained in raw data. Stable output uses the normalized eight-digit base code. #### 6.6 Geography Canonical supported voivodeships: - `dolnośląskie` - `kujawsko-pomorskie` - `lubelskie` - `lubuskie` - `łódzkie` - `małopolskie` - `mazowieckie` - `opolskie` - `podkarpackie` - `podlaskie` - `pomorskie` - `śląskie` - `świętokrzyskie` - `warmińsko-mazurskie` - `wielkopolskie` - `zachodniopomorskie` Match regions and cities case-insensitively after whitespace normalization. Output values remain Polish. #### 6.7 Status enum `TenderStatus`: - `open` - `closed` - `awarded` - `cancelled` - `unknown` Prefer an explicit official source status. If a deterministic derived status is required, document the precedence in `docs/source-audit.md` and test it. A suggested fallback precedence is: 1. explicit cancellation → `cancelled`; 2. published award/result indicating completion → `awarded`; 3. active deadline in the future → `open`; 4. expired deadline without a published result → `closed`; 5. insufficient or contradictory data → `unknown`. #### 6.8 Procurement type enum `ProcurementType`: - `services` - `supplies` - `works` - `mixed` - `unknown` Use official source data. Do not infer solely from CPV when source data conflicts. #### 6.9 Procedure types The source audit must enumerate current official procedure-type values and map them to stable lowercase English machine codes. Output contains: - `procedureType`: stable normalized code; - `procedureTypeOriginal`: original Polish value. New or unmapped source values must become `unknown`, not crash the Actor. Input `procedureTypes` accepts only implemented normalized codes, including `unknown`. #### 6.10 Value filters - Values are never converted between currencies. - Internal calculations use decimal-safe arithmetic. - Missing or confidential values are `null`. - A tender without usable value data does not pass a value filter. - For the MVP, value filters apply only to records whose normalized top-level currency is `PLN`; records in another or unknown currency do not pass. - Treat a tender value as an interval: - exact estimate: `[estimatedValue, estimatedValue]`; - source range: `[valueMin, valueMax]`; - one known bound: use only that bound for the relevant comparison. - With both `minValue` and `maxValue`, the tender passes when its known value interval intersects the requested interval. #### 6.11 Cross-filter logic - Different filter groups use AND. - Values inside an array use OR, except keyword behavior is selected by `keywordMatch`. - An absent or empty filter group imposes no restriction. - Filters apply to the current aggregated procedure state. - `OUTPUT.effectiveInput` must contain the normalized input actually applied. ### 7. Stable output contract: Tender Schema v1 Every Dataset item is one JSON object. Field names are English and `camelCase`. Source text values remain in Polish. Every stable field must exist, using `null` for missing scalars and `[]` for missing collections, except optional `rawData`, which is absent when disabled. #### 7.1 Top-level schema | Field | Type | Required | Meaning | |---|---|---:|---| | `schemaVersion` | string | yes | Always `"1.0"` for this contract. | | `tenderId` | string | yes | Stable scraper identifier. | | `sourceTenderId` | string or `null` | yes | Official procedure ID when available. | | `source` | string | yes | Always `"BZP"`. | | `sourceLanguage` | string | yes | Always `"pl"`. | | `sourceUrl` | string or `null` | yes | Best official URL for the current procedure/notice. | | `title` | string | yes | Current official title. | | `description` | string or `null` | yes | Current official subject/description. | | `publicationDate` | date or timestamp string | yes | First procurement-notice publication date. | | `latestNoticeDate` | date or timestamp string or `null` | yes | Most recent related notice publication. | | `sourceUpdatedAt` | timestamp string or `null` | yes | Source update timestamp when available. | | `submissionDeadline` | date or timestamp string or `null` | yes | Current deadline. | | `status` | `TenderStatus` | yes | Stable status enum. | | `procurementType` | `ProcurementType` | yes | Stable subject type. | | `procedureType` | string | yes | Stable normalized procedure code. | | `procedureTypeOriginal` | string or `null` | yes | Original Polish source value. | | `primaryCpvCode` | string or `null` | yes | Normalized eight-digit base code. | | `cpvCodes` | string\[] | yes | Unique primary and additional normalized codes. | | `estimatedValue` | number or `null` | yes | Exact estimate when available. | | `valueMin` | number or `null` | yes | Lower estimate bound. | | `valueMax` | number or `null` | yes | Upper estimate bound. | | `awardedValue` | number or `null` | yes | Aggregate awarded value when reliably available. | | `currency` | string or `null` | yes | ISO 4217 code. | | `buyerName` | string or `null` | yes | Official buyer name. | | `buyerNationalId` | string or `null` | yes | Official NIP/REGON/other ID as supplied. | | `buyerCity` | string or `null` | yes | Buyer city. | | `buyerRegion` | string or `null` | yes | Buyer voivodeship. | | `buyerPostalCode` | string or `null` | yes | Buyer postal code. | | `buyerCountry` | string or `null` | yes | ISO 3166-1 alpha-2, normally `PL`. | | `buyerWebsite` | string or `null` | yes | Officially published website. | | `buyerEmail` | string or `null` | yes | Officially published email. | | `buyerPhone` | string or `null` | yes | Officially published phone. | | `contactPersonName` | string or `null` | yes | Officially published contact person. | | `lots` | `Lot[]` | yes | Procedure lots. | | `documents` | `Document[]` | yes | Attachment metadata only. | | `awards` | `Award[]` | yes | Published results and suppliers. | | `notices` | `Notice[]` | yes | Reliably linked notice metadata, newest first. | | `changeType` | `"new" \| "updated"` | yes | Delivery reason. Snapshot records are `new`. | | `changedFields` | string\[] | yes | Changed top-level business fields; empty for new records. | | `firstSeenAt` | timestamp string | yes | First observed by this state partition/run context. | | `lastSeenAt` | timestamp string | yes | Most recent successful observation. | | `fetchedAt` | timestamp string | yes | Current fetch time. | | `rawData` | object | no | Unstable source payload when requested. | #### 7.2 Lot schema Each `lots` element contains all fields below: | Field | Type | |---|---| | `lotId` | string or `null` | | `title` | string or `null` | | `description` | string or `null` | | `cpvCodes` | string\[] | | `estimatedValue` | number or `null` | | `valueMin` | number or `null` | | `valueMax` | number or `null` | | `currency` | string or `null` | | `deadline` | date or timestamp string or `null` | | `status` | `TenderStatus` or `null` | Preserve deterministic lot order using an official sequence/identifier when available, otherwise a documented stable fallback. #### 7.3 Document schema Each `documents` element contains: | Field | Type | |---|---| | `documentId` | string or `null` | | `title` | string or `null` | | `documentType` | string or `null` | | `url` | string | | `publishedAt` | date or timestamp string or `null` | Do not download files or extract their text. #### 7.4 Award schema Each `awards` element contains: | Field | Type | |---|---| | `awardId` | string or `null` | | `lotId` | string or `null` | | `supplierName` | string or `null` | | `supplierNationalId` | string or `null` | | `supplierCountry` | string or `null` | | `awardedValue` | number or `null` | | `currency` | string or `null` | | `awardDate` | date or timestamp string or `null` | | `contractDate` | date or timestamp string or `null` | | `status` | string or `null` | Return `[]` when no award information is officially available. Do not enrich suppliers externally. #### 7.5 Notice schema Each `notices` element contains: | Field | Type | |---|---| | `noticeId` | string | | `noticeNumber` | string or `null` | | `noticeType` | string | | `publicationDate` | date or timestamp string | | `sourceUpdatedAt` | timestamp string or `null` | | `sourceUrl` | string or `null` | | `isLatest` | boolean | Normalize notice types to stable codes after source audit. The supported lifecycle must cover procurement, change/correction, cancellation, result, contract, and contract-execution notices when present. Unknown values become `unknown`. #### 7.6 Critical fields A procedure may be delivered only when all of these are available: - `tenderId`; - `title`; - `publicationDate`; - at least one official source identifier or official URL. If a critical field cannot be established after retries and parsing, skip the procedure, emit a warning, and set `isPartial: true`. ### 8. Date and timezone rules - Normalize timestamps to UTC as `YYYY-MM-DDTHH:mm:ssZ`. - If BZP supplies local time without an offset, interpret it as `Europe/Warsaw`, including daylight-saving rules. - If the source supplies only a calendar date, preserve `YYYY-MM-DD`; do not invent midnight. - `publicationDate` is the first procurement-notice publication date and never changes because a result, correction, or cancellation was later published. - `latestNoticeDate` is the newest related notice publication date. - `sourceUpdatedAt` is the source's latest technical/content update when available. - `updatedFrom`/`updatedTo` use `sourceUpdatedAt`, falling back to `latestNoticeDate`. - Deadline filters use the current aggregated `submissionDeadline`. ### 9. Aggregation and normalization #### 9.1 Candidate retrieval Use the BZP list/search API to discover candidate notices for the relevant acquisition window. Apply source-side date filters and pagination where verified and reliable. Never assume the list response is complete enough for the final record. #### 9.2 Detail retrieval For every candidate procedure, retrieve all official detail data needed for: - title and description; - identifiers and notice relationships; - buyer and contact information; - CPV codes; - deadlines; - procedure and procurement type; - lots; - documents; - cancellations; - results, awards, and contracts; - values and currencies. Cache identical detail requests within one run. #### 9.3 Aggregation rules - Group only reliably linked notices. - Sort `notices` newest to oldest. - Exactly one notice has `isLatest: true` when notices are non-empty. - Build top-level fields from the current official procedure state. - Apply corrections/changes deterministically. - Cancellation and result notices must update status and relevant fields. - Changes to `notices` are business changes and can trigger `updated`. - Full historical source bodies are not stored unless represented in optional `rawData` for the current run. The exact BZP notice relationship and precedence model must be documented and fixture-tested after the source audit. ### 10. Change detection Create a canonical business payload and a stable hash. Exclude these fields from the hash: - `changeType`; - `changedFields`; - `firstSeenAt`; - `lastSeenAt`; - `fetchedAt`; - `rawData`; - any run-specific links or metrics. Include normalized business timestamps such as `sourceUpdatedAt`, and include `notices`, lots, documents, awards, buyer data, values, status, CPV data, title, and description. Canonicalization requirements: - deterministic key ordering; - deterministic collection ordering where source order is not semantic; - normalized Unicode and whitespace; - decimal-safe numeric normalization; - no unstable source-only fields; - hash generated from canonical UTF-8 JSON using `orjson` or an equally deterministic mechanism. For incremental mode: - unseen `tenderId` → `changeType: "new"`, `changedFields: []`; - changed hash → `changeType: "updated"` and a sorted list of changed top-level business fields; - unchanged hash → skip; - changed hash with `includeUpdates: false` → update observation state as appropriate but do not write or charge the item. Technical timestamps alone must never trigger `updated`. ### 11. Incremental state Use a named Apify Key-Value Store that persists across runs. Do not use the run's default ephemeral KVS for cross-run state. Suggested store name: `poland-public-tenders-state` Partition keys by sanitized `stateKey`. #### 11.1 State metadata Store per-state metadata including: - schema/state version; - normalized filter fingerprint; - last fully completed cursor; - last complete run ID and timestamp; - source contract/version marker when available; - lock/lease metadata. #### 11.2 Item state Store only: - `tenderId`; - canonical hash; - `firstSeenAt`; - `lastSeenAt`; - `sourceUpdatedAt`. Do not store the complete tender record in incremental state. Split item state across multiple KVS records, for example deterministic hash-prefix shards. Do not store all procedures in one unbounded JSON object. #### 11.3 Filter fingerprint Include business filters and matching semantics in the fingerprint. Exclude purely operational/output settings such as: - `maxItems`; - `includeRawData`; - `includeUpdates`; - `overlapHours`; - `resetState`. Bounded acquisition fields `incrementalFrom` and `incrementalTo` are included in the fingerprint when present. Their absent `null` defaults are omitted so existing unbounded state partitions retain their previous fingerprint. Reusing a bounded state partition for a different range would otherwise make its cursor ambiguous. If a state exists and its fingerprint differs, fail before source requests with an actionable message instructing the user to choose a new `stateKey` or use `resetState: true`. #### 11.4 Retention - Remove item-state entries not seen for 730 days. - Cleanup runs only after a fully successful incremental scan. - Never remove the cursor or filter fingerprint during retention cleanup. - Cleanup does not affect previous run Datasets. #### 11.5 Locking Reject overlapping runs with the same `stateKey`. Implement the strongest feasible distributed lease using Apify capabilities available at implementation time. Prefer an atomic or conditional-write facility if officially supported. Otherwise use a lease record with unique owner token, expiry, verification after acquisition, renewal, and safe release. Document any unavoidable race limitations. A stale lease must be recoverable. ### 12. Pagination, rate limits, and resilience - Use asynchronous `httpx`. - Concurrency and requests per second are internal, conservative settings, not user input. - Determine exact safe defaults from the source audit and benchmarks. - Send a descriptive `User-Agent` identifying the Actor. - Respect `Retry-After`. - On `429`, reduce request rate and back off. - Retry transient network errors and `5xx` responses with exponential backoff and jitter. - Use finite per-request timeouts. - Stop BZP partition scanning after two consecutive exhausted search contexts; report `BZP_UPSTREAM_UNAVAILABLE` and preserve source-failure cursor safety. - Do not retry permanent validation/client errors blindly. - Do not use proxies to bypass source limits. - Stream processing; never retain all results in memory. - Use a 1024 MB production default and one asynchronous process without multiprocessing. Keep source-page and procedure buffering bounded even when a run has more memory available. ### 13. Partial failures - Already written tender records remain in the Dataset if a later request fails. - A run with some unrecoverable request/detail failures may succeed with `isPartial: true`. - A procedure whose required detail response remains unavailable is skipped, warning recorded, and the run becomes partial. - Incremental cursor does not advance on a partial or incomplete scan. - A complete failure before any usable result must fail the Actor run. - A mass source-contract mismatch must fail rather than emit potentially corrupt records. - Reaching `maxItems` in snapshot mode sets `limitReached: true` but is not itself a partial failure. - Reaching `maxItems` before completing an incremental scan sets `limitReached: true`, `isPartial: true`, and does not advance the cursor. ### 14. Run summary (`OUTPUT`) Write a non-billable summary object to the run's default Key-Value Store under key `OUTPUT`. Never push the summary to the Dataset. Expected shape: ```json { "status": "SUCCEEDED", "mode": "snapshot", "isPartial": false, "limitReached": false, "itemsFound": 0, "itemsScanned": 0, "itemsWritten": 0, "newItems": 0, "updatedItems": 0, "duplicatesSkipped": 0, "filteredOut": 0, "requestsMade": 0, "requestsFailed": 0, "warningsTotal": 0, "warnings": [], "startedAt": "2026-07-31T12:00:00Z", "finishedAt": "2026-07-31T12:00:01Z", "datasetId": "...", "datasetUrl": "...", "jsonUrl": "...", "csvUrl": "...", "stateKey": null, "cursorBefore": null, "cursorAfter": null, "acquisitionStartedAt": "2026-07-30T22:00:00Z", "acquisitionEndedAt": "2026-07-31T22:00:00Z", "sourceApiVersion": null, "effectiveInput": {} } ``` Rules: - Empty result is a successful run with an empty Dataset. - `warnings` contains at most 100 representative entries. - Aggregate repeated warnings by stable warning code. - Preserve total warning counts separately. - On a controlled fatal failure, attempt to write a `FAILED` summary before re-raising, without masking the original exception. Every newly emitted summary also contains structured `sources` and `totals` blocks. Per-source counters expose fetched and normalized notices, emitted and filtered records, unsupported and duplicate records/notices, change events, candidate links, requests, retries, warnings, partial/limit status, and the cursor decision. `cursorNotAdvancedReason` is a stable enum such as `not_incremental`, `bounded_state_disabled`, `limit_reached`, `partial_run`, or `source_failed`. The legacy BZP root counters and CSV contract remain present. ### 15. JSON and CSV delivery Use the default Apify Dataset as the source of results. Do not create separate `results.json` or `results.csv` files. Define Apify schemas and output links so users can access: - JSON Dataset items; - CSV Dataset items with fixed field order and without `rawData`. CSV rules: - UTF-8; - comma delimiter; - fixed column order; - scalar null → empty cell; - empty array → `[]`; - booleans → `true`/`false`; - dates/timestamps use the stable formats above; - monetary values contain no currency symbols; - nested `cpvCodes`, `lots`, `documents`, `awards`, `notices`, and `changedFields` serialize as compact JSON strings; - new Schema v1 columns may be added only at the end. Fixed CSV column order: 1. `schemaVersion` 2. `tenderId` 3. `sourceTenderId` 4. `source` 5. `sourceLanguage` 6. `sourceUrl` 7. `title` 8. `description` 9. `publicationDate` 10. `latestNoticeDate` 11. `sourceUpdatedAt` 12. `submissionDeadline` 13. `status` 14. `procurementType` 15. `procedureType` 16. `procedureTypeOriginal` 17. `primaryCpvCode` 18. `cpvCodes` 19. `estimatedValue` 20. `valueMin` 21. `valueMax` 22. `awardedValue` 23. `currency` 24. `buyerName` 25. `buyerNationalId` 26. `buyerCity` 27. `buyerRegion` 28. `buyerPostalCode` 29. `buyerCountry` 30. `buyerWebsite` 31. `buyerEmail` 32. `buyerPhone` 33. `contactPersonName` 34. `lots` 35. `documents` 36. `awards` 37. `notices` 38. `changeType` 39. `changedFields` 40. `firstSeenAt` 41. `lastSeenAt` 42. `fetchedAt` Use an Apify Dataset view and/or explicit `fields` export parameter to guarantee order and exclude `rawData`. Add an integration test for actual exported CSV behavior. ### 16. Stable schema versioning - Every tender record contains `schemaVersion: "1.0"`. - Within `1.x`, only optional fields may be appended. - Existing fields may not be renamed, removed, or change type. - Breaking changes require `2.0`. - JSON property order is not guaranteed. - CSV column order is guaranteed. - Enums may gain values; consumers must handle `unknown`. - `rawData` is explicitly outside stability guarantees and is excluded from CSV. ### 17. Apify integration Required files: - `.actor/actor.json` - `.actor/input_schema.json` - `.actor/dataset_schema.json` - `.actor/output_schema.json` - `Dockerfile` Requirements: - Python 3.12 runtime. - Default normal execution to 1024 MB, with a 256 MB minimum and 1024 MB maximum. This non-browser Actor still needs headroom for complete BZP HTML notice pages and Python parsing overhead. - Input schema must generate a usable Apify Console form and set `additionalProperties: false`. - Dataset schema must validate Tender Schema v1 and define a useful overview. - Output schema must expose Dataset JSON, stable CSV, and the `OUTPUT` record. - Use `Actor.push_data()` only for billable tender records. - Use a named KVS for persistent incremental state. - Use the run default KVS only for `OUTPUT` and run-local artifacts. The Actor has no internal scheduler. Users run it through: - Apify Console; - REST API; - Actor Tasks; - Apify Schedules. Recommended incremental cadence is hourly or daily depending on the user's needs and source constraints. ### 18. Monetization Positioning: pay per successfully delivered normalized tender. Billing rules: - charge only for tender items written to the run's default Dataset; - `new` and `updated` items cost the same; - skipped duplicates, filtered records, errors, warnings, and `OUTPUT` are not billable; - an item with many lots remains one billable item; - when `includeUpdates: false`, changed records are not written and are not billed. Use the custom `normalized-tender` event after successful default-Dataset delivery. Configure the initial Public Beta price as $0.0001 in Apify Console; the application does not hardcode that price. Disable the synthetic `apify-default-dataset-item` event (or price it at zero) to prevent double charging. Respect the run's maximum total charge and use deterministic idempotency for retry-safe charges. See `docs/monetization.md`. ### 19. Logging and privacy Use structured logs at `INFO`, `WARNING`, and `ERROR`. Log: - run stage; - request counts; - retries and status categories; - tender ID for processing failures; - pagination progress; - final metrics. Never log: - Apify tokens; - authorization headers; - complete source payloads or `rawData`; - buyer emails or phone numbers; - unredacted sensitive query parameters; - very large response bodies. Aggregate repeated warnings. No user-exposed debug mode in the MVP. ### 20. Technology stack Required: - Python 3.12; - `pyproject.toml`; - `uv` for dependency management and lock file; - Apify SDK for Python; - `httpx` for asynchronous HTTP; - Pydantic for input, source, state, and output models; - `tenacity` for retries; - `orjson` for deterministic and efficient JSON serialization; - `pytest`; - `pytest-asyncio`; - `respx`; - `pytest-cov`; - `ruff` for linting and formatting; - `mypy` in strict mode. Do not use: - Crawlee; - Playwright/Selenium/Puppeteer; - FastAPI; - an external database; - multiprocessing. Use `Decimal` internally for money and serialize stable JSON numbers at the output boundary with tests covering cent-level precision. ### 21. Expected repository structure ```text . ├── .actor/ │ ├── actor.json │ ├── input_schema.json │ ├── dataset_schema.json │ └── output_schema.json ├── .github/ │ └── workflows/ │ ├── ci.yml │ ├── deploy.yml │ └── live-contract.yml ├── docs/ │ ├── source-audit.md │ ├── architecture.md │ ├── schema.md │ ├── operations.md │ └── monetization.md ├── src/ │ └── poland_public_tenders/ │ ├── __init__.py │ ├── __main__.py │ ├── main.py │ ├── config.py │ ├── errors.py │ ├── logging.py │ ├── models/ │ │ ├── input.py │ │ ├── output.py │ │ ├── source.py │ │ └── state.py │ ├── source/ │ │ ├── client.py │ │ ├── contracts.py │ │ └── mappings.py │ ├── services/ │ │ ├── aggregation.py │ │ ├── change_detection.py │ │ ├── filtering.py │ │ ├── normalization.py │ │ ├── pipeline.py │ │ └── state_store.py │ └── utils/ │ ├── cpv.py │ ├── dates.py │ ├── money.py │ └── text.py ├── tests/ │ ├── fixtures/ │ │ └── bzp/ │ ├── integration/ │ ├── live/ │ └── unit/ ├── .env.example ├── .gitignore ├── AGENTS.md ├── CODEX.md ├── Dockerfile ├── LICENSE ├── README.md ├── pyproject.toml └── uv.lock ``` The exact module split may be refined, but source transport, source models, aggregation, normalization, filtering, state, and Apify orchestration must remain separated and testable. ### 22. Source audit deliverable Before implementing production parsing, create `docs/source-audit.md` using official sources and live requests. It must record: - official source and terms links; - current base URL and endpoint family; - HTTP methods; - required and optional parameters; - pagination semantics and maximum page size; - date filtering semantics and timezone; - sort order; - response content types; - list versus detail capabilities; - notice type codes; - stable procedure IDs; - notice linkage fields; - CPV representation; - buyer, lot, value, document, result, award, and contract fields; - null/missing-field behavior; - error responses; - observed throttling and `Retry-After` behavior; - API version markers; - discrepancies between old documents and current live behavior; - legal/technical conditions relevant to Store publication. Save representative unmodified response fixtures for each included notice type. Do not include secrets. Freeze fixtures for unit tests. Live API tests remain separate. If the live API contradicts this specification on a source fact, preserve the product contract where possible and document the adapter decision. Do not silently weaken stable output. ### 23. Testing requirements #### 23.1 Unit tests Cover at minimum: - every input validation rule; - unknown fields; - whitespace and duplicate normalization; - Polish Unicode/casefold behavior; - keyword `any` and `all`; - CPV exact and hierarchy-prefix matching; - buyer, geography, status, procurement, procedure, date, deadline, and value filters; - filter group AND/OR semantics; - date-only and timezone conversion; - status mapping; - notice-type mapping; - procedure identity and fallback keys; - aggregation precedence; - lot/document/award normalization; - canonical hash stability; - excluded technical fields; - `changedFields` generation; - state fingerprinting; - retention cleanup; - warning aggregation; - CSV nested JSON serialization and column order. #### 23.2 Fixture-based source tests Use frozen official BZP responses for: - procurement notice; - correction/change; - cancellation; - result/award; - contract or execution notice; - multiple notice versions; - multiple lots; - missing optional fields; - unknown new enum values; - malformed or contract-breaking payload. No live requests in ordinary unit tests. #### 23.3 Integration tests Cover: - complete snapshot run; - empty snapshot run; - `maxItems` stop; - two sequential incremental runs with no duplicate delivery; - updated procedure with `includeUpdates: true`; - updated procedure with `includeUpdates: false`; - filter fingerprint mismatch; - state reset; - same-state lock rejection; - different-state parallel allowance; - overlap-window behavior; - partial page/detail failure; - `429`, `Retry-After`, `5xx`, timeout, and retry exhaustion; - cursor not advancing on partial run; - `OUTPUT` content; - Apify Dataset JSON and CSV exports; - Dataset schema validation. #### 23.4 Live contract test A separately marked live test verifies the official API contract with a very small, respectful request volume. It must not run in the default PR test suite. #### 23.5 Quality gates - minimum line coverage: 80%; - `ruff check` passes; - `ruff format --check` passes; - `mypy --strict` passes; - all non-live tests pass; - no network access in unit tests; - cloud build and run on Apify succeeds; - JSON and CSV are manually inspected before `v1.0.0`. ### 24. Performance targets - Target: no more than 15 minutes per 1,000 fully normalized procedures under normal BZP availability. - Production default memory: 1024 MB; bounded scans should avoid growth with total BZP notice-body volume. - Process results incrementally and push after normalization. - `maxItems = 100000` is functionally supported but not guaranteed to finish in a short run. - Do not compromise source rate-limit compliance to meet the target. - Record benchmarks in `docs/operations.md` and use them to set monetization. ### 25. CI/CD #### CI On every push and pull request: 1. install Python 3.12 and `uv`; 2. sync locked dependencies; 3. run `ruff check`; 4. run `ruff format --check`; 5. run `mypy --strict`; 6. run non-live tests with coverage threshold. #### Live contract workflow Run manually and optionally on a conservative schedule. External API instability must not block ordinary pull requests. #### Deploy After successful checks on `main`, deploy to Apify using the current official Apify deployment method. Use an `APIFY_TOKEN` GitHub Secret. Never commit secrets or generated credentials. `main` must remain deployable. Use semantic versioning: - initial implementation: `v0.1.0`; - stable public MVP: `v1.0.0`. ### 26. Using and deploying the Actor #### 26.1 First deployment, step by step Apify runs this project as a small cloud robot. You give it `INPUT`; it reads the selected official BZP and/or TED source, normalizes source-backed procedures, applies your filters, and places finished records in a Dataset. Its `OUTPUT` record is a report card for the run. To deploy from the repository root: ```bash npm install --global apify-cli apify login apify push ``` `apify login` authenticates your computer. `apify push` uploads the project and starts an Apify cloud build. Never commit `.env` or put a real token in a command example. Full deployment, CI, state, and troubleshooting instructions are in [`docs/operations.md`](docs/operations.md). For automatic deployment, add a GitHub Actions repository secret named `APIFY_TOKEN`. After CI succeeds on `master`, the deploy workflow uploads that tested revision with the official Apify CLI. The workflow pins the CLI version so an upstream output-format change cannot silently alter the release gate. Apify is not automatically free without limits. As verified on 2026-07-31, the Free plan is $0 and includes $5 of monthly platform usage; builds and runs use that allowance. Free-plan usage stops when the allowance is exhausted instead of creating a Free-plan overage invoice. Plans can change, so check . Deploying a private Actor does not earn money; Store publication and monetization are separate. See [`docs/monetization.md`](docs/monetization.md). #### 26.2 Apify Console inputs In **Apify Console → Actors → Poland Public Tenders Scraper → Input**, use a small snapshot first. Replace the example dates with the BZP publication days you need: ```json { "mode": "snapshot", "publishedFrom": "2026-07-30", "publishedTo": "2026-07-31", "maxItems": 10 } ``` For repeat monitoring, use incremental mode and keep the same `stateKey`: ```json { "mode": "incremental", "stateKey": "daily-monitor", "initialLookbackDays": 2, "overlapHours": 24, "includeUpdates": true, "maxItems": 100 } ``` Different feeds need different keys, for example `daily-it` and `daily-construction`. To deliberately forget only one feed's memory, run it once with its existing key and `"resetState": true`, then turn reset off again. #### 26.3 REST API Use an authorization header so the token is not placed in the URL. Start an asynchronous run (replace the Actor ID and token placeholders): ```bash curl --request POST \ --header "Authorization: Bearer YOUR_APIFY_TOKEN" \ --header "Content-Type: application/json" \ --data '{"mode":"snapshot","publishedFrom":"2026-07-30","publishedTo":"2026-07-31","maxItems":10}' \ "https://api.apify.com/v2/acts/YOUR_USERNAME~poland-public-tenders-scraper/runs" ``` For a small run that waits and directly returns default-Dataset items, use the synchronous endpoint. It can wait at most 300 seconds, so use asynchronous runs for larger work: ```bash curl --request POST \ --header "Authorization: Bearer YOUR_APIFY_TOKEN" \ --header "Content-Type: application/json" \ --data '{"mode":"snapshot","publishedFrom":"2026-07-30","publishedTo":"2026-07-31","maxItems":10}' \ "https://api.apify.com/v2/acts/YOUR_USERNAME~poland-public-tenders-scraper/run-sync-get-dataset-items" ``` #### 26.4 Python and JavaScript clients Python with the official `apify-client` package: ```python import os from apify_client import ApifyClient client = ApifyClient(os.environ["APIFY_TOKEN"]) run = client.actor("YOUR_USERNAME/poland-public-tenders-scraper").call( run_input={ "mode": "snapshot", "publishedFrom": "2026-07-30", "publishedTo": "2026-07-31", "maxItems": 10, } ) items = client.dataset(run["defaultDatasetId"]).list_items().items ``` JavaScript with the official `apify-client` package: ```javascript import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('YOUR_USERNAME/poland-public-tenders-scraper').call({ mode: 'snapshot', publishedFrom: '2026-07-30', publishedTo: '2026-07-31', maxItems: 10, }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); ``` #### 26.5 Results, Tasks, and Schedules After a run, the Apify **Output** page exposes three links: - **Tender records (JSON)** is the canonical default Dataset; - **Tender records (stable CSV)** has the fixed 42-column contract and excludes `rawData`; - **Run summary** is the non-billable `OUTPUT` record. API consumers can retrieve JSON from `GET /v2/datasets/DATASET_ID/items?format=json`. Retrieve `OUTPUT` from `GET /v2/key-value-stores/KEY_VALUE_STORE_ID/records/OUTPUT`, read its `csvUrl`, and download that URL for stable CSV. The Dataset and Key-Value Store IDs are returned on the completed run object. To automate monitoring in Console, save a tested incremental input as an **Actor Task**, then create an Apify **Schedule** that starts that Task hourly or daily. Do not schedule overlapping runs with the same `stateKey`. Read `OUTPUT` before trusting a run. `isPartial: false` means the scan completed. `isPartial: true` means some coverage was incomplete, the cursor did not advance, and `warnings` explains why. `itemsWritten` is the count of billable tender records; warnings, summaries, filtered records, and the CSV projection are not billed as tender items. Public Beta uses the custom `normalized-tender` event. Public Beta pricing also includes the Actor-start event and separate Apify platform usage described in the public pricing section; production pricing configuration remains an account-owner Console setting. Before releasing version `1.0.0`, run the manually triggered two-run cloud verification described in [`docs/INCREMENTAL_CLOUD_TEST.md`](docs/INCREMENTAL_CLOUD_TEST.md). It uses an isolated state partition and does not publish the Actor to the public Store. ### 27. Legal and attribution Every record identifies its source and preserves an official source URL when available. Tender Schema v1 uses `source: "BZP"`; the unified contract uses `source.system` with `BZP` or `TED`. Documentation and Store copy must state: - the Actor is independent and is not affiliated with or endorsed by the Polish Public Procurement Office or e-Zamówienia; - source records originate from the selected official BZP/e-Zamówienia and/or TED public interface; - users must verify legally critical deadlines, conditions, and documents in the original notice; - normalization does not constitute legal advice and cannot guarantee legal completeness; - code licensing does not override source-data terms; - source terms and data-reuse conditions must be reviewed again before Store publication. Do not use Polish government logos or visual branding that implies official status. ### 28. License Project code is licensed under the MIT License. This license applies to the code, not to official BZP data or third-party content. ### 29. Definition of done The MVP is complete only when all conditions below are met: - official BZP contract is audited and documented; - snapshot and incremental modes work; - one item equals one aggregated procedure; - all agreed filters work and are documented; - stable Tender Schema v1 is validated; - JSON and fixed-order CSV exports work; - incremental state, fingerprinting, retention, and lock behavior work; - new/updated change detection works; - partial failures and cursor safety work; - no external database or browser automation is present; - Apify input, dataset, and output schemas are complete; - `OUTPUT` is written correctly; - pay-per-item behavior maps only to delivered tender records; - test coverage is at least 80%; - Ruff, mypy strict, and tests pass; - GitHub Actions are green; - Apify cloud build succeeds; - a real small cloud run is manually verified in JSON and CSV; - secrets are absent from repository history and logs; - English documentation and legal disclaimers are complete; - all deviations and unresolved source risks are documented. ### 30. Official references to verify during implementation - e-Zamówienia integration materials: `https://ezamowienia.gov.pl/pl/integracja/` - Apify Actor input schema: `https://docs.apify.com/actors/development/actor-definition/input-schema` - Apify Dataset schema: `https://docs.apify.com/storage/dataset-schema` - Apify Actor output schema: `https://docs.apify.com/actors/development/actor-definition/output-schema` - Apify Dataset export: `https://docs.apify.com/storage/dataset` - Apify pay-per-event: `https://docs.apify.com/actors/publishing/monetize/pay-per-event` - Apify SDK for Python: `https://docs.apify.com/sdk/python/` - Codex `AGENTS.md`: `https://developers.openai.com/codex/agent-configuration/agents-md` ### 31. Unified BZP and TED extension The original BZP product contract remains the compatibility default. When `sources` is omitted, the Actor behaves exactly as before and emits Tender Schema v1 plus its fixed-order CSV projection. The source choices are: - `["BZP"]`: legacy BZP-only Tender Schema v1 (default); - `["TED"]`: official TED Search API v3 and `pl-procurement-v1` JSON; - `["BZP", "TED"]`: both official sources in the same `pl-procurement-v1` consumer contract. TED published-notice search is anonymous. Do not configure a `TED_API_KEY`. The adapter uses `POST https://api.ted.europa.eu/v3/notices/search`, validates its generated expert query by default, and uses `ITERATION` pagination. The opaque iteration token exists only for one point-in-time scan; durable state is a semantic cursor containing publication date, publication number, notice ID, and notice version. Example bounded TED snapshot: ```json { "mode": "snapshot", "sources": ["TED"], "publishedFrom": "2025-05-02", "publishedTo": "2025-05-02", "maxItems": 100, "ted": { "polandScope": "either", "languages": ["POL", "ENG"], "pageSize": 100, "validateQuery": true } } ``` `ted.polandScope` has three exact modes: - `buyer_country`: `buyer-country = POL`; - `place_of_performance`: `place-of-performance IN (POL)`; - `either` (default): the OR of both clauses. This includes cross-border procedures whose buyer is not Polish but whose performance includes Poland. `sourceFailureMode` defaults to `fail_fast`. Set it to `allow_partial_sources` only when a consumer explicitly accepts a structured partial result. Each source uses independent state; TED state keys are derived from `stateKey` under a separate `ted_` namespace. A bounded TED incremental replay does not open or mutate TED state unless `advanceBoundedTedState` is explicitly `true`, which is intended for isolated cloud verification. Unified JSON records preserve official source notice/procedure/version and OJS publication identifiers, all lifecycle notice references seen in the scan, eForms SDK provenance, source/XML URLs, normalized organizations, lots, CPV, places, values, deadlines, awards, contracts, documents, conservative links, and deterministic events. Legacy TED XML is detected but not silently parsed with eForms assumptions. See `docs/TED_INTEGRATION.md` and `docs/TED_CLOUD_TEST.md` for field limitations and operations. Production-like verification is manual and non-publishing. The `Production cloud verification` GitHub Actions workflow exercises BZP-only, TED-only, combined, bounded TED, and two consecutive TED incremental runs against an already deployed Actor revision. Docker packaging is verified by CI. See `docs/PRODUCTION_VERIFICATION.md` for prerequisites, exact commands, artifacts, and the remaining release gates. PostgreSQL and Redis are explicitly deferred; Apify Dataset and Key-Value Store remain the production adapters. SaaS consumers should branch on `schemaVersion` during migration: ```python for item in dataset_items: if item["schemaVersion"] == "pl-procurement-v1": source = item["source"]["system"] stable_key = item["procedureId"] else: # legacy default BZP-only mode source = item["source"] stable_key = item["tenderId"] ``` # Actor input Schema ## `mode` (type: `string`): Snapshot reads a bounded YYYY-MM-DD publication range without a durable cursor. Incremental delivers new and changed procedures using persistent state. ## `sources` (type: `array`): Choose BZP, TED, or both official sources. BZP alone preserves Tender Schema v1; selecting TED enables the unified pl-procurement-v1 JSON contract. ## `sourceFailureMode` (type: `string`): Fail fast by default, or explicitly allow successful sources to be delivered with a partial failed summary. ## `ted` (type: `object`): Anonymous official TED Search API v3 settings. The default either scope includes Polish buyers and Polish places of performance. ## `publishedFrom` (type: `string`): Required in snapshot mode. Format: YYYY-MM-DD. ## `publishedTo` (type: `string`): Required in snapshot mode and includes the full selected day. Format: YYYY-MM-DD. ## `incrementalFrom` (type: `string`): Optional inclusive acquisition instant for a controlled incremental replay. Must be paired with incrementalTo and include a timezone. ## `incrementalTo` (type: `string`): Optional exclusive acquisition instant for a controlled incremental replay. Must be paired with incrementalFrom and include a timezone. ## `updatedFrom` (type: `string`): Keep procedures updated on or after this date. Format: YYYY-MM-DD. ## `updatedTo` (type: `string`): Keep procedures updated on or before this inclusive date. Format: YYYY-MM-DD. ## `deadlineFrom` (type: `string`): Keep procedures whose current deadline is on or after this date. Format: YYYY-MM-DD. ## `deadlineTo` (type: `string`): Keep procedures whose current deadline is on or before this inclusive date. Format: YYYY-MM-DD. ## `keywords` (type: `array`): Phrases searched in the title and description. ## `keywordMatch` (type: `string`): Choose whether any phrase or every phrase must match. ## `cpvCodes` (type: `array`): Match primary and additional normalized CPV codes. ## `cpvMatch` (type: `string`): Match an exact normalized CPV code or a CPV hierarchy prefix. ## `buyerNames` (type: `array`): Match official buyer names. ## `buyerNameMatch` (type: `string`): Choose substring or exact matching for buyer names. ## `buyerNationalIds` (type: `array`): Exact official identifiers such as NIP or REGON when published. ## `regions` (type: `array`): BZP-only buyer-address filter. It is rejected when TED is selected because the verified TED mapping does not expose an equivalent field. ## `cities` (type: `array`): BZP-only buyer-address filter. It is rejected when TED is selected because the verified TED mapping does not expose an equivalent field. ## `statuses` (type: `array`): Keep procedures with any selected stable status. ## `procurementTypes` (type: `array`): Keep procedures with any selected stable procurement type. ## `procedureTypes` (type: `array`): Stable normalized procedure codes implemented by the Actor. ## `minValue` (type: `number`): Minimum requested value. The documented interval-overlap rule is applied. ## `maxValue` (type: `number`): Maximum requested value. Must not be lower than the minimum value. ## `maxItems` (type: `integer`): Counts unique procedures delivered to the default Dataset. The safe default is 100; values above 1000 can materially increase run time and cost. ## `includeUpdates` (type: `boolean`): In incremental mode, also deliver previously seen procedures whose business content changed. ## `includeRawData` (type: `boolean`): Adds the unstable rawData field to JSON records. CSV always excludes it. ## `initialLookbackDays` (type: `integer`): Used only for a new incremental state partition. ## `overlapHours` (type: `integer`): Re-read this many hours before the saved cursor to catch late source changes. ## `stateKey` (type: `string`): Separates independent incremental cursors and fingerprints. ## `resetState` (type: `boolean`): Allowed only in incremental mode. ## `advanceBoundedTedState` (type: `boolean`): DANGER: enabling this lets a bounded TED replay advance durable incremental state. Leave it off for tests and historical samples; use only with an intentional stateKey and verified bounds. ## Actor input object example ```json { "mode": "snapshot", "sources": [ "BZP", "TED" ], "sourceFailureMode": "fail_fast", "ted": { "polandScope": "either", "languages": [ "POL", "ENG" ], "pageSize": 100, "validateQuery": true }, "publishedFrom": "2026-07-01", "publishedTo": "2026-07-31", "incrementalFrom": "2026-01-01T19:22:37Z", "incrementalTo": "2026-01-01T19:22:38Z", "updatedFrom": "2026-07-01", "updatedTo": "2026-07-31", "deadlineFrom": "2026-08-01", "deadlineTo": "2026-08-31", "keywords": [ "roboty budowlane" ], "keywordMatch": "all", "cpvCodes": [ "45000000-7" ], "cpvMatch": "prefix", "buyerNames": [ "Urząd Miasta" ], "buyerNameMatch": "contains", "buyerNationalIds": [ "0000000000" ], "regions": [ "mazowieckie" ], "cities": [ "Warszawa" ], "statuses": [ "open" ], "procurementTypes": [ "works" ], "procedureTypes": [ "basic_no_negotiation" ], "minValue": 100000, "maxValue": 500000, "maxItems": 100, "includeUpdates": true, "includeRawData": false, "initialLookbackDays": 7, "overlapHours": 24, "stateKey": "customer-a", "resetState": false, "advanceBoundedTedState": false } ``` # Actor output Schema ## `tenders` (type: `string`): Tender Schema v1 in default BZP-only runs; pl-procurement-v1 when TED is selected. ## `tendersCsv` (type: `string`): Legacy BZP-only UTF-8 CSV in the documented fixed column order. Unified runs use the JSON Dataset. ## `runSummary` (type: `string`): The run-summary-v1 OUTPUT record with status, counters, billing, warnings, links, and effective input. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "mode": "snapshot", "sources": [ "BZP" ], "publishedFrom": "2026-04-05", "publishedTo": "2026-04-05", "maxItems": 1, "includeRawData": false }; // Run the Actor and wait for it to finish const run = await client.actor("toughtcrime/poland-public-tenders-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "mode": "snapshot", "sources": ["BZP"], "publishedFrom": "2026-04-05", "publishedTo": "2026-04-05", "maxItems": 1, "includeRawData": False, } # Run the Actor and wait for it to finish run = client.actor("toughtcrime/poland-public-tenders-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "mode": "snapshot", "sources": [ "BZP" ], "publishedFrom": "2026-04-05", "publishedTo": "2026-04-05", "maxItems": 1, "includeRawData": false }' | apify call toughtcrime/poland-public-tenders-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,toughtcrime/poland-public-tenders-scraper" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/xTez3RjzydSfp67bX/builds/QwjlT1ARLeCGjSRwF/openapi.json