Tender & Procurement Feed Normalizer

Audit tender scraper output before it reaches your bid tracker: normalize dates and values, reject ambiguous rows, flag duplicates, and retain original fields. Optional tender-stage checks keep supplied award/planning rows out; a SHA-256 receipt seals supplied input, settings and delivered output. This is a downstream procurement-data companion, not another scraper, OCDS validator, or verification that an opportunity is genuine, current or suitable.

First run: two fictional rows, one clean result

1. Open Tender Feed Normalizer in Apify Console. Switch Input to JSON and replace the whole input with this example:

{
  "records": [
    {"title": "Example tender A (fictional)", "deadline": "2027-12-31", "buyer": "Example buyer", "reference": "EX-A", "value": "USD 12000"},
    {"title": "Example tender B (fictional)", "deadline": "20270120", "buyer": "Other buyer", "reference": "EX-B", "value": 5000, "currency": "EUR"}
  ],
  "currency": "USD",
  "fieldMap": {}
}

Download the same input as tender_normalizer_records.json. Both notices are fictional, not procurement opportunities.

2. Use 128 MB memory, the validated default for the bounded examples. Set Maximum charge per run to $0.01 in run options, not inside input JSON. Check current Store pricing before starting. At the published prices checked 2026-09-23, this example costs $0.00105: $0.00005 start + one $0.001 clean row. The rejected EUR row has no row fee. The $0.01 limit is a ceiling, not the expected charge; see cost details. Large individual fields can require more memory even below the 10,000-row cap.
3. Start the Actor. After a successful run, open Clean tenders (tenders) in Output for the default dataset. Its complete JSON result is:

[
  {
    "row": 0,
    "title": "Example tender A (fictional)",
    "deadline": "2027-12-31T00:00:00+00:00",
    "deadlineRaw": "2027-12-31",
    "value": 12000.0,
    "valueRaw": "USD 12000",
    "currency": "USD",
    "buyer": "Example buyer",
    "reference": "EX-A",
    "expired": false,
    "errors": [],
    "errorCodes": [],
    "clean": true,
    "sourceFields": {"title": "Example tender A (fictional)", "deadline": "2027-12-31", "buyer": "Example buyer", "reference": "EX-A", "value": "USD 12000"},
    "mappedFrom": {"title": "title", "deadline": "deadline", "value": "value", "buyer": "buyer", "reference": "reference", "currency": null}
  }
]

This is the expected output before the deadline, not a cloud-run receipt. expired becomes true after that instant; expired rows remain clean. mappedFrom.currency is null here because USD came from the amount, not a currency column. JSON export includes provenance even when the table view hides it.

4. Open Validation report (report) in Output, or key-value storage → REPORT. Check these fields (an excerpt, not the complete report):

{
  "normalizerVersion": "0.3.0",
  "rowsIn": 2,
  "rowsClean": 1,
  "rowsPassed": 1,
  "rowsWithIssues": 1,
  "duplicates": [],
  "conflicts": [],
  "errorCounts": {"currency_mismatch_expected": 1}
}

issues[0].row is 1 (indices start at zero), issues[0].currency is "EUR", and issues[0].errorCodes is ["currency_mismatch_expected"]. The original EUR record remains in issues[0].sourceFields; it was not converted or discarded. A successful run does not mean every input row passed.

To process that fictional EUR notice separately, paste tender_normalizer_retry_eur.json. It submits only row B with expected currency EUR: one clean row, no issues, another $0.00105 at the same settings. Do not change a real notice's currency merely to remove an error, or resend already delivered rows unintentionally.

Choose one input source

Input	What to supply	Limit / common mistake
records	Array of record objects	Maximum 10,000 rows. Nonempty records override all other sources.
sourceUrl	Public HTTPS UTF-8 CSV or JSON: array, single record, or {"records": [...]}	Maximum 5 MiB and 10,000 rows. Port 443 only; no redirects, compressed feeds, HTML/XML, credentials, fragments, or private hosts.
datasetId	An accessible Apify dataset ID, not its URL or the upstream Actor ID	10–32 alphanumeric characters. Reads only the first 10,000 rows; split larger datasets upstream.
Integration payload	Automatic payload.resource.defaultDatasetId from Apify Actor-to-Actor Integrations	Used only when no other source is selected. It need not appear in the input form.

Clear the fictional records prefill before using a URL or dataset. Priority is nonempty records → sourceUrl → explicit datasetId → implicit integration payload. Sources are not merged. An empty feed fails the run; it does not produce a successful zero-row report. For API/MCP or a manual run, provide a real datasetId; template braces are not expanded by the Actor.

The code reports normalizerVersion: "0.3.0"; this is not the Apify build number. Check the run's build in Console instead of using a README version as deployment evidence.

Dates

Use ISO dates to avoid day/month ambiguity. Supported: ISO YYYY-MM-DD with optional time, fractional seconds and offset (2027-01-15, 2027-01-15T12:30:45.123Z, ...+02:00); %d/%m/%Y, %m/%d/%Y, %d.%m.%Y, 15 January 2027. Ambiguous slash dates (01/02/2027) are rejected. Dates without times mean midnight UTC; expired compares against run time.

All-digit values follow an explicit grammar: exactly 8 digits is a compact YYYYMMDD date (20270115); up to 10 digits is unix seconds; exactly 13 digits is unix milliseconds. 11–12 digits is rejected as ambiguous rather than guessed.

Money

Amounts accept unsigned decimals with up to two decimal places, comma thousands grouping, or consistently grouped European notation. Negative values, ranges, malformed grouping, and ambiguous dot grouping are rejected. Missing values are allowed; missing titles or deadlines are reported.

Currencies come from recognized codes in the value (USD 100, 100USD), symbols ($, €, R$), or an explicit currency column. Supported codes: USD EUR GBP JPY CAD AUD CHF SEK NOK DKK PLN BRL MXN INR CNY ZAR SGD HKD NZD KRW RUB ILS VND PHP UAH; RMB is also accepted and canonicalized to CNY. Other codes are not supported. Codes match letter boundaries, so USDTOKENS is not USD; r$ is BRL. Symbols $ and ¥ assume USD/JPY — use the correct explicit code instead for other currencies. Conflicting declarations are rejected (currency_conflict); no conversion occurs.

The optional top-level currency is a check, not a default. It flags a known different currency (currency_mismatch_expected), but does not fill a missing currency or require a value. Omit it for an intentionally mixed-currency feed. Missing amounts/currencies remain null, not zero or USD.

Duplicates and conflicts

Two rows are the same notice only when normalized title, deadline, buyer and reference all match. A different buyer or reference always survives — two authorities can run identically titled tenders on the same day. An exact repeat is reported duplicate; the same identity with a different stated value or currency is reported conflict and never silently merged. A missing amount or currency is a wildcard: an enriched copy is a duplicate, not a conflict. Only earlier clean rows can suppress later copies, so an invalid row cannot hide its corrected copy.

Deduplication is within one run only, not across daily runs. Input order decides which valid copy survives; an enriched later copy can be a duplicate of an earlier incomplete copy. Keep buyer/reference columns and review conflicts before importing results. row is the original zero-based position, not a persistent notice ID.

Read and resolve REPORT

Import the default dataset into your downstream system; route REPORT.issues to a review queue. rowsClean is the number delivered; rowsPassed includes valid rows withheld by the spending limit. rowsWithIssues includes all rejected or withheld rows. errorCounts counts issue occurrences, not rows: one row can carry several codes. duplicates and conflicts contain original row indices.

issues[*].errorCodes	Action before retrying
bad_record, missing_title, bad_title	Supply an object with a nonempty string title; check column mapping.
missing_deadline, bad_deadline	Check mappedFrom.deadline. Resolve the date from the source and use ISO format; do not guess day/month order.
bad_value	Supply an unsigned, unambiguous amount. Use null only when the source genuinely has no amount.
bad_currency, currency_conflict	Correct unsupported codes or conflicting amount/column declarations from source evidence. No currency conversion is performed.
currency_mismatch_expected	Separate currencies into appropriate batches, or omit the expected-currency check if a mixed feed is intended.
duplicate, conflict	Review the referenced earlier row and the source notice. Choose the authoritative version; do not silently merge amounts.
withheld_billing	The row passed validation but exceeded the delivery budget. After reconciling the run, submit only withheld source records with a suitable cap.

Retry corrected issues[*].sourceFields objects, not the normalized issue envelopes, as a new records array. Keep the same applicable fieldMap. Each new run has fresh row indices and its own start/clean-row charges. Never automatically replay a failed paid run: dataset writes and billing are not one transaction; inspect the dataset and run charges first.

Withheld rows retain raw source, index, and reason only — not a free normalized result. Other rejected rows retain their normalized fields, original source, and reasons. Export JSON when reviewing provenance. sourceFields preserves spreadsheet formulas and any sensitive upstream fields verbatim: treat exports as untrusted input and restrict dataset/report access.

If a dataset read returns 401/403, grant appropriate storage-read permissions, or share the dataset by link only if its contents may be shared. Never paste a token into input. If a URL fetch fails, supply a direct public CSV/JSON URL meeting the limits above, or authorized inline records. Input/fetch failures may stop before REPORT exists; inspect the run log.

Schema handshake: aliases and fieldMap

Each row needs a title and parseable deadline. The Actor recognizes these top-level column names, in priority order. Matching ignores case and separators: Notice_Title and notice title match. The first present alias wins, even if its value is empty.

Standard field	Built-in aliases (first match wins)
title	title, name, subject, notice title, notice_title, description
deadline	deadline, closes, closing, close date, submission deadline
value	value, amount, budget, estimated value, estimated_value, contract value
buyer	buyer, authority, organisation, organization, contracting authority, issuer
reference	id, notice id, notice_id, reference, ref, tender id, ocid
currency	currency, currency code, currency_code, iso currency, iso4217

fieldMap direction is your column → standard field, for example {"responseDeadline": "deadline"}. Targets must use the exact standard names above. Matching explicit mappings override aliases; the first matching mapping for a target wins. Unmatched sources and unknown targets are ignored, not rejected. Mapping one column does not remove its alias role for another field. Nested paths such as tender.period.endDate are not traversed; flatten them upstream first.

Before connecting a full feed, paste one or two representative rows with your map. Inspect mappedFrom, normalized values, and REPORT. A clean row alone does not prove the right buyer/reference was selected. Preserve the original notice identifier for downstream joins.

Check what the date means. Version 0.3.0 removes implicit end date / end_date / endDate deadline matching: contract end is not a bid deadline. If your field genuinely means submission deadline, explicitly map it with fieldMap; this is an intentional migration, not automatic inference. Award-only feeds remain unsuitable. Field maps cannot certify business meaning.

Download reusable workflow inputs

The public workflow library contains JSON inputs only; it does not require access to this Actor's private source. Open a file, choose Raw, and copy its JSON into Input.

Download	Use
tender_normalizer_records.json	Exact fictional first-run input above; one intentional EUR rejection.
tender_normalizer_retry_eur.json	Retry only that fictional EUR notice, with the correct expected currency.
tender_normalizer_dataset.json	Manual/API/MCP input; replace the dataset-ID placeholder. Uses built-in aliases until you add a map.
tender_normalizer_samgov_sample.json	Fictional row using published SAM.gov scraper field names; inspect mapping on a small sample before a real feed.
tender_normalizer_samgov_integration.json	Actor-to-Actor input using the triggering run's dataset and the same SAM.gov field map.
tender_normalizer_sourceurl.json	URL/CSV mapping template. example.org is a placeholder, not a working tender feed.

Maintainers with a source checkout only

These commands require the Actor's source tree, which the public input library does not provide. Buyers can use the Console quickstart above without local code. From tools/apify-tender-normalizer in a source checkout:

python scripts/run_tender_demo.py    # existing CSV + fieldMap and direct-JSON golden fixtures
python scripts/verify_all.py         # unit suites plus that demo
python -m unittest discover -q       # unit suites only

examples/tender_feed/ contains fictional CSV/JSON fixtures and golden outputs. The offline demo calls the production normalization path with an in-process Apify stand-in. It does not fetch upstream data, run a cloud Actor, prove billing, or exercise every template listed above.

Repeat-use workflow: upstream dataset → clean table + review queue

The published output example for scrapesage/sam-gov-scraper documents title, responseDeadline, office, noticeId, and solicitationNumber. The following is a documentation-grounded mapping candidate, not a live-tested integration or endorsement. Source checked 2026-09-22; inspect your actual rows before using it.

{
  "datasetId": "{{resource.defaultDatasetId}}",
  "fieldMap": {
    "responseDeadline": "deadline",
    "office": "buyer",
    "noticeId": "reference"
  }
}

title already matches an alias. The map uses the supplied contracting office as buyer and notice ID as reference; solicitationNumber stays in sourceFields. There is no amount/currency in that published example, so both remain null. No USD default is invented. The fictional sample file has deadline 2027-12-31T10:00:00-07:00, expected normalized deadline 2027-12-31T17:00:00+00:00, and reference FICTIONAL-SAM-001.

1. Inspect a completed upstream run's output. Confirm the chosen fields are present and dates are parseable. That scraper documents exact deadlines only with fetchFullDetails enabled; notices without deadlines are rejected, not invented.
2. In the upstream Actor/task's Integrations tab, add this normalizer and choose Run succeeded. Paste the integration JSON above. Remove any prefilled records and sourceUrl.
3. Set a finite maximum charge for the normalizer in integration run options. Cap and price the upstream scraper separately. Integration test runs are real runs, not free previews.
4. On each successful upstream run, Apify replaces {{resource.defaultDatasetId}} with that run's dataset ID. Alternatively, omit datasetId and let the implicit payload.resource.defaultDatasetId supply it. Do not keep a fixed historical ID in a recurring workflow.
5. After each normalizer run, import its clean dataset, archive its REPORT, and route issues for review. Filter expired downstream if you need open notices only. Recheck the schema when the producer changes it.

The official integration guide documents interpolation and the automatic payload. Outside that integration UI, replace the placeholder with the real dataset ID yourself. Dataset access starts without credentials, then retries 401/403 with the run token if available; limited-permission runs cannot read arbitrary private datasets.

For a daily feed, schedule the upstream collector, not a normalizer pinned to yesterday's dataset. Cross-run deduplication/upserts belong in your destination, using source namespace + notice ID and your own amendment policy. The normalizer does not track changes across runs; rerunning yesterday's clean records can charge for them again. An empty upstream dataset fails normalization; do not loop retries on an empty feed.

Cost math (published pricing)

Public pricing checked 2026-09-23: $0.001 per delivered clean row (normalized-tender), plus the platform start event. At the default 128 MB, one start event costs $0.00005:

Delivered clean rows in one run	Row fees	Start fee	Actor event total
0 (all rows rejected)	$0	$0.00005	$0.00005
1 (quickstart)	$0.001	$0.00005	$0.00105
100	$0.100	$0.00005	$0.10005
1,000	$1.000	$0.00005	$1.00005

Rejected, duplicate, conflicting, and billing-withheld rows have no clean-row fee; the start fee still applies. The start event is memory-based (one event per GB, minimum one), so increased memory can increase that fee. The current minimum allowed run cap is $0.00105, even when every row will be rejected.

Set maxTotalChargeUsd through run options, not input JSON. Rows beyond the available budget are withheld and listed in REPORT as withheld_billing; rowsPassed may then exceed rowsClean. A small cap can deliver only part of an otherwise valid feed. This cap does not cover upstream scraper runs or your entire workflow.

These totals cover this Actor's events, not external tools or post-run storage/API use. The Store currently includes in-run platform usage in the event price; Apify documents separate post-run storage operations. Prices and account-specific discounts can change. Check the Store pricing before running; the public Actor metadata exposes event prices, default memory, and the minimum charge cap. No prices are set by these templates.

Operator billing hook (disabled by default)

Configure a positive-price pay-per-event event named normalized-tender, remove apify-default-dataset-item from that Actor's pricing, and require a positive finite run spending limit before setting NORMALIZER_ENABLE_BILLING=1. An optional platform start event is separate; its already-charged cost is included in the SDK's remaining budget. Each delivered clean row uses the SDK's charged push_data; rejected rows do not generate this custom event. Withheld rows retain only row index, raw source data, and reason in REPORT, not their normalized output. Missing/invalid pricing, non-PPE mode, duplicate synthetic dataset charging, invalid/unbounded spending limits, and mismatched charge acknowledgements fail the run. No prices or remote settings are configured by this source tree. See https://docs.apify.com/sdk/python/reference/class/Actor#push_data.

Billing writes are not a transaction: the SDK persists dataset rows before completing custom-event charging. A charge transport failure makes the run fail; do not automatically retry failed paid runs without reconciliation. Cloud owner-run checks verify mechanics, not external customer payments.

No scraping, revenue, accuracy, or publication approval guarantees. Actor pricing and platform charges depend on the published Console configuration.

Tender-stage gate and reproducibility receipt

For a flat feed with a documented stage column, add "stageField": "stage". Only supplied tender labels pass this gate (case and outer whitespace ignored). Missing values get missing_stage; other labels get non_tender_stage. Rejected rows remain in REPORT with raw fields and no clean-row fee. The field name is exact and case-sensitive. Omit the option when the upstream contract has no stage field. The gate does not turn an untrusted label into verified evidence.

For Ciel Labs' documented UK tender-stage columns, use the candidate recipe below. Its producer already harmonizes and deduplicates: use this extra step only if you need common rejection/checksum receipts across sources. This is a published-schema example, not an end-to-end certified producer integration.

{
  "datasetId": "{{resource.defaultDatasetId}}",
  "stageField": "stage",
  "fieldMap": {
    "title": "title", "deadline": "deadline", "buyerName": "buyer",
    "valueAmount": "value", "valueCurrency": "currency", "ocid": "reference"
  }
}

Clear fictional records and sourceUrl before using a dataset. The template works in Apify Integrations, not as a literal dataset ID in API/MCP calls. A procurement-process ocid is not an immutable notice revision; no OCDS merge is performed.

Open Reproducibility receipt (AUDIT) in Output. It contains inputSha256, settingsSha256, deliveredSha256, effective settings and row counts. Hash UTF-8 JSON with insertion-ordered object keys, no ASCII escaping, compact separators and finite numbers. Before encoding each number, convert integral floats to integers (including negative zero to zero): Apify storage emits 12000.0 as 12000. Object and array order remain significant because colliding field names and mapping precedence can be order-sensitive. Input means the selected parsed row array, not original CSV/JSON bytes or unused sources. Settings include normalizer version, field map, expected currency and stage field. Nonfinite source numbers fail before dataset delivery. Use the public workflow library's verify_audit.py for the exact recipe; preserve the downloaded JSON's key order. Reformatting/reordering objects or a storage serialization change can invalidate a checksum without changing business meaning.

Hashes prove consistency only: not source authenticity, procurement-stage truth, completeness of the upstream dataset, or predictive accuracy. expired is evaluated at processing time, so delivered-output hashes can change across a deadline even when raw input is identical. Inline normalization makes no upstream requests; hosted input/output/billing still use Apify platform services.

Release notes — audit companion

• Strict common-ISO parsing and one normalized-key index per row; uncommon formats retain the existing parser.
• Minimal Python container, SDK 3.4.1, no direct browserforge or old Pydantic compatibility pin; source-only image copy.
• Explicit tender-stage rejection and an input/settings/output checksum receipt.
• Intentional semantic migration: contract-end aliases require explicit, meaning-checked mapping.
• Existing prices and LIMITED_PERMISSIONS unchanged. No claim of customers, sales, universal upstream compatibility or speed superiority over network scrapers.