steadfetch/ted-tender-monitor

Monitor new public procurement notices from two official sources in one Actor:

• TED (Tenders Electronic Daily), the official EU supplement.
• USAspending, the official U.S. federal spending API.

Saved searches, a source filter, one unified schema and delta-only results: each run returns only the notices and awards it has never delivered before.

What it does

• Calls two official, keyless JSON APIs:
  • TED Search API v3 (POST https://api.ted.europa.eu/v3/notices/search);
  • USAspending (POST https://api.usaspending.gov/api/v2/search/spending_by_award/).
• Runs 1-50 saved queries per run. TED queries accept text, cpv, countries and minValueEur; USAspending queries accept text, agencies and naics. A query with only free text (or no filters) runs on both sources.
• Returns one row per notice/award in a unified camelCase schema, with a mandatory source field (ted or usaspending).
• Delta / dedupe: delivered (source, noticeId) pairs are stored in the named key-value store ted-tender-monitor-state. A second run with the same filters returns 0 new rows and therefore 0 result charges.
• Enforces TED's fair-use limits locally: 700 requests/minute and 600 downloads/6 minutes, with exponential backoff on 429/5xx.
• No person-level data. TED contact/person/UBO fields are never requested and are scrubbed again before delivery (RGPD). USAspending rows use contract and organisation fields only.
• Discards schema-invalid items instead of failing the whole run; discarded items are never pushed and never billed.

Input

{
  "source": "all",
  "queries": [
    { "cpv": ["48000000"], "countries": ["ES"], "minValueEur": 100000, "text": "software" },
    { "agencies": ["Department of Energy"], "naics": "511210", "text": "cloud" }
  ],
  "maxResults": 500
}

Field	Type	Default	Description
source	ted, usaspending or all	all	Which source(s) to query.
queries	array, 1-50	required	Saved searches. All filters inside a query are optional.
queries[].text	string	-	Free-text keywords, applied to TED and/or USAspending.
queries[].cpv	array of strings	-	TED CPV code or prefix (48000000, 48).
queries[].countries	array of strings	-	TED buyer countries, ISO2 or ISO3.
queries[].minValueEur	number	-	TED only: minimum published estimated value in EUR. Rows without a published EUR value are excluded.
queries[].agencies	array of strings	-	USAspending top-tier agency name, slug or abbreviation (Department of Energy, DOE).
queries[].naics	string	-	USAspending NAICS code.
maxResults	integer, 1-10000	500	Maximum new rows delivered in the run, across all queries and sources.

Each run looks back 7 days on both sources (TED publication date, USAspending last modification date), so a monitor never walks the full historical corpus. Duplicates across queries are removed within the run.

Minimum input

{}

No field is required. Empty input returns notices from the Actor's default recent publication window (up to 200 results). Filters: query (free text), countries (ISO2/ISO3 codes), cpvCode (code or prefix), publishedSince (YYYY-MM-DD), maxResults.

Output fields

• notice_id - TED publication number (e.g. 674231-2026).
• publication_date, deadline_date - YYYY-MM-DD; deadline is null when the notice does not publish one.
• buyer_name, buyer_country - contracting organisation and ISO2 country. Organisation only, never a person.
• title - notice title, preferring the English translation returned by TED.
• procedure_type - normalised value, e.g. Open procedure.
• contract_value and currency - sum of lot estimated values when published; null when unknown.
• cpv_codes - deduplicated CPV classification codes.
• notice_url and url - public TED detail page; scrapedAt - UTC run timestamp.

Example output

Real item from the saved USAspending fixture (test/fixtures/usaspending-awards.json):

{
  "source": "usaspending",
  "noticeId": "CONT_AWD_36C25024F0188_3600_47QTCA19D00MV_4732",
  "title": "GEO TAB - SAGINAW",
  "buyer": "Department of Veterans Affairs",
  "country": "US",
  "cpv": [],
  "value": 13162,
  "currency": "USD",
  "publishedAt": "2023-12-01",
  "deadlineAt": null,
  "periodEndAt": "2024-11-30",
  "url": "https://www.usaspending.gov/award/CONT_AWD_36C25024F0188_3600_47QTCA19D00MV_4732",
  "firstSeenAt": "2026-10-19T12:00:00.000Z",
  "scrapedAt": "2026-10-19T12:00:00.000Z"
}

TED rows use the same fields:

Field	TED	USAspending
noticeId	TED publication number	generated_internal_id
title	Notice title (English preferred)	Award description, or award ID and recipient
buyer	Contracting authority	Awarding agency
country	Buyer country, ISO2	US
cpv	CPV codes	Empty array (the source uses NAICS)
value / currency	Summed lot estimate and its currency	Award amount and USD
publishedAt	Publication date	Award start date
deadlineAt	Receipt-of-tenders deadline	null
periodEndAt	null	End of the period of performance
firstSeenAt	When this Actor first delivered the row	Same

value and currency are exactly what the source publishes: no conversion and no invented amounts. url points to the official notice/award page.

Pricing

• 2.00 USD per 1,000 new rows (apify-default-dataset-item = 0.002 USD per row), plus the Apify-recommended apify-actor-start event (0.00005 USD per run, first 5 seconds of compute included).
• A re-run with no new rows emits 0 rows = 0 result charges.
• Empty results, errors, retries and previously delivered rows are not charged.

Cost per 1,000 results

Measured locally on 22 October 2026 with npm run fixture:bench (saved official fixtures, no APIFY_TOKEN, 0 Apify credit):

• extract 1,000 rows + validate them against the dataset schema: about 19 ms of local CPU;
• at 256 MB and 0.20 USD per GB-hour, that is about 0.0000003 USD of compute per 1,000 rows;
• both APIs are keyless and free, no proxy is used.

Measured cost per 1,000 results is therefore well below 0.001 USD; at 2.00 USD per 1,000 the gross margin is above 99.9%. Full method in docs/measurements.md.

Legal and access (step 0, 22 October 2026)

Both sources were probed without credentials before building:

• TED: POST https://api.ted.europa.eu/v3/notices/search returned HTTP 200 with 3 notices and totalNoticeCount: 1995 for a free-text + date query. No API key is required, so the bulk daily download fallback is not needed.
• USAspending: POST https://api.usaspending.gov/api/v2/search/spending_by_award/ returned HTTP 200 with 2 awards. The API is public and keyless.

Terms-of-service sweep, both sources CLEAR:

• TED legal notice: procurement notices in the Supplement to the Official Journal "can be freely reused, for commercial or non-commercial purposes"; editorial content is CC BY 4.0. No clause forbids automated extraction. https://ted.europa.eu/robots.txt only disallows login, private dashboards and internal Liferay patterns; the API host is not restricted (/robots.txt 404).
• USAspending: U.S. federal government data in the public domain (17 U.S.C. section 105), documented public API with no login or key. The API host has no robots rules (/robots.txt 404); the www site robots file does not apply to the API host. No anti-scraping clause in the endpoint docs.

No login, paywall, CAPTCHA or private content is accessed.

Monitor state and backfill

• Delivered pairs live in the named key-value store ted-tender-monitor-state (key seen-notices), with (source, noticeId) keys.
• Entries are retained for 365 days and capped at 200,000 pairs, newest first.
• To backfill from scratch, delete that named store in the Apify Console before the run. Backfilled rows are new to the state and are billed once.

Upgrading from v1

v2 is a breaking schema change, intentional and documented in the build: v1 emitted snake_case TED-only fields (notice_id, buyer_name, contract_value, cpv_codes, ...). v2 emits the unified camelCase schema above and adds source, firstSeenAt and periodEndAt. Update any consumer that reads v1 field names. The Actor version is now 0.2.

Works on the Apify free plan - about 2,499 tenders for the $5 monthly credit ((5.00 USD - 0.0002 USD run-start at the default 4 GB memory) / 0.002 USD per tender).

FAQ

Does it need an API key? No. Both sources are public and keyless.

What is the difference between TED and USAspending? TED publishes EU procurement notices with deadlines and CPV codes. USAspending publishes U.S. federal contract awards with a period of performance and USD amounts. One Actor normalises both.

Why is buyer the awarding agency for USAspending? In the unified schema buyer means the public body buying the goods or services. The award recipient appears in title (the description or "award ID - recipient").

Why is cpv empty for USAspending? USAspending classifies contracts with NAICS, not CPV. The Actor never copies NAICS into cpv.

Why did my second run return nothing? It is a monitor: already delivered (source, noticeId) pairs are not repeated or billed. Delete the named state store to backfill.

How do I run only one source? Set "source": "ted" or "source": "usaspending".

Development and tests

Everything below runs offline against saved official fixtures and spends no Apify credit:

npm install --include=dev
npm test                # lint + typecheck + Vitest
npm run fixture:bench   # local cost measurement, no APIFY_TOKEN
npm run publish:check   # Steadfetch pre-publication gate
npm run build           # tsc production build

The suite covers both extractors, schema acceptance and rejection, request construction, agency resolution, query-to-source routing, the two-source orchestration and the persistent delta (a second run with the same filters returns 0 new rows).

Maintenance

This Actor is tested daily by the Steadfetch catalog QA workflow with fixture-based tests and a minimal platform run (source: all, one saved query, 10 rows). If a source changes a field, schema validation discards the invalid rows and logs them instead of delivering bad data.