colombia-secop-scraper

Apify Actor for Colombia's SECOP II public procurement awards: awarded state contracts (buyer, supplier NIT, amount COP, dates) read from the national open-data mirror datos.gov.co (Socrata dataset jbjy-vk9h, "SECOP II - Contratos Electrónicos") over plain HTTP — no browser, no proxy, no key (see docs/FINDINGS.md for the probe matrix that settled this lane; the SECOP portals themselves are IP-gated).

Input

All filters are optional; an unscoped run returns the most recently signed contracts:

{"keyword": "computador"}
{"buyer": "MINISTERIO DE EDUCACION"}
{"supplier": "900156264"}
{"dateFrom": "2026-08-01", "dateTo": "2026-08-31"}
{"keyword": "aeronaves", "dateFrom": "2026-08-01", "dateTo": "2026-08-31", "maxResults": 500}

• keyword / buyer: text fragments matched case-insensitively against the contract object and the buyer entity name (3-char minimum).
• supplier: NIT/document number (e.g. 900156264) or name fragment (e.g. NUEVA EPS).
• contractId: exact SECOP contract id, e.g. CO1.PCCNTR.3524727.
• dateFrom + dateTo (ISO YYYY-MM-DD, both or neither): inclusive window on the contract-signing date.
• maxResults (1–5000, default 100): the Actor pages through the mirror 1000 rows at a time and slices.

Unpublished drafts (estado_contrato = Borrador — no date, placeholder suppliers) are excluded from every query. The unscoped default also requires a signing date — Socrata would otherwise sort undated contracts ahead of the fecha_de_firma DESC "latest" page.

Output

One dataset item per awarded contract: contractKey (datos.gov.co:contrato:{idContrato}), idContrato, referenciaContrato, idProceso (notice ref), buyerNit/buyerName, departamento/ciudad, supplierDocType/supplierNit/supplierName, amount + currency (always COP) + amountPaid (valor pagado), signedAt/startDate/endDate (ISO), objeto (description), tipoContrato, modalidad, estado, url (SECOP process page), sourceQuery, source, scrapedAt. Registry placeholders ("No Definido", "Sin Descripcion") are null.

How it works

1. GET https://www.datos.gov.co/resource/jbjy-vk9h.json with SoQL params: $select (20 columns), $where (draft exclusion + scopes; text fragments as upper(field) LIKE '%…%', dates on fecha_de_firma), $order (fecha_de_firma DESC, id_contrato DESC for stable offsets), $limit/$offset (1000-row pages).
2. Keyless, cookieless, plain datacenter egress — 6.0M-row mirror of the contracts table, updated by Colombia Compra Eficiente.
3. Records are normalized to the unified award shape (same conventions as the SEACE registry actor: composite key, sourceQuery, null-for-blank).
4. Slow-lane hardening (issue #189): 180 s read timeout plus a 4-attempt 2/4/8s-backoff retry on 429/5xx/transport/JSON failures — a cold unscoped sort can take minutes before the first byte. If the query still fails, the run exits SUCCEEDED with a one-item dataset note (note/error) instead of crashing.

Run tests

uv venv .venv && uv pip install -p .venv/bin/python httpx
PYTHONPATH=src .venv/bin/python -m unittest discover -s tests -v

Fixtures under tests/fixtures/ are live captures from 2026-09-05.

Cost

One JSON request per 1000 records, no proxy: a 40-record date-window run is ~1 request/~1.5 s — ≈ $0.0002/run; a 5000-record pull (5 pages) is ≈ $0.0006.