# Changelog of Grants Gov Scraper (`normdata/grants-gov-scraper`) Actor

- **URL**: https://apify.com/normdata/grants-gov-scraper/changelog.md
- **Full Actor documentation**: https://apify.com/normdata/grants-gov-scraper.md

## Changelog

### 2026-09-15 (3)

- **Real bug found by Fabri from live Console output, fixed:** raw HTML entities were left literal in
  plain fields (a real title showed `–` instead of the real "–"), and the long-form description
  still carried raw markup (`<p>`, `<ul>`, `<li>`) - the HTML-cleaning helper SAM.gov has was never
  ported to this Actor. Fixed with a `decodeEntities()` covering every text field (not just the
  description) and a `stripHtml()` for the long-form fields, both applied consistently now.
- Found a second real bug while fixing the first: a first version of `decodeEntities()` replaced
  `&nbsp;` before `&amp;`, so a real double-escaped source string (`&amp;nbsp;`) was never caught -
  the `&nbsp;` regex doesn't match `&amp;nbsp;` literally. Fixed by decoding `&amp;` first.
- Scanned 200 real, diverse opportunities end to end for any leftover raw entity or tag after the fix
  (0 found) - caught two more real entities this way that weren't in the original fix (`&middot;`,
  `&sect;`, `&eacute;`, `&bull;`), each confirmed against real live text before adding.
- 29 unit tests, 86 `expect()` calls (up from 26/76), including the exact real opportunity
  (`RFA-NS-28-008`) that first exposed this in Fabri's own Console output.

### 2026-09-15 (2)

- **Real bug found by intensive testing before shipping, and fixed:** a `fundingCategories`-filtered
  live search (server confirmed narrowing results correctly) came back with `funding_categories: []`
  on every single row - not a filtering problem, a data-reading one. Root cause: a FORECASTED
  opportunity (not yet formally posted) carries its funding/eligibility/contact data under a
  completely separate `forecast` key, with `synopsis` absent entirely - confirmed live on a real
  record. The code only ever read `synopsis`, so every forecasted opportunity (a normal, common part
  of Grants.gov's own default `forecasted|posted` status filter) silently returned nulls for funding,
  eligibility, and contact fields. Fixed by reading whichever of `synopsis`/`forecast` is real, with a
  new `deadline_is_estimated` field so a forecast's estimated deadline is never presented as a
  confirmed one.
- Re-verified every filter (`agencies`, `fundingCategories`, `fundingInstruments`, `eligibilities`,
  `oppStatuses`) against live results field-by-field after the fix - 0 mismatches across 4 separate
  filtered searches (was 8/8 and 7/8 wrong before the fix).
- Concurrency tuned by direct measurement on real 100-row runs: 8 → 30s, 16 → 9-19s (real run-to-run
  variance), 20 → 48s, 30 → 30s - pushing concurrency past ~16-20 reliably got slower, a real
  server-side effect under heavier concurrent load, not a client bottleneck. Settled on 10.
- 26 unit tests, 76 `expect()` calls (up from 21/63), including a dedicated regression fixture for a
  real forecasted opportunity.

### 2026-09-15

- Initial build: search (keyword/agency/category/instrument/eligibility/status/date filters),
  lookup (by public opportunity number), and monitor (opportunities posted since a date) modes.
- Confirmed live, contrary to Grants.gov's own documented parameters, that `search2`'s
  `postedFrom`/`postedTo` have no effect on results in any date format or field-name variant tried
  (identical hit counts with and without them). Fixed by filtering on the real `posted_date` client-side
  instead of trusting the unverified server parameter.
- Confirmed live that `fetchOpportunity` only resolves the internal numeric opportunity id, not the
  public opportunity number a real user would have - lookup mode resolves the number to its id via
  an exact (not fuzzy) match against `search2`'s own keyword results first.
- **Real bug found and fixed before shipping:** the per-revision "what changed" field
  (`synopsisModifiedFields`) was initially read from inside each `opportunityHistoryDetails` entry,
  where it is confirmed live to always be empty - the real, populated field is a sibling of
  `opportunityHistoryDetails` at the top level of the detail response. Verified against a real
  opportunity (`PA-FPH-27-001`) that had a real revision (a due-date push) before fixing.
- Confirmed live that `agencies` requires the exact sub-agency code as it appears in a real record
  (e.g. `HHS-OPHS`), not the parent department code alone (`HHS` alone matches nothing).
- 21 unit tests, 63 `expect()` calls, against real saved API-response fixtures (a real search page, a
  real full opportunity detail with funding/eligibility/revision data, and a real not-found response).
- Full live field audit: search+detail rows (32 fields), lookup not-found rows (+`query`/`error`),
  and monitor rows (+`change_type`) diffed to zero discrepancy against `dataset_schema.json`.