All notable changes to the H1B Visa Database Scraper (haketa/h1b-visa-database-scraper) are documented here. Dates are in YYYY-MM-DD.
- Fresh verification run (datacenter HTTP): a six-employer FY-2024 query (
amazon, microsoft, google, infosys, tata consultancy services, cognizant) returned 500 certified petitions in ~5 seconds with 100% coverage on every core field (employer, job title, base salary, city, state, submit date, start date, year, case status, case number, source URL) and a 0.00% duplicate rate on the DOL case number.
- Cross-product merge confirmed: a follow-up narrow run (
stripe + databricks + snowflake × software engineer) returned 215 petitions across all three sponsors with zero duplicates, verifying the multi-employer fan-out and global de-duplication.
- README refresh: replaced the Output JSON examples with real rows from this run; documented
baseSalaryDisplay as the site's comma-grouped, symbol-less format (e.g. 137,300) rather than a $-prefixed string; corrected the year field description (queried year with a submitDate fallback); added a dedicated Pagination & scale section; refreshed the smoke-test recipe and runtime numbers.
- Added this standalone
CHANGELOG.md (previously only an inline README changelog existed).
- No code changes, no input changes, no output field renames. The parser, de-dup logic, and schema are unchanged — the actor was already healthy.
- Docs refresh — README brought in line with the current engine: cheerio replaced by a memory-bounded streaming
<tbody> regex scan (handles 18 MB / 30k-row pages without OOM), 90-second request timeout, DOL case-number primary de-dup key, 200-row batched pushes, and clean exit (not a failed run) when a search legitimately matches nothing.
popularEmployers dropdown and empty-run sample behaviour documented.
- No input/output field names changed.
- Maintenance release — actor verified and maintained; data pipeline tested for quality, structure and freshness; selectors/endpoints confirmed against the live site.
- Initial public release of the H1B Visa Database Scraper — HTTP-only via
got-scraping, full h1bdata.info filter parity (employer × job title × city × year), salary-band post-filter, and DOL-case-number de-duplication.
- (Early builds used a cheerio DOM parse and failed the run on empty results; both were superseded — see 1.0.5.)
- The same actor ID was previously published as
salary-com-scraper. It was repivoted to H1B disclosure data because Salary.com largely duplicated freely-available BLS/Glassdoor content, while the H1B public-disclosure niche is genuinely unique, primary-source, and high-value for immigration law, recruiting, journalism, and policy research. The actor ID is unchanged; only the source target, schema, and engine differ.