BidNet Direct Scraper

Scrape US government bid solicitations, RFPs and RFQs from BidNet Direct — keyless, datacenter proxy.

What does BidNet Direct Scraper do?

BidNet Direct is the largest US public procurement portal, aggregating state, county, municipal and federal solicitations across regional purchasing groups. This actor reads the public solicitation listing pages at bidnetdirect.com/public/solicitations, filters by keyword and publication window, and returns structured bid records. No login, no API key. Datacenter proxy is required — BidNet blocks requests without a proxy.

Output fields

Field	Type	Description
title	string	Solicitation title
bid_id	string	BidNet numeric solicitation ID
agency	string	Issuing organization (when available)
location	string	State or locality (when available)
publication_date	string	Publication date (when available)
closing_date	string	Response deadline (when available)
status	string	open, closed or awarded
url	string	Direct solicitation detail URL
source_keyword	string	Keyword that produced this row
source_page_url	string	Search page URL
scraped_at	string	ISO timestamp

Who is it for?

• Government contractors monitoring new solicitations matching their NAICS codes
• Bid/proposal teams tracking deadlines across multiple purchasing groups
• Sales teams identifying agencies buying specific goods or services
• Procurement analysts studying bid volume by state, category or agency
• AI agents and MCP users pulling live government bid data for analysis

Pricing

Pay per result: $1.20 per 1,000 bids.

Charged only for records actually written to the dataset.

How to use it

1. Open the actor in Apify Console.
2. Enter one or more keywords (e.g. water, construction, IT services).
3. Optionally filter by state, bid status and publication window.
4. Optionally cap results per keyword.
5. Leave the proxy at the default (datacenter) and click Start.

Output example

{
  "title": "Regional Water Metering & Billing Study",
  "bid_id": "444118548053",
  "agency": "Wyoming Purchasing Group",
  "location": "WY",
  "publication_date": "08/04/2026 03:21 PM EDT",
  "closing_date": "08/24/2026 07:00 PM EDT",
  "status": "open",
  "url": "https://www.bidnetdirect.com/wyoming/solicitations/open-bids/statewide/2026-Regional-Water-Metering-Billing-Study/444118548053",
  "source_keyword": "water",
  "source_page_url": "https://www.bidnetdirect.com/public/solicitations/open?keywords=water&publishDate=DAYS_7",
  "scraped_at": "2026-10-02T12:00:00.000Z"
}

Technical details

• Stack: Node.js 24, apify SDK, native fetch, cheerio for HTML parsing.
• Endpoint: https://www.bidnetdirect.com/public/solicitations/open with keyword and publication-window query parameters. Publicly accessible without login.
• Proxy: datacenter proxy is required. BidNet blocks requests without a proxy. The actor defaults to the Apify datacenter pool.
• Pagination: page-based, up to 20 pages per keyword.
• Retries: 429 and 5xx responses are retried up to three times with backoff.
• Parsing: HTML anchor extraction with row-context metadata lookup, plus JSON-LD fallback.

Known limits

• Detail pages are not fetched. Some fields (agency, location, exact dates) are only reliably present on detail pages. The actor extracts what is available on the listing page to avoid the detail-page fragility reported by competing actors.
• Proxy required. Datacenter proxy is mandatory; residential is not needed but datacenter is the confirmed working configuration.
• Page structure can change. BidNet is a JS-rendered React application. The actor reads server-rendered content. If the site moves to full client-side rendering, the parser needs updating.
• No full-text search. BidNet's advanced search filters are not all exposed in the public URL. Keyword plus publication window is the supported subset.

FAQ

Do I need an API key? No. The public listing pages are accessible without login.

Do I need a proxy? Yes. BidNet Direct blocks requests without a proxy. Datacenter proxy is sufficient and is the default.

Why not fetch detail pages? Competing actors report frequent "Could not extract required fields" errors on detail pages, which indicates the site's page structure changes often. The actor stays on the list page for reliability.

Can I filter by state? Yes, via the states input. The URL patterns for state-specific listings are supported.

What is the difference between open, closed and awarded? Open solicitations are currently accepting responses. Closed solicitations have passed their deadline. Awarded solicitations have a winning vendor.

Support

Open an issue on the Actor's page for bugs or feature requests.

Changelog

0.1 — 2026-10-02

• Initial build. Keyless extraction from BidNet Direct public solicitation pages.
• Keyword search with publication-window filter.
• Bid status filter (open, closed, awarded, all).
• State filter (via URL patterns).
• HTML anchor extraction with row-context metadata lookup.
• JSON-LD fallback for structured data.
• Pagination: up to 20 pages per keyword.
• Datacenter proxy required and enabled by default.
• Retries on 429 and 5xx with backoff.
• No login, no API key, no detail-page dependency.