# BidNet Direct Scraper (`aurenic/bidnet-direct-scraper`) Actor

Scrape US government bid solicitations, RFPs and RFQs from BidNet Direct. Filter by keyword, state, status, category and agency type. Keyless, datacenter proxy.

- **URL**: https://apify.com/aurenic/bidnet-direct-scraper.md
- **Developed by:** [Aurenic](https://apify.com/aurenic) (community)
- **Security:** Limited permissions
- **Stats:** 3 total users, 3 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.20 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## BidNet Direct Scraper

Scrape US government bid solicitations, RFPs and RFQs from BidNet Direct — keyless, datacenter proxy.

### What does BidNet Direct Scraper do?

BidNet Direct is the largest US public procurement portal, aggregating state, county, municipal and federal solicitations across regional purchasing groups. This actor reads the public solicitation listing pages at `bidnetdirect.com/public/solicitations`, filters by keyword and publication window, and returns structured bid records. No login, no API key. Datacenter proxy is required — BidNet blocks requests without a proxy.

### Output fields

| Field | Type | Description |
|---|---|---|
| title | string | Solicitation title |
| bid_id | string | BidNet numeric solicitation ID |
| agency | string | Issuing organization (when available) |
| location | string | State or locality (when available) |
| publication_date | string | Publication date (when available) |
| closing_date | string | Response deadline (when available) |
| status | string | open, closed or awarded |
| url | string | Direct solicitation detail URL |
| source_keyword | string | Keyword that produced this row |
| source_page_url | string | Search page URL |
| scraped_at | string | ISO timestamp |

### Who is it for?

- **Government contractors** monitoring new solicitations matching their NAICS codes
- **Bid/proposal teams** tracking deadlines across multiple purchasing groups
- **Sales teams** identifying agencies buying specific goods or services
- **Procurement analysts** studying bid volume by state, category or agency
- **AI agents and MCP users** pulling live government bid data for analysis

### Pricing

**Pay per result: $1.20 per 1,000 bids.**

Charged only for records actually written to the dataset.

### How to use it

1. Open the actor in Apify Console.
2. Enter one or more keywords (e.g. water, construction, IT services).
3. Optionally filter by state, bid status and publication window.
4. Optionally cap results per keyword.
5. Leave the proxy at the default (datacenter) and click **Start**.

### Output example

```json
{
  "title": "Regional Water Metering & Billing Study",
  "bid_id": "444118548053",
  "agency": "Wyoming Purchasing Group",
  "location": "WY",
  "publication_date": "08/04/2026 03:21 PM EDT",
  "closing_date": "08/24/2026 07:00 PM EDT",
  "status": "open",
  "url": "https://www.bidnetdirect.com/wyoming/solicitations/open-bids/statewide/2026-Regional-Water-Metering-Billing-Study/444118548053",
  "source_keyword": "water",
  "source_page_url": "https://www.bidnetdirect.com/public/solicitations/open?keywords=water&publishDate=DAYS_7",
  "scraped_at": "2026-10-02T12:00:00.000Z"
}
```

### Technical details

- **Stack:** Node.js 24, `apify` SDK, native `fetch`, `cheerio` for HTML parsing.
- **Endpoint:** `https://www.bidnetdirect.com/public/solicitations/open` with keyword and publication-window query parameters. Publicly accessible without login.
- **Proxy:** datacenter proxy is required. BidNet blocks requests without a proxy. The actor defaults to the Apify datacenter pool.
- **Pagination:** page-based, up to 20 pages per keyword.
- **Retries:** 429 and 5xx responses are retried up to three times with backoff.
- **Parsing:** HTML anchor extraction with row-context metadata lookup, plus JSON-LD fallback.

### Known limits

- **Detail pages are not fetched.** Some fields (agency, location, exact dates) are only reliably present on detail pages. The actor extracts what is available on the listing page to avoid the detail-page fragility reported by competing actors.
- **Proxy required.** Datacenter proxy is mandatory; residential is not needed but datacenter is the confirmed working configuration.
- **Page structure can change.** BidNet is a JS-rendered React application. The actor reads server-rendered content. If the site moves to full client-side rendering, the parser needs updating.
- **No full-text search.** BidNet's advanced search filters are not all exposed in the public URL. Keyword plus publication window is the supported subset.

### FAQ

**Do I need an API key?**
No. The public listing pages are accessible without login.

**Do I need a proxy?**
Yes. BidNet Direct blocks requests without a proxy. Datacenter proxy is sufficient and is the default.

**Why not fetch detail pages?**
Competing actors report frequent "Could not extract required fields" errors on detail pages, which indicates the site's page structure changes often. The actor stays on the list page for reliability.

**Can I filter by state?**
Yes, via the states input. The URL patterns for state-specific listings are supported.

**What is the difference between open, closed and awarded?**
Open solicitations are currently accepting responses. Closed solicitations have passed their deadline. Awarded solicitations have a winning vendor.

### Support

Open an issue on the Actor's page for bugs or feature requests.

### Changelog

#### 0.1 — 2026-10-02

- Initial build. Keyless extraction from BidNet Direct public solicitation pages.
- Keyword search with publication-window filter.
- Bid status filter (open, closed, awarded, all).
- State filter (via URL patterns).
- HTML anchor extraction with row-context metadata lookup.
- JSON-LD fallback for structured data.
- Pagination: up to 20 pages per keyword.
- Datacenter proxy required and enabled by default.
- Retries on 429 and 5xx with backoff.
- No login, no API key, no detail-page dependency.

# Actor input Schema

## `keywords` (type: `array`):

Keywords to search bids by, e.g. water, construction, IT services.

## `states` (type: `array`):

Two-letter US state codes to filter by, e.g. CA, TX, NY. Leave empty for all states.

## `bidStatus` (type: `string`):

Filter by solicitation status.

## `publishedWithin` (type: `string`):

Only include bids published within this window.

## `maxResultsPerKeyword` (type: `integer`):

Cap records written per keyword.

## `proxy` (type: `object`):

RESIDENTIAL proxy is required. BidNet Direct blocks datacenter IPs and unproxied requests.

## Actor input object example

```json
{
  "keywords": [
    "water",
    "construction",
    "IT services"
  ],
  "states": [],
  "bidStatus": "open",
  "publishedWithin": "HOURS_24",
  "maxResultsPerKeyword": 100,
  "proxy": {
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL"
    ]
  }
}
```

# Actor output Schema

## `dataset` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("aurenic/bidnet-direct-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("aurenic/bidnet-direct-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call aurenic/bidnet-direct-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,aurenic/bidnet-direct-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/5soW9s5cl2YzQGeiz/builds/aVpwTXLkhHreeOiDA/openapi.json