Tenders Direct Notice Scraper

Apify Actor that scrapes tender notice details from Tenders Direct. The Actor handles authentication with persistent session storage and extracts notice details from individual notice pages.

Features

• Robust Authentication: Handles login with session persistence across runs
• Azure AD Support: Works with both standard login forms and Azure AD SSO flows
• Configurable Timeouts: Adjust navigation timeouts for different network conditions
• Debug Mode: Toggle browser visibility and debug screenshots for troubleshooting
• Flexible Input: Accepts single URLs or arrays of URLs in multiple formats

Input Configuration

The Actor accepts the following input fields (all optional except notice URLs):

Required

• noticeURL (array or string)
  • One or more Tenders Direct notice URLs to scrape
  • Accepts: single string, array of strings, or array of {url: "..."} objects
  • Example: ["https://app.tendersdirect.co.uk/notice/12345"]

Authentication

• email (string)

  • Login email for Tenders Direct
  • Falls back to TD_EMAIL environment variable if not provided
  • Example: "your.email@company.com"

• password (string, secret)

  • Login password for Tenders Direct
  • Falls back to TD_PASSWORD environment variable if not provided
  • Stored securely and masked in UI

Timeouts

• loginTimeout (integer, default: 25000)

  • Maximum time in milliseconds to wait for login page navigation
  • Increase for slow networks or Azure AD SSO flows
  • Range: 5000-120000ms

• noticeTimeout (integer, default: 30000)

  • Maximum time in milliseconds to wait for notice page navigation
  • Increase for large or slow-loading notice pages
  • Range: 5000-120000ms

Debug Options

• headless (boolean, default: true)

  • Run browser in headless mode (no visible window)
  • Set to false when debugging login issues locally

• saveDebugScreenshots (boolean, default: true)

  • Save screenshots and HTML to Key-Value Store when errors occur
  • Useful for debugging but consumes storage
  • Set to false to reduce storage usage in production

Example Input

Minimal (using environment variables)

{
  "noticeURL": [
    "https://app.tendersdirect.co.uk/notice/12345",
    "https://app.tendersdirect.co.uk/notice/67890"
  ]
}

Note: Set TD_EMAIL and TD_PASSWORD in Actor environment variables.

Complete (all options)

{
  "noticeURL": [
    "https://app.tendersdirect.co.uk/notice/12345",
    "https://app.tendersdirect.co.uk/notice/67890"
  ],
  "email": "your.email@company.com",
  "password": "your-password",
  "loginTimeout": 45000,
  "noticeTimeout": 60000,
  "headless": true,
  "saveDebugScreenshots": true
}

Output

The Actor pushes data to the default dataset with the following structure:

{
  "url": "https://app.tendersdirect.co.uk/notice/12345",
  "title": "Notice Title",
  "noticeDetailsHtml": "<div>...</div>",
  "noticeDetailsText": "Plain text content...",
  "scrapedAt": "2025-01-15T10:30:00.000Z"
}

Local Development

Prerequisites

$npm install

Environment Setup

Create environment variables or provide them via input:

export TD_EMAIL="your.email@company.com"
export TD_PASSWORD="your-password"

Run Locally

$npm start

Code Quality

# Format code
npm run format

# Check formatting
npm run format:check

# Lint code
npm run lint

# Fix lint issues
npm run lint:fix

Troubleshooting

Login Failures

1. Set headless: false to see the browser and debug visually
2. Check debug screenshots in Key-Value Store (login-fail.html, login-fail.png)
3. Increase loginTimeout if using Azure AD SSO (try 45000-60000ms)
4. Verify credentials are correct

Notice Extraction Failures

1. Check debug screenshots (notice-miss.html, notice-miss.png)
2. Increase noticeTimeout for slow-loading pages
3. Verify the notice URL is correct and accessible when logged in

Session Issues

• The Actor saves session state to avoid repeated logins
• If experiencing auth issues, the session state is automatically refreshed on login
• Session state is stored in Actor's Key-Value Store as storageState.json

Migration from Environment Variables

If you're currently using TD_EMAIL and TD_PASSWORD environment variables:

1. Your Actor will continue working without changes (backward compatible)
2. To migrate, add email and password to your input JSON
3. Input fields take precedence over environment variables
4. You can remove environment variables after migration

Technical Details

• Built with Crawlee and Playwright
• Uses Chromium browser
• Docker base image: apify/actor-node-playwright-chrome:22-1.54.1
• Node.js with ES modules

Resources

• Apify Platform documentation
• Crawlee documentation
• Playwright documentation
• Apify SDK for JavaScript

Support

For issues or questions:

• Check debug screenshots in Key-Value Store
• Review Actor logs for error details
• Adjust timeout values for your network conditions