Werk.nl Search Service

This service allows searching and extracting candidate profiles from Werk.nl using Playwright for browser automation.

Features

• Automated Search: Searches for candidate profiles in Werk.nl based on specific criteria.
• Profile Extraction: Extracts detailed information from candidate profiles.
• Two-Factor Authentication with Telegram: Handles the 2FA process through a Telegram bot.
• Firebase Integration: Stores and manages extracted data.
• Candidate Ranking: Evaluates and ranks candidates according to predefined criteria.

Technologies Used

• Playwright: Browser automation library.
• Apify SDK: Toolkit for building Python scrapers.
• Firebase: Cloud database for data storage.
• Telegram Bot API: For handling two-factor authentication.

Setup

1. Clone this repository
2. Copy .env.example to .env and configure environment variables
3. Install dependencies with pip install -r requirements.txt
4. Run the service with python -m src

Two-Factor Authentication with Telegram

This service includes support for two-factor authentication (2FA) through a Telegram bot. When Werk.nl requests a two-factor authentication code during login, the service:

1. Sends a message to the user through Telegram informing that 2FA is required.
2. Requests an authentication code through the Telegram bot API.
3. Waits to receive the code through a streaming connection.
4. Enters the code in Werk.nl's 2FA form.
5. Notifies the user when login is successful.

Telegram Bot Setup

To use this functionality, you need to:

1. Create a Telegram bot using BotFather.
2. Implement a bot API that supports the following endpoints:
   • /2fa: For requesting a 2FA code.
   • /request_two_factor/{id}: For receiving the code through streaming.
   • /send_message: For sending messages to the user.
3. Configure the environment variables TELEGRAM_BOT_API_URL and TELEGRAM_USER_ID in your .env file.

2FA Workflow

1. The service attempts to log in to Werk.nl with the provided credentials.
2. If Werk.nl requests 2FA, the service sends a message to the user through Telegram.
3. The user receives the authentication code (via SMS, email, etc.) and provides it to the Telegram bot.
4. The service receives the code through the bot API and enters it in Werk.nl.
5. Login is completed and the service continues with the search process.

Resources

• Playwright for web scraping in 2023
• Scraping single-page applications with Playwright
• How to scale Puppeteer and Playwright
• Integration with Zapier, Make, GitHub, Google Drive and other apps
• Video guide on getting data using Apify API
• Telegram Bot API documentation