DealWatch

DealWatch scrapes product listings, stores results in SQLite, and can send alert messages.

This README covers local setup through running the scraper test script: app/scripts/test_scraper.py.

1. Prerequisites

• Python 3.11+
• uv installed
• Linux desktop/headless environment capable of running Playwright/Camoufox

2. Clone And Enter Project

git clone <your-repo-url>
cd dealwatch

3. Set Python Module Path

Run commands from the project root and set:

$export PYTHONPATH=.

If you prefer per-command usage, prefix commands like:

$PYTHONPATH=. uv run python app/scripts/test_scraper.py

4. Configure Environment

Create .env in the project root:

cat > .env << 'EOF'
DB_PATH=sql.db
EOF

DB_PATH is the SQLite file used by the DAO layer.

5. Install Dependencies

$uv sync

6. Install Browser Runtime

For the current Home Depot config (app/configs/yaml/homedepot.com.yaml), Camoufox is enabled, so fetch Camoufox:

$uv run python -m camoufox fetch

Playwright browser install :

$uv run python -m playwright install

7. Seed Required Database Records

test_scraper.py uses domain_name="homedepot.com" and keyword search.
You must have:

• a websites row for homedepot.com
• at least one keyword in keywords

Add website:

$uv run python app/scripts/add_website.py --domain-name homedepot.com

Add one keyword:

$uv run python app/scripts/add_keyword.py --keyword screwdriver

Or bulk load from keywords.txt:

$uv run python app/scripts/add_keyword_from_txt.py

8. Add Telegram Account Details

If you want alert delivery, add at least one Telegram bot account to the database.

Insert account:

uv run python app/scripts/add_account.py \
  --token "<telegram_bot_token>" \
  --chat-id "<telegram_chat_id>"

Send a test alert to all stored Telegram accounts:

$uv run python app/services/telegram/alert.py "DealWatch setup test message"

If this command reports sent=1 (or more), Telegram setup is working.

9. Run Scraper Test

$uv run python app/scripts/test_scraper.py

The script will launch ListingScraper, run the configured Home Depot flow, and use stored Telegram account details for alert delivery.

10. Common Setup Errors

ModuleNotFoundError: No module named 'app'

• Run from repo root and set PYTHONPATH=..

Website with domain 'homedepot.com' not found in database.

• Run the add_website.py command above.

settings.use_camoufox=true but camoufox is not installed

• Run uv sync and uv run python -m camoufox fetch.

Scraper starts but does nothing

• Add at least one keyword with add_keyword.py or add_keyword_from_txt.py.

11. Configuring Zip Codes

To scrape across multiple store locations, add a zip_codes list to the search_module section of the domain YAML and mark the zip code input action with zip_code_input: true:

search_module:
  zip_codes:
    - "28546"
    - "90210"
  actions_before_search:
    - name: input_action
      selector: '[data-testid="zip-code-text-field"] input'
      zip_code_input: true
      submit: false
      loading_delay: 2

The scraper will run the full keyword search once per zip code. See docs/setup_guide.md for full details.