Indeed Search Scraper (Playwright, stealth, no proxy)

Scrapes job title / company / location / salary / link from Indeed search result pages, using a headless Chromium browser with stealth patches to reduce the chance of triggering a bot check.

Important — read before running

• No proxy or paid CAPTCHA solver is wired up in this version (per your choice). That means: if Indeed shows you a CAPTCHA/verification page, the scraper will detect it, log a warning, and stop that search rather than push through it. You'll need to slow down further, wait a while, switch networks, or add a proxy later (there's a PROXY slot ready in config.py for when you do).
• Keep request volume low and delays realistic. Aggressive scraping is the #1 reason IPs get flagged. The defaults in config.py are intentionally conservative — don't crank them down to zero.
• Indeed's HTML structure changes periodically. If you suddenly get zero results, open a real search page in your browser, inspect the job cards, and update the selectors in scraper.py (CARD_SELECTORS and the fields pulled from each card).
• This is for personal/research use. Scraping Indeed is against their Terms of Service — that's a business/legal decision for you to weigh, not something this tool resolves.

Setup

python -m venv venv
source venv/bin/activate        # Windows: venv\Scripts\activate

pip install -r requirements.txt
playwright install chromium

Configure

Edit config.py:

• KEYWORDS / LOCATIONS — what to search for
• MAX_PAGES_PER_SEARCH — how many result pages to pull per search
• HEADLESS = False — while debugging, so you can watch the browser and see what Indeed is actually serving you
• PROXY — fill in later if you get a proxy (format is in the comment)

Run

$python main.py

Results append to data/indeed_jobs.csv.

Next steps you might want later

• Add a proxy (PROXY in config.py) once request volume goes up — this matters more than any stealth trick for avoiding blocks.
• Add full job-description scraping by visiting each job_link (a second pass, same stealth-context pattern, with its own delays).
• Swap CSV for SQLite if the dataset grows and you want to query/dedupe it. #� �I�n�d�e�e�d�_�s�c�r�a�p�e�r� � �