OpenSooq Scraper (v0.2)

Give it a mix of direct OpenSooq listing URLs and/or search or category URLs. Search/category URLs get crawled automatically for listing links first; every listing (direct or discovered) is then scraped for: title, price, currency, listing id, city, brand, model, condition, description.

What makes this different from a basic scraper

• ai_tags: OpenSooq's own AI-generated tag list per listing (type, features, warranty, etc.) — structured metadata that's easy to skip if you only scrape the obvious fields.
• seller: rating, review count, member-since date, total listing count — lets a buyer or analyst judge seller trustworthiness, not just read the listing.
• scraped_at timestamp on every record — run this on a schedule and you get your own price-history dataset over time, which OpenSooq itself doesn't expose.
• Search/category crawling, not just single listings — so a user doesn't need to already have a list of listing URLs in hand.

What's proven vs what needs testing (v0.2 status)

• Proven (v0.1, tested across 4 real listings — phones, car, property rental): JSON-LD + spec-table extraction of the core fields works reliably; price=0 "Ask For Price" listings are correctly dropped instead of reported as free.
• New in v0.2, NOT yet tested against a real page:
  • find_listing_links() — the logic that reads a search/category page and pulls out individual listing URLs. The link selector is a reasonable guess (any <a> whose href ends in a long numeric id), but hasn't been run against a real OpenSooq search page yet.
  • ai_tags and seller extraction — built from the same one example listing as v0.1's core fields, not yet cross-checked on other listings.
  • First thing to check: run it with one search/category URL and watch the log. If it logs "0 links found", the selector in find_listing_links() needs adjusting against that page's real HTML.

How to run it locally (test before deploying to Apify)

pip install -r requirements.txt
python3 -m src.main

You'll need an apify_storage/key_value_stores/default/INPUT.json file with your input (or run it through the Apify CLI, which sets this up for you: apify run after apify init pulls in this project).

How to deploy to Apify Store

1. Install the Apify CLI: npm install -g apify-cli
2. From this folder: apify login, then apify push
3. This uploads the Actor to your Apify account — from there you can test runs, set pricing, write a public description, and publish it to the Store from the Apify Console (console.apify.com).