Aqarmap KSA Property Scraper

A thin Apify actor shell connecting sa.aqar.fm to the CleanedWeb source package aqarmap_ksa_property_scraper.

This repo is not the scraper engine. It handles platform packaging: public input validation, actor metadata, README/schema polish, calling CleanedWeb, and saving returned listing records or user-facing artifacts. Source-specific request construction, pagination, parsing, enrichment, normalization, proxy routing, browser strategy, maps, and report generation live in /Users/fatihtahta/allrepos/cleanedweb-com under cleanedweb/sources/<source_id>/.

Runtime Flow

Apify input
-> src/models.py validates property type, location, filters, and limit
-> src/cleanedweb_client.py calls CleanedWeb source-run API
-> CleanedWeb executes cleanedweb/sources/<source_id>/
-> this actor saves returned records to the default dataset
-> this actor saves returned maps/reports/artifacts to the key-value store

Actor logs should stay calm and public: Start, Prep, Work, Save, Done. Keep headers, cookies, proxy strategy, retries, debug internals, and blocked-response details in CleanedWeb telemetry.

Project Layout

• src/main.py: Apify entrypoint.
• src/models.py: public real-estate input and CleanedWeb response models.
• src/cleanedweb_client.py: source-run API client.
• src/output.py: dataset and key-value store saving.
• src/crawler.py: thin runtime orchestration.
• .actor/: Apify packaging, public input, dataset/output/KVS schemas.
• scripts/register_apify_actor.py: create/register the Apify actor and add missing CleanedWeb env vars without overwriting existing actor env vars.
• scripts/check_dataset.py: local dataset sanity checker.

Do not add target parsers, request builders, pagination loops, proxy ladders, or browser automation to this repo. Add those to cleanedweb/sources/<source_id>/.

Public Inputs

• property_type: one target-supported listing category. Defaults to all; options cover Aqar's sale, rent, and furnished-booking routes.
• location_city
• location_direction
• location_district
• single-select filters for family type, payment frequency, bedrooms, living rooms, bathrooms, floor, property age, usage type, street direction, minimum street width, apartment count, and store count
• furnished: a boolean checkbox
• amenities: a multi-select of Aqar-supported property features
• optional minimum/maximum price, price-per-square-metre, and area ranges
• limit

The actor shows clear property-type labels while CleanedWeb owns the matching Aqar route, category ID, listing mode, pagination, and record interpretation. For example, apartment_for_rent maps to Aqar's apartment rental route and power_stations_for_sale maps to its power-station sale route. Filter availability differs by property type. CleanedWeb validates the selected combination against Aqar's category-specific search controls and returns a clear error instead of silently dropping an unsupported filter.

Keep the public input panel focused on user-facing listing scope and output volume. Broker URL, token, proxy routing, retries, cookies, sessions, and strategy choices are environment/configuration concerns, not public input fields.

CleanedWeb Configuration

Use cleanedweb.env.example as the local reference. Required values:

Variable	Purpose
CLEANEDWEB_SOURCE_ID	Source package id, matching cleanedweb/sources/<source_id>/.
CLEANEDWEB_API_URL or CLEANEDWEB_BROKER_URL	CleanedWeb source-run service base URL.
CLEANEDWEB_API_TOKEN or CLEANEDWEB_BROKER_TOKEN	Bearer token for CleanedWeb.
CLEANEDWEB_ACTOR_SLUG	Stable shell slug for telemetry and registry grouping.
CLEANEDWEB_ACTOR_ID	Stable platform-neutral shell id.
CLEANEDWEB_PLATFORM	Usually apify for this template.

Optional Apify identity:

• CLEANEDWEB_PLATFORM_ACTOR_ID
• CLEANEDWEB_PLATFORM_ACTOR_SLUG
• CLEANEDWEB_ACTOR_RUN_ID
• CLEANEDWEB_PLATFORM_RUN_ID

Never expose CleanedWeb credentials or strategy controls in .actor/input_schema.json, public README sections, dataset rows, or logs.

Register The Apify Actor

After replacing placeholders in .actor/actor.json, run:

python scripts/register_apify_actor.py --dry-run
python scripts/register_apify_actor.py --apply

When this repo is cloned from the delegated template, replace the inherited template remote before registration. The actor shell needs its own cloud GitHub repo; do not push back into delegated-actor-template-*.

git remote remove origin 2>/dev/null || true
gh repo create Fatiht8a/<actor-repo-slug> --private
git remote add origin git@github.com:Fatiht8a/<actor-repo-slug>.git
git push -u origin main
git remote get-url origin
git remote -v
git status --short --branch
git ls-remote --heads origin main

If the GitHub repo already exists:

git remote remove origin 2>/dev/null || true
git remote add origin git@github.com:Fatiht8a/<actor-repo-slug>.git
git push -u origin main
git remote get-url origin
git remote -v
git status --short --branch
git ls-remote --heads origin main

The helper:

• finds or creates the Apify actor
• connects the Apify actor version to this Git repo as sourceType=GIT_REPO
• writes non-secret CleanedWeb identity into .actor/actor.json metadata
• provisions missing CleanedWeb env vars on the Apify actor
• preserves existing actor env vars by default
• sends CLEANEDWEB_BROKER_TOKEN as a secret

The helper uses git config --get remote.origin.url by default. Pass --git-repo-url only for a target-specific SSH Git URL such as git@github.com:Fatiht8a/<actor-repo-slug>.git. It rejects HTTPS, local filesystem remotes, and delegated-template remotes because private actor shells need Apify deploy-key access. For private repositories, add Apify's deployment key to the GitHub repo deploy keys with read access, or report that as the remaining blocker.

Do not run registration until git ls-remote --heads origin main returns a refs/heads/main line. Otherwise the cloud repo is not synced yet and Apify will not build the local actor shell you just edited.

When catalog metadata is known, also update the central registry:

python scripts/register_apify_actor.py --apply \
  --cleanedweb-root /Users/fatihtahta/allrepos/cleanedweb-com \
  --primary-source-domain example.com \
  --vertical property_listing \
  --primary-record-type property \
  --record-type property \
  --target-schema real_estate_property

Leave unknown catalog fields out instead of guessing.

Local Development

Install dependencies:

$pip install -r requirements.txt -r requirements-dev.txt

Run checks:

apify validate-schema
ruff check .
mypy src
pytest

For runtime smokes, point the actor at a reachable CleanedWeb source-run service and inspect both the default dataset and any key-value store artifacts:

export CLEANEDWEB_API_URL='https://broker.cleanedweb.com'
export CLEANEDWEB_API_TOKEN='<private-cleanedweb-token>'
export CLEANEDWEB_SOURCE_ID='<source_id>'
export CLEANEDWEB_ACTOR_SLUG='<actor-slug>'
export CLEANEDWEB_ACTOR_ID='<actor-slug>'
export CLEANEDWEB_PLATFORM='apify'

apify run --purge --input '{"property_type":"apartment_for_rent","location_city":"riyadh","location_direction":"South of Riyadh","family_type":"family","bedrooms":"2","amenities":["air_conditioned","maid_room"],"min_price":3000,"max_price":4000,"limit":5}'
python scripts/check_dataset.py --min-items 0

If a source run fails, debug from CleanedWeb with narrow filters first:

curl -sS "$CLEANEDWEB_BROKER_URL/v1/telemetry/fetch-attempts?actor=<actor-slug>&platformRunId=<run-id>&limit=50" \
  -H "Authorization: Bearer $CLEANEDWEB_BROKER_TOKEN"

curl -sS "$CLEANEDWEB_BROKER_URL/v1/telemetry/fetch-attempts?traceId=<trace-id>&limit=10" \
  -H "Authorization: Bearer $CLEANEDWEB_BROKER_TOKEN"

Do not treat a successful process exit as proof. Inspect saved records and artifacts.