Magicbricks Property Search Scraper

Scrapes publicly rendered Magicbricks property-search result pages with a bounded Crawlee HTML workflow. The Actor uses the page's public HTML; it does not solve CAPTCHAs, bypass login, call private APIs, spoof fingerprints, or inject stealth scripts.

Inputs

searchUrls is the preferred input. It accepts up to 20 HTTPS Magicbricks search URLs, including URLs with filters already applied.

Field	Type	Default	Description
searchUrls	array	—	Public search URLs.
startUrls, sourceUrls, urls, url	array/string	—	Backward-compatible aliases for searchUrls; provide only one alias.
maxPropertiesPerSearch	integer	100	Maximum normal property records per search URL. maxItems and maxProducts are accepted aliases.
maxPages	integer	5	Maximum bounded scroll/pagination cycles per URL.
includeDiagnostics	boolean	false	Emit minimal {url,error,errorCode,scrapedAt} rows for empty/failed pages.
requestDelayMs	integer	250	Delay before each public HTML request, from 0 to 5000 ms.
requestTimeoutSecs	integer	120	Public HTML request timeout, from 10 to 180 seconds.
maxRequestRetries	integer	2	Bounded crawler retries, from 0 to 3.
maxConcurrency	integer	1	Public-request concurrency, from 1 to 3.
fixtureHtml / fixtureFile	string	—	Local QA mode; parses fixture HTML instead of making a network request. Use only one.
fixtureSourceUrl	string	—	Optional Magicbricks source context for fixture records.
proxyConfiguration	object	disabled	Optional standard Apify Proxy configuration.
proxy	object	—	Backward-compatible alias for proxyConfiguration.

Example input:

{
  "searchUrls": [
    "https://www.magicbricks.com/property-for-sale/residential-real-estate?proptype=Multistorey-Apartment&cityName=Mumbai&language=en"
  ],
  "maxPropertiesPerSearch": 25,
  "maxPages": 3,
  "includeDiagnostics": true
}

Dataset output

Each normal row keeps the Actor's existing rich property fields and adds stable/provenance fields:

• recordType: magicbricksProperty
• recordId: same source property identifier as propertyId
• sourceDomain, sourceUrl, searchUrl, pageUrl
• pageNumber, position, and ISO scrapedAt
• existing price, area, location, project, seller, amenity, media, status, and date fields

Image fields contain only absolute HTTP(S) image URLs from Magicbricks image hosts and common image extensions; logo/icon/sprite-like assets are excluded. Missing optional values are omitted rather than replaced with fabricated commercial facts.

Diagnostic rows are opt-in and deliberately minimal:

{
  "url": "https://www.magicbricks.com/...",
  "error": "No public Magicbricks listing records were found on the page.",
  "errorCode": "NO_RESULTS",
  "scrapedAt": "2026-09-09T00:00:00.000Z"
}

The OUTPUT_SUMMARY key-value record reports run status, normal/diagnostic counts, failed requests, limits, proxy usage, and elapsed time. OUTPUT remains available as the compatibility summary key.

Local development

npm ci --ignore-scripts --no-audit --no-fund
npm test
npm run lint
npm run check
npm run validate

The lint script is a bounded JavaScript syntax check for this legacy Actor; it does not scan the repository root. npm run validate checks existing local dataset files. Use an isolated APIFY_LOCAL_STORAGE_DIR for a clean QA run so older shared storage is not overwritten or silently migrated.

Code structure

• src/actor.js owns Apify lifecycle and error handling.
• src/runner.js validates input, fetches public HTML with bounded retries, extracts result cards, and writes summaries.
• src/pagination.js contains the bounded scroll behavior used to request additional public result batches.
• src/property.js normalizes and validates rich listing records, filters media, and adds provenance.
• test/property.test.js covers the normalization/output contract.
• validate-datasets.js checks local rows for required identity/provenance fields and forbidden transport/debug metadata.

Limitations

Magicbricks may vary result-card markup, return no rows, or require a normal user interaction before results are rendered. The Actor records an opt-in diagnostic for these cases and does not attempt CAPTCHA or access-control bypass. Pagination is best-effort and bounded by maxPages.