All notable changes to the Craigslist Scraper [From $0.50💰] Search, Leads & Housing will be documented in this file.
- Multi-Metro Regional Clustering (
regionPreset): Target high-demand metropolitan clusters in a single crawl run (socal, norcal_bayarea, florida, texas, tri_state_ny, pacific_northwest, midwest_chicago) with automatic balanced metro quota sampling and cross-metro deduplication.
- Custom Multi-Subdomain Targeting (
subdomains): Provide an explicit array of cities or subdomains (e.g. ["miami", "orlando", "tampa"]) to monitor or scrape arbitrary combinations of markets in a single actor execution.
- Real-Time Delta & Hourly Monitor Mode (
onlyNewListings): Stops crawling dynamically as soon as previously scraped listings are reached in the frontier, outputting only net-new postings and charging $0.00 for duplicates. Seamlessly connects to hourly Apify schedules and webhooks.
- Listing Freshness Filter (
maxHours): Discovery-level cutoff that discards listings older than N hours directly from Craigslist SAPI posting timestamps without consuming HTTP requests or detail credits.
- Full Mode as Default (
scrapeDetails: true): Out-of-the-box configuration delivers rich vehicle specifications (VIN, year, make, model, odometer, transmission, title status), housing specs (beds, baths, sqft, laundry, parking, pets), contact details, and full high-resolution image galleries.
- Zero-HTTP Regional Area ID Caching: Pre-populated internal lookup table for all cluster subdomains to guarantee instant 0ms, zero-overhead SAPI discovery queries without redirect penalties.
- Sort Order Filter (
sort): Added user-selectable sorting (date for newest first, priceasc for low-to-high, pricedsc for high-to-low, and rel for relevance) across SAPI queries and direct search URL decomposition.
- Postal / ZIP Code & Search Radius Targeting (
postal, radius): Exposed hyper-local distance radius filtering (1–500 miles around target ZIP code) in input_schema.json.
- Item Condition Filter (
condition): Added 6-tier item condition filtering (10 New, 20 Like New, 30 Excellent, 40 Good, 50 Fair, 60 Salvage) for automotive, electronics, and classified goods.
- Apify Store Category Expansion: Added
AUTOMATION to .actor/actor.json to capture high-volume automation user discovery.
- Storefront SEO & Title Optimization: Refactored actor title to
Craigslist Scraper — Worldwide & Leads to front-load the primary exact-match search keyword, preventing BM25 relevance dilution while retaining value propositions in the short description.
- URL Decomposer Hardening: Automatically extracts
sort and condition query parameters from any pasted Craigslist browser search URL.
- Real-Data Test Coverage Matrix: Expanded
scripts/test_all_inputs.py to 37 automated live scenarios covering sort orders, ZIP radius, and condition filtering with 100% pass rate.
- UnitBytes Central Dashboard Telemetry: Integrated hardened, privacy-compliant
telemetry.py standard library module. Auto-activates ApifyDatasetSink when TELEMETRY_DATASET_ID and TELEMETRY_TOKEN are set in Apify secrets, enabling real-time central fleet monitoring across external user runs.
- Diagnostic Error & Stage Tracking: Logs structured
error_type, error_stage, stop_reason, and HTTP blocker status code counters (challenge_codes.407, 403, 429) with automatic URL and credential masking.
- Input Shape Metrics: Tracks high-level input parameters (
category, subdomain, scrapeDetails, hasImage, postedToday, minPrice, maxPrice) while redacting sensitive search keywords and PII.
- Option A UI Streamlining: Cleanly divided input schema into 4 intuitive sections (
🎯 Target & Search Settings, ⚙️ Filters & Listing Criteria, ⚡ Limits & Extraction Depth, and 🧠 Smart Deduplication & Proxies). Simplified the user experience by keeping hasImage, postedToday, and enableDeduplication toggle, while eliminating technical configuration fields (deduplicationHours, customStoreName) from UI (they maintain smart 24h production defaults).
- Universal Regional & City Search URL Decomposer: Full extraction of
/search/city/..., /search/area/..., and /search/subarea/... paths, along with geographic query parameters (lat, lon, radius/search_distance, postal).
- Dynamic Multi-Region Area Resolution: Enabled dynamic resolution of regional area IDs across all international and regional subdomains (e.g. Canadian cities, rural areas) using Craigslist metadata endpoints.
- Hardened 407 Proxy Fallback: Resolved Apify proxy credit exhaustion errors (
curl 407 CONNECT tunnel failed) by adding an immediate automatic fallback to direct connection and a retry loop in region lookups.
- Form-URL Isolation: Prevented default form values (such as default
subdomain: losangeles or seoul) from overriding or polluting explicit browser URLs pasted into searchUrl.
- User-Facing Clean Logs: Replaced technical internal error traces and cURL codes with actionable, user-friendly tips and direct browser verification URLs.
- Official Apify Store Discounts: Activated automatic tiered volume pricing for Apify subscribers. Paying users on Bronze (Starter), Silver (Scale), and Gold/Platinum (Business/Enterprise) plans automatically receive up to 30% savings on both Search Discovery and Detail Enrichment runs.
- End-to-End Real-Data Test Suite: Added
scripts/test_all_inputs.py exercising 11 real-world input permutations (keywords, metros, housing specs, service leads, subareas, filters, and 24h deduplication).
- Performance & Throughput Benchmarks: Added
scripts/benchmark_200.py for automated latency, memory, and throughput profiling across 200 items.
- Input Validation & Security: Hardened URL routing with strict domain validation and input parameter sanitization.
- Documentation & Presets: Refreshed README with transparent Store Discounts schedule and cleaned formatting.
- Domain Guard & Host Sanitization: Added strict URL verification ensuring only valid Craigslist endpoints are accepted for custom search and direct listing inputs.
- High-Speed Search Discovery Mode ($0.50 / 1k items): Enabled lightweight search discovery (
scrapeDetails: false) returning titles, prices, map coordinates, URLs, and thumbnails at over 48 items/second.
- 24-Hour Intelligent Delta Deduplication: Added
enableDeduplication with a configurable 24-hour cache window (deduplicationHours: 24). Filtered duplicate items cost $0.00—users only pay for newly discovered listings during automated monitoring schedules.
- Expanded Category Support: Added full support for Housing & Apartments (
apa, rea, roo, sub), Jobs (jjj), Gigs (ggg), Local Services (bbb), Resumes (res), and Community (ccc).
- Free Direct Phone & Email Lead Extraction: Automatically parses direct seller phone numbers and email addresses from listing descriptions with $0 additional add-on fees.
- Advanced Subarea URL Parsing: Added automatic decomposition for complex subarea URLs (e.g.
/wst/apa, /lac/cto), query parameters, owner/dealer purveyor filters, and price bounds.
- Anti-Blocking Resilience: Enhanced automated proxy session rotation and intelligent retry handling for high-volume enterprise crawls.
- Deleted Listing Graceful Fallback: Listings removed or returning HTTP 404/410 between search discovery and detail fetch are preserved with basic metadata instead of dropped.
- Clean Logging: Streamlined user-facing run console output with clear status milestones.
- Pay-Per-Event (PPE) Monetization: Pay only for scraped results with $0 server compute and RAM fees.
- 50+ US Metro Support: Built-in dropdown selection covering major US markets.
- Automotive & Classifieds Intelligence: Full extraction of prices, vehicle VINs, mileage, transmission, high-resolution photo galleries (up to 24 images), and geographic coordinates.
- Multi-Format Export: Direct download support for Excel, CSV, JSON, and Apify API integration.
