Pricing

from $0.01 / 1,000 results

Business license leads scraper (US and Canada)

Build local B2B lead lists from official city, county and state business license registers across 18 US and Canadian jurisdictions. Business name, trade name, licence type, activity, address, phone and issue date. Public open data, no API key.

Pricing

from $0.01 / 1,000 results

Rating

0.0

(0)

Developer

Mangudäi

Actor stats

Bookmarked

Total users

Monthly active users

3 days ago

Last modified

BBB Scraper — Ratings, Reviews, Complaints & Leads

memo23/bbb-scraper

Scrape bbb.org for B2B leads — owner name, decoded email, phone, website, social profiles, BBB rating, accreditation, address + coordinates, business type, licenses, hours, plus reviews & complaints. Search by term + location or paste BBB URLs. USA & Canada, no login. JSON or CSV.

Muhamed Didovic

BBB Scraper - Business Leads, Ratings & Accreditation

scrapesage/bbb-scraper

Scrape Better Business Bureau businesses by category & city: name, BBB rating (A+ to F), accreditation, phone, full address, website, email, years in business, complaints, customer reviews & owner contacts. US & Canada B2B lead generation. Export JSON, CSV, Excel.

Scrape Sage

BBB Scraper

haketa/bbb-scraper

Better Business Bureau (BBB) scraper & API: export US & Canada company profiles, BBB ratings & accreditation status, customer reviews & complaints, phone, address, website, category and years in business. B2B lead generation, due diligence and business reputation data — fast, no login.

Haketa

Google Maps Scraper

get-leads/google-maps-scraper---best-value-for-money

Extract business leads from Google Maps with verified emails, phones & social profiles. 50+ niches (HVAC, dentists, lawyers, real estate & more). DNS/SMTP email verification. $0.003/lead — up to 10x cheaper than alternatives. HubSpot/Salesforce export. MCP-ready for AI agents.

Japi Cricket

Psychology Today Scraper - Therapist & Mental Health Leads

scrapesage/psychology-today-scraper

Scrape Psychology Today therapists, psychiatrists & treatment centers by city, state, ZIP or profile URL. Get names, credentials, license, phone, addresses, specialties, modalities, fees, insurance accepted & lead score.

Scrape Sage

Businesses For Sale Scraper

solidcode/businesses-for-sale-scraper

[💰 $0.85 / 1K] Extract business-for-sale listings from BusinessesForSale.com — title, asking price, revenue, cash flow, EBITDA, location, photos, broker contacts, and full descriptions. Search by keyword, country, industry, and price range, or paste BusinessesForSale.com URLs directly.

SolidCode

SimplyHired Jobs Scraper | All In One

fatihtahta/simplyhired-jobs-scraper

Extract structured SimplyHired job listings with titles, salaries, company data, locations, full descriptions and more across US, Canada, and all localized markets. Built for enterprise-grade hiring intelligence, labor market analysis, and automated recruiting or analytics pipelines.

Fatih Tahta

5.0

(1)

Zillow Agents Leads [Scrape All Agents] [PPE]

memo23/zillow-agents-leads-scraper-ppe

Unlock the power of Zillow with automated agent profile scraping. Effortlessly gather key insights and contact data, including emails and phone numbers, from real estate agents.

Muhamed Didovic

387

5.0

(9)

Apartments.com Scraper

haketa/apartments-com-scraper

Apartments.com scraper & API: search US rental listings by city or area and export address, rent price, beds, baths, square feet, amenities, pet policy, photos, contact phone, coordinates and availability. Rental market data and real-estate lead generation — fast, no login.

Haketa

Zillow Agents | Search | Detail pages | with Email Scraper

memo23/apify-zillow-agents-cheerio

Unlock the power of Zillow with automated agent profile scraping – gather key insights and data on real estate agents effortlessly.

Muhamed Didovic

609

5.0

(7)

Website Contact Extractor - Emails & Phones

harvestlab/contact-extractor

Website contact extractor for B2B lead lists and CRM enrichment. Extract emails, phones, social profiles, addresses, contact pages, tech signals, role labels, and optional MCP connector summaries.

Nick

128

{ "actorSpecification": 1, "name": "business-license-leads-scraper", "title": "Business license leads scraper (US and Canada)", "description": "Build local B2B lead lists from official city, county and state business license registers across 18 US and Canadian jurisdictions. Business name, trade name, licence type, activity, address, phone and issue date. Public open data, no API key.", "version": "0.1", "buildTag": "latest", "meta": { "templateId": "python-empty" }, "dockerfile": "../Dockerfile", "input": { "title": "Business license leads scraper input", "description": "Pick the jurisdictions you want, then narrow with a keyword and an issue date. Newest licences come first, so an issue date in the last few weeks gives you fresh new-business leads.", "type": "object", "schemaVersion": 1, "properties": { "sources": { "title": "Jurisdictions", "type": "array", "description": "Official open data portals to pull from. Each one is a government business licence or business tax register.", "editor": "select", "items": { "type": "string", "enum": [ "chicago", "new-york-city", "seattle", "los-angeles", "san-francisco", "new-orleans", "denver", "kansas-city-mo", "berkeley-ca", "marin-county-ca", "norfolk-va", "providence-ri", "gainesville-fl", "everett-wa", "delaware", "calgary", "edmonton", "winnipeg" ], "enumTitles": [ "Chicago, IL", "New York City, NY", "Seattle, WA", "Los Angeles, CA", "San Francisco, CA", "New Orleans, LA", "Denver, CO", "Kansas City, MO", "Berkeley, CA", "Marin County, CA", "Norfolk, VA", "Providence, RI", "Gainesville, FL", "Everett, WA", "Delaware (statewide)", "Calgary, AB", "Edmonton, AB", "Winnipeg, MB" ] }, "prefill": [ "chicago", "new-york-city", "seattle", "los-angeles", "san-francisco" ], "default": [ "chicago", "new-york-city", "seattle", "los-angeles", "san-francisco" ] }, "searchTerm": { "title": "Keyword filter", "type": "string", "description": "Full text search across every column of the source register. Use it for a trade or licence type, for example restaurant, salon, roofing, daycare, contractor, cannabis, tobacco, or for a company name.", "editor": "textfield", "example": "restaurant" }, "issuedSince": { "title": "Issued on or after (YYYY-MM-DD)", "type": "string", "description": "Keep only licences issued on or after this date. Set it to the last 30 days to get newly licensed businesses, which are the strongest leads. Registers without an issue date column ignore this filter.", "editor": "textfield", "example": "2026-06-01" }, "maxItems": { "title": "Max results", "type": "integer", "description": "Maximum number of business records to return across every selected jurisdiction.", "editor": "number", "prefill": 200, "default": 200, "minimum": 1, "maximum": 100000 }, "maxItemsPerSource": { "title": "Max results per jurisdiction", "type": "integer", "description": "Optional per-jurisdiction cap. Leave empty to split the max results evenly across the jurisdictions you picked.", "editor": "number", "sectionCaption": "Advanced options", "sectionDescription": "Ordering and output shape.", "minimum": 1, "maximum": 100000 }, "sortNewestFirst": { "title": "Newest licences first", "type": "boolean", "description": "Sort each register by issue date, newest first, and skip records with a blank or future-dated issue date. Turn this off to read a register in its natural order.", "editor": "checkbox", "default": true }, "includeRawRecord": { "title": "Include the raw source record", "type": "boolean", "description": "Add a raw field to every row holding the untouched record from the source portal, including columns this Actor does not map.", "editor": "checkbox", "default": false } } }, "storages": { "dataset": { "actorSpecification": 1, "title": "Business licence records", "views": { "overview": { "title": "Business licence leads", "transformation": { "fields": [ "businessName", "tradeName", "businessActivity", "licenseType", "licenseStatus", "phone", "email", "fullAddress", "city", "state", "zip", "naicsDescription", "ownerName", "issueDate", "expirationDate", "licenseNumber", "sourceName", "country", "portalUrl" ] }, "display": { "component": "table", "properties": { "businessName": { "label": "Business name", "format": "text" }, "tradeName": { "label": "Trade name / DBA", "format": "text" }, "businessActivity": { "label": "Activity", "format": "text" }, "licenseType": { "label": "Licence type", "format": "text" }, "licenseStatus": { "label": "Status", "format": "text" }, "phone": { "label": "Phone", "format": "text" }, "email": { "label": "Email", "format": "text" }, "fullAddress": { "label": "Address", "format": "text" }, "city": { "label": "City", "format": "text" }, "state": { "label": "State / province", "format": "text" }, "zip": { "label": "ZIP", "format": "text" }, "naicsDescription": { "label": "NAICS", "format": "text" }, "ownerName": { "label": "Owner", "format": "text" }, "issueDate": { "label": "Issued", "format": "text" }, "expirationDate": { "label": "Expires", "format": "text" }, "licenseNumber": { "label": "Licence #", "format": "text" }, "sourceName": { "label": "Jurisdiction", "format": "text" }, "country": { "label": "Country", "format": "text" }, "portalUrl": { "label": "Source portal", "format": "link" } } } } } } }, "output": { "actorOutputSchemaVersion": 1, "title": "Business license leads output", "properties": { "results": { "type": "string", "title": "Results", "template": "{{links.apiDefaultDatasetUrl}}/items" } } } }

1"""Business license leads scraper. 2 3Pulls business license and business tax registry records from official city, 4county and state open data portals (Socrata SODA API) and returns one clean, 5normalized row per licensed business. 6""" 7 8from __future__ import annotations 9 10import asyncio 11import json 12import re 13import urllib.parse 14from datetime import datetime, timezone 15from typing import Any 16 17import httpx 18from apify import Actor 19 20# Date types: "ts" = floating timestamp (2026-07-17T00:00:00.000) 21# "ymd" = plain text YYYYMMDD 22SOURCES = { 23 "chicago": { 24 "label": "Chicago, IL", 25 "domain": "data.cityofchicago.org", 26 "dataset": "r5kz-chrr", 27 "country": "US", 28 "defaultCity": "Chicago", 29 "defaultState": "IL", 30 "dateField": "date_issued", 31 "dateType": "ts", 32 "map": { 33 "businessName": "legal_name", 34 "tradeName": "doing_business_as_name", 35 "licenseNumber": "license_number", 36 "licenseType": "license_description", 37 "licenseStatus": "license_status", 38 "businessActivity": "business_activity", 39 "address": "address", 40 "city": "city", 41 "state": "state", 42 "zip": "zip_code", 43 "latitude": "latitude", 44 "longitude": "longitude", 45 "issueDate": "date_issued", 46 "expirationDate": "expiration_date", 47 }, 48 }, 49 "new-york-city": { 50 "label": "New York City, NY", 51 "domain": "data.cityofnewyork.us", 52 "dataset": "w7w3-xahh", 53 "country": "US", 54 "defaultCity": "New York", 55 "defaultState": "NY", 56 "dateField": "license_creation_date", 57 "dateType": "ts", 58 "addressParts": ["address_building", "address_street_name"], 59 "map": { 60 "businessName": "business_name", 61 "tradeName": "dba_trade_name", 62 "licenseNumber": "license_nbr", 63 "licenseType": "license_type", 64 "licenseStatus": "license_status", 65 "businessActivity": "business_category", 66 "phone": "contact_phone", 67 "city": "address_city", 68 "state": "address_state", 69 "zip": "address_zip", 70 "latitude": "latitude", 71 "longitude": "longitude", 72 "issueDate": "license_creation_date", 73 "expirationDate": "lic_expir_dd", 74 }, 75 }, 76 "seattle": { 77 "label": "Seattle, WA", 78 "domain": "data.seattle.gov", 79 "dataset": "wnbq-64tb", 80 "country": "US", 81 "defaultCity": "Seattle", 82 "defaultState": "WA", 83 "dateField": "license_start_date", 84 "dateType": "ymd", 85 "map": { 86 "businessName": "business_legal_name", 87 "tradeName": "trade_name", 88 "licenseNumber": "city_account_number", 89 "licenseType": "ownership_type", 90 "naicsCode": "naics_code", 91 "naicsDescription": "naics_description", 92 "phone": "business_phone", 93 "address": "street_address", 94 "city": "city", 95 "state": "state", 96 "zip": "zip", 97 "issueDate": "license_start_date", 98 }, 99 }, 100 "los-angeles": { 101 "label": "Los Angeles, CA", 102 "domain": "data.lacity.org", 103 "dataset": "6rrh-rzua", 104 "country": "US", 105 "dateField": "location_start_date", 106 "dateType": "ts", 107 "map": { 108 "businessName": "business_name", 109 "licenseNumber": "location_account", 110 "naicsCode": "naics", 111 "naicsDescription": "primary_naics_description", 112 "address": "street_address", 113 "city": "city", 114 "zip": "zip_code", 115 "issueDate": "location_start_date", 116 }, 117 }, 118 "san-francisco": { 119 "label": "San Francisco, CA", 120 "domain": "data.sfgov.org", 121 "dataset": "g8m3-pdis", 122 "country": "US", 123 "defaultCity": "San Francisco", 124 "defaultState": "CA", 125 "dateField": "location_start_date", 126 "dateType": "ts", 127 "map": { 128 "businessName": "ownership_name", 129 "tradeName": "dba_name", 130 "licenseNumber": "certificate_number", 131 "naicsCode": "self_reported_naics_code", 132 "address": "full_business_address", 133 "city": "city", 134 "state": "state", 135 "zip": "business_zip", 136 "issueDate": "location_start_date", 137 "expirationDate": "location_end_date", 138 }, 139 }, 140 "new-orleans": { 141 "label": "New Orleans, LA", 142 "domain": "data.nola.gov", 143 "dataset": "hjcd-grvu", 144 "country": "US", 145 "dateField": "businessstartdate", 146 "dateType": "ts", 147 "map": { 148 "businessName": "businessname", 149 "licenseNumber": "businesslicensenumber", 150 "businessActivity": "businesstype", 151 "ownerName": "ownername", 152 "phone": "phonenumber", 153 "address": "address", 154 "city": "city", 155 "state": "state", 156 "zip": "zip", 157 "latitude": "latitude", 158 "longitude": "longitude", 159 "issueDate": "businessstartdate", 160 }, 161 }, 162 "gainesville-fl": { 163 "label": "Gainesville, FL", 164 "domain": "data.cityofgainesville.org", 165 "dataset": "hk2b-em59", 166 "country": "US", 167 "defaultCity": "Gainesville", 168 "defaultState": "FL", 169 "dateField": "start_date", 170 "dateType": "ts", 171 "map": { 172 "businessName": "name", 173 "ownerName": "owner", 174 "businessActivity": "business_type", 175 "phone": "business_phone", 176 "email": "email", 177 "address": "physical_address", 178 "city": "physical_city", 179 "state": "physical_state", 180 "latitude": "latitude", 181 "longitude": "longitude", 182 "issueDate": "start_date", 183 }, 184 }, 185 "norfolk-va": { 186 "label": "Norfolk, VA", 187 "domain": "data.norfolk.gov", 188 "dataset": "dpi6-sct5", 189 "country": "US", 190 "defaultCity": "Norfolk", 191 "defaultState": "VA", 192 "dateField": "business_opened_date", 193 "dateType": "ts", 194 "map": { 195 "businessName": "trading_as_name", 196 "ownerName": "primary_owner", 197 "naicsCode": "naics", 198 "address": "location_address", 199 "latitude": "latitude", 200 "longitude": "longitude", 201 "issueDate": "business_opened_date", 202 }, 203 }, 204 "berkeley-ca": { 205 "label": "Berkeley, CA", 206 "domain": "data.cityofberkeley.info", 207 "dataset": "rwnf-bu3w", 208 "country": "US", 209 "defaultCity": "Berkeley", 210 "defaultState": "CA", 211 "dateField": None, 212 "map": { 213 "businessName": "b1_business_name", 214 "tradeName": "dba", 215 "licenseNumber": "recordid", 216 "licenseType": "b1_per_sub_type", 217 "businessActivity": "busdesc", 218 "naicsCode": "naics", 219 "address": "b1_full_address", 220 "city": "b1_situs_city", 221 "state": "b1_situs_state", 222 "zip": "b1_situs_zip", 223 "employeeCount": "employee_num", 224 }, 225 }, 226 "denver": { 227 "label": "Denver, CO", 228 "domain": "data.colorado.gov", 229 "dataset": "s9wt-dsfz", 230 "country": "US", 231 "defaultCity": "Denver", 232 "defaultState": "CO", 233 "dateField": None, 234 "map": { 235 "businessName": "entity_name", 236 "tradeName": "trade_name", 237 "licenseNumber": "bfn", 238 "licenseType": "license_type", 239 "licenseStatus": "license_status", 240 "businessActivity": "license_sub_type", 241 "expirationDate": "expiration_date", 242 }, 243 }, 244 "kansas-city-mo": { 245 "label": "Kansas City, MO", 246 "domain": "data.kcmo.org", 247 "dataset": "pnm4-68wg", 248 "country": "US", 249 "defaultCity": "Kansas City", 250 "defaultState": "MO", 251 "dateField": None, 252 "map": { 253 "businessName": "dba_name", 254 "businessActivity": "business_type", 255 "licenseNumber": "id", 256 "licenseStatus": "valid_license_for", 257 "address": "address", 258 "city": "city", 259 "state": "state", 260 "zip": "zipcode", 261 }, 262 }, 263 "marin-county-ca": { 264 "label": "Marin County, CA", 265 "domain": "data.marincounty.gov", 266 "dataset": "jfy7-shxv", 267 "country": "US", 268 "defaultState": "CA", 269 "dateField": "license_issue_date", 270 "dateType": "ts", 271 "map": { 272 "businessName": "active_business_name", 273 "businessActivity": "business_type", 274 "licenseNumber": "open_data_number", 275 "address": "address", 276 "city": "city_town", 277 "zip": "zipcode", 278 "issueDate": "license_issue_date", 279 }, 280 }, 281 "providence-ri": { 282 "label": "Providence, RI", 283 "domain": "data.providenceri.gov", 284 "dataset": "ui7z-kv69", 285 "country": "US", 286 "defaultCity": "Providence", 287 "defaultState": "RI", 288 "dateField": "permit_date", 289 "dateType": "ts", 290 "map": { 291 "businessName": "dba", 292 "licenseNumber": "bus_lic_id", 293 "licenseType": "primary_lic_desc", 294 "businessActivity": "primary_lic_purpose", 295 "address": "address", 296 "issueDate": "permit_date", 297 "expirationDate": "expiration_date", 298 }, 299 }, 300 "everett-wa": { 301 "label": "Everett, WA", 302 "domain": "data.everettwa.gov", 303 "dataset": "qvfa-2v26", 304 "country": "US", 305 "defaultCity": "Everett", 306 "defaultState": "WA", 307 "dateField": "issuedate", 308 "dateType": "ts", 309 "map": { 310 "businessName": "legalentityname", 311 "tradeName": "primarydba", 312 "licenseNumber": "registration", 313 "naicsCode": "naics", 314 "issueDate": "issuedate", 315 }, 316 }, 317 "delaware": { 318 "label": "Delaware (statewide)", 319 "domain": "data.delaware.gov", 320 "dataset": "5zy2-grhr", 321 "country": "US", 322 "defaultState": "DE", 323 "dateField": "current_license_valid_from", 324 "dateType": "ts", 325 "map": { 326 "businessName": "business_name", 327 "tradeName": "trade_name", 328 "licenseNumber": "license_number", 329 "businessActivity": "category", 330 "address": "address_1", 331 "city": "city", 332 "state": "state", 333 "zip": "zip", 334 "issueDate": "current_license_valid_from", 335 "expirationDate": "current_license_valid_to", 336 }, 337 }, 338 "calgary": { 339 "label": "Calgary, AB", 340 "domain": "data.calgary.ca", 341 "dataset": "vdjc-pybd", 342 "country": "CA", 343 "defaultCity": "Calgary", 344 "defaultState": "AB", 345 "dateField": "first_iss_dt", 346 "dateType": "ts", 347 "map": { 348 "businessName": "tradename", 349 "licenseNumber": "getbusid", 350 "licenseType": "licencetypes", 351 "licenseStatus": "jobstatusdesc", 352 "address": "address", 353 "issueDate": "first_iss_dt", 354 "expirationDate": "exp_dt", 355 }, 356 }, 357 "edmonton": { 358 "label": "Edmonton, AB", 359 "domain": "data.edmonton.ca", 360 "dataset": "qhi4-bdpu", 361 "country": "CA", 362 "defaultCity": "Edmonton", 363 "defaultState": "AB", 364 "dateField": "originalissuedate", 365 "dateType": "ts", 366 "map": { 367 "businessName": "business_name", 368 "licenseType": "licencetype", 369 "businessActivity": "business_licence_category", 370 "licenseNumber": "externalid", 371 "address": "business_address", 372 "issueDate": "originalissuedate", 373 "expirationDate": "expiry_date", 374 }, 375 }, 376 "winnipeg": { 377 "label": "Winnipeg, MB", 378 "domain": "data.winnipeg.ca", 379 "dataset": "d5k3-sfzx", 380 "country": "CA", 381 "defaultCity": "Winnipeg", 382 "defaultState": "MB", 383 "dateField": "issue_date", 384 "dateType": "ts", 385 "map": { 386 "businessName": "trade_name", 387 "licenseType": "folder_type", 388 "businessActivity": "folder_description", 389 "licenseStatus": "status", 390 "address": "address", 391 "issueDate": "issue_date", 392 "expirationDate": "expiry_date", 393 }, 394 }, 395} 396 397SOURCE_KEYS = list(SOURCES.keys()) 398 399PAGE_SIZE = 1000 400MAX_RETRIES = 4 401REQUEST_TIMEOUT = 60.0 402USER_AGENT = "Mozilla/5.0 (compatible; ApifyBusinessLicenseLeads/1.0)" 403 404 405def clean(value: Any) -> str | None: 406 """Normalize a raw Socrata value into a trimmed string, or None.""" 407 if value is None: 408 return None 409 if isinstance(value, bool): 410 return "true" if value else "false" 411 if isinstance(value, (int, float)): 412 return str(value) 413 if isinstance(value, dict): 414 human = value.get("human_address") 415 if human: 416 try: 417 parsed = json.loads(human) 418 joined = " ".join( 419 str(parsed.get(k, "")).strip() 420 for k in ("address", "city", "state", "zip") 421 ).strip() 422 return joined or None 423 except (ValueError, TypeError): 424 return None 425 return None 426 text = str(value).strip() 427 return text or None 428 429 430def normalize_phone(value: Any) -> str | None: 431 """Return a readable North American phone number where possible.""" 432 raw = clean(value) 433 if not raw: 434 return None 435 digits = re.sub(r"\D", "", raw) 436 if not digits or set(digits) == {"0"}: 437 return None 438 if len(digits) == 11 and digits.startswith("1"): 439 digits = digits[1:] 440 if len(digits) == 10: 441 return f"({digits[0:3]}) {digits[3:6]}-{digits[6:]}" 442 return raw 443 444 445def normalize_date(value: Any) -> str | None: 446 """Return an ISO date (YYYY-MM-DD) from the several formats Socrata uses.""" 447 raw = clean(value) 448 if not raw: 449 return None 450 if re.fullmatch(r"\d{8}", raw): 451 return f"{raw[0:4]}-{raw[4:6]}-{raw[6:8]}" 452 match = re.match(r"(\d{4})-(\d{2})-(\d{2})", raw) 453 if match: 454 return f"{match.group(1)}-{match.group(2)}-{match.group(3)}" 455 for fmt in ("%m/%d/%Y", "%d/%m/%Y", "%Y/%m/%d"): 456 try: 457 return datetime.strptime(raw[:10], fmt).strftime("%Y-%m-%d") 458 except ValueError: 459 continue 460 return raw 461 462 463PLACEHOLDER_NAMES = {"new business", "vacant", "unknown", "n/a", "na", "none"} 464 465 466def is_usable(row: dict) -> bool: 467 """Drop rows that carry no identifiable business.""" 468 for key in ("businessName", "tradeName"): 469 name = (row.get(key) or "").strip(" *-").strip() 470 if name and name.lower() not in PLACEHOLDER_NAMES: 471 return True 472 return False 473 474 475def extract_point(record: dict) -> tuple[str | None, str | None]: 476 """Pull latitude and longitude out of a Socrata point or location column.""" 477 for key in ("location", "location_1", "point", "geolocation", "geocoded_point"): 478 value = record.get(key) 479 if isinstance(value, dict): 480 coords = value.get("coordinates") 481 if isinstance(coords, list) and len(coords) >= 2: 482 return str(coords[1]), str(coords[0]) 483 lat = clean(value.get("latitude")) 484 lon = clean(value.get("longitude")) 485 if lat and lon: 486 return lat, lon 487 return None, None 488 489 490def build_where(config: dict, issued_since: str | None, newest_first: bool) -> str | None: 491 """Build a SoQL $where clause for the issue-date filter. 492 493 When sorting newest first the clause also drops records with an empty or 494 future-dated issue date, so the top of the dataset is real, already-issued 495 licenses instead of blanks and pre-registrations. 496 """ 497 field = config.get("dateField") 498 if not field: 499 return None 500 date_type = config.get("dateType", "ts") 501 today = datetime.now(timezone.utc).strftime("%Y-%m-%d") 502 503 def literal(day: str) -> str: 504 return day.replace("-", "") if date_type == "ymd" else f"{day}T23:59:59.999" 505 506 clauses: list[str] = [] 507 if issued_since: 508 low = issued_since.replace("-", "") if date_type == "ymd" else f"{issued_since}T00:00:00.000" 509 clauses.append(f"{field} >= '{low}'") 510 if newest_first: 511 clauses.append(f"{field} IS NOT NULL") 512 clauses.append(f"{field} <= '{literal(today)}'") 513 return " AND ".join(clauses) if clauses else None 514 515 516def map_record(record: dict, source_key: str, config: dict, include_raw: bool) -> dict: 517 """Turn one portal-specific record into the shared output shape.""" 518 field_map = config["map"] 519 row: dict[str, Any] = { 520 "source": source_key, 521 "sourceName": config["label"], 522 "country": config["country"], 523 "businessName": None, 524 "tradeName": None, 525 "licenseNumber": None, 526 "licenseType": None, 527 "licenseStatus": None, 528 "businessActivity": None, 529 "naicsCode": None, 530 "naicsDescription": None, 531 "ownerName": None, 532 "phone": None, 533 "email": None, 534 "employeeCount": None, 535 "address": None, 536 "city": None, 537 "state": None, 538 "zip": None, 539 "latitude": None, 540 "longitude": None, 541 "issueDate": None, 542 "expirationDate": None, 543 } 544 545 for target, column in field_map.items(): 546 value = record.get(column) 547 if target == "phone": 548 row[target] = normalize_phone(value) 549 elif target in ("issueDate", "expirationDate"): 550 row[target] = normalize_date(value) 551 else: 552 row[target] = clean(value) 553 554 parts = config.get("addressParts") 555 if parts and not row.get("address"): 556 joined = " ".join(clean(record.get(p)) or "" for p in parts).strip() 557 row["address"] = joined or None 558 559 if not row.get("latitude") or not row.get("longitude"): 560 lat, lon = extract_point(record) 561 row["latitude"] = row.get("latitude") or lat 562 row["longitude"] = row.get("longitude") or lon 563 564 row["city"] = row.get("city") or config.get("defaultCity") 565 row["state"] = row.get("state") or config.get("defaultState") 566 567 if row.get("city"): 568 row["city"] = row["city"].title() if row["city"].isupper() else row["city"] 569 570 address_bits = [row.get("address"), row.get("city"), row.get("state"), row.get("zip")] 571 row["fullAddress"] = ", ".join(b for b in address_bits if b) or None 572 row["portalUrl"] = f"https://{config['domain']}/resource/{config['dataset']}" 573 row["scrapedAt"] = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") 574 575 if include_raw: 576 row["raw"] = {k: v for k, v in record.items() if not k.startswith(":@")} 577 578 return row 579 580 581async def fetch_page( 582 client: httpx.AsyncClient, 583 config: dict, 584 offset: int, 585 limit: int, 586 search_term: str | None, 587 issued_since: str | None, 588 newest_first: bool, 589) -> list[dict]: 590 """Fetch one page from a Socrata dataset with retries and backoff.""" 591 params: dict[str, Any] = {"$limit": limit, "$offset": offset} 592 if search_term: 593 params["$q"] = search_term 594 where = build_where(config, issued_since, newest_first) 595 if where: 596 params["$where"] = where 597 if newest_first and config.get("dateField"): 598 params["$order"] = f"{config['dateField']} DESC" 599 600 url = f"https://{config['domain']}/resource/{config['dataset']}.json" 601 query = urllib.parse.urlencode(params) 602 603 for attempt in range(MAX_RETRIES): 604 try: 605 response = await client.get(f"{url}?{query}") 606 if response.status_code == 200: 607 return response.json() 608 if response.status_code in (400, 404): 609 Actor.log.warning( 610 "%s returned %s, retrying without ordering: %s", 611 config["label"], 612 response.status_code, 613 response.text[:200], 614 ) 615 params.pop("$order", None) 616 params.pop("$where", None) 617 query = urllib.parse.urlencode(params) 618 continue 619 if response.status_code in (429, 500, 502, 503, 504): 620 await asyncio.sleep(2 ** attempt) 621 continue 622 Actor.log.warning( 623 "%s returned HTTP %s", config["label"], response.status_code 624 ) 625 return [] 626 except (httpx.HTTPError, ValueError) as exc: 627 Actor.log.warning("%s request failed: %s", config["label"], exc) 628 await asyncio.sleep(2 ** attempt) 629 return [] 630 631 632async def scrape_source( 633 client: httpx.AsyncClient, 634 source_key: str, 635 search_term: str | None, 636 issued_since: str | None, 637 newest_first: bool, 638 include_raw: bool, 639 limit_for_source: int, 640) -> list[dict]: 641 """Page through one portal until the per-source cap is reached.""" 642 config = SOURCES[source_key] 643 collected: list[dict] = [] 644 offset = 0 645 646 while len(collected) < limit_for_source: 647 page_size = min(PAGE_SIZE, limit_for_source - len(collected)) 648 records = await fetch_page( 649 client, config, offset, page_size, search_term, issued_since, newest_first 650 ) 651 if not records: 652 break 653 for record in records: 654 row = map_record(record, source_key, config, include_raw) 655 if is_usable(row): 656 collected.append(row) 657 if len(records) < page_size: 658 break 659 offset += len(records) 660 await asyncio.sleep(0.25) 661 662 Actor.log.info("%s: %s records", config["label"], len(collected)) 663 return collected 664 665 666async def main() -> None: 667 async with Actor: 668 actor_input = await Actor.get_input() or {} 669 670 requested = actor_input.get("sources") or [ 671 "chicago", 672 "new-york-city", 673 "seattle", 674 "los-angeles", 675 "san-francisco", 676 ] 677 sources = [s for s in requested if s in SOURCES] 678 unknown = [s for s in requested if s not in SOURCES] 679 if unknown: 680 Actor.log.warning("Ignoring unknown sources: %s", ", ".join(unknown)) 681 if not sources: 682 Actor.log.warning("No valid sources given, falling back to all portals.") 683 sources = SOURCE_KEYS 684 685 search_term = (actor_input.get("searchTerm") or "").strip() or None 686 issued_since = (actor_input.get("issuedSince") or "").strip() or None 687 if issued_since and not re.fullmatch(r"\d{4}-\d{2}-\d{2}", issued_since): 688 Actor.log.warning( 689 "issuedSince must look like YYYY-MM-DD, ignoring %r", issued_since 690 ) 691 issued_since = None 692 693 newest_first = actor_input.get("sortNewestFirst", True) 694 include_raw = actor_input.get("includeRawRecord", False) 695 max_items = int(actor_input.get("maxItems") or 200) 696 per_source = int( 697 actor_input.get("maxItemsPerSource") 698 or max(1, -(-max_items // max(1, len(sources)))) 699 ) 700 701 Actor.log.info( 702 "Scraping %s portal(s): %s", len(sources), ", ".join(sources) 703 ) 704 705 pushed = 0 706 headers = {"User-Agent": USER_AGENT, "Accept": "application/json"} 707 async with httpx.AsyncClient( 708 timeout=REQUEST_TIMEOUT, headers=headers, follow_redirects=True 709 ) as client: 710 for source_key in sources: 711 if pushed >= max_items: 712 break 713 remaining = max_items - pushed 714 rows = await scrape_source( 715 client, 716 source_key, 717 search_term, 718 issued_since, 719 newest_first, 720 include_raw, 721 min(per_source, remaining), 722 ) 723 if rows: 724 await Actor.push_data(rows) 725 pushed += len(rows) 726 727 Actor.log.info("Done. %s business records pushed to the dataset.", pushed) 728 if pushed == 0: 729 Actor.log.warning( 730 "No records matched. Try a broader search term, an earlier " 731 "issuedSince date, or more source portals." 732 )

.git .mise.toml .nvim.lua storage # The rest is copied from https://github.com/github/gitignore/blob/main/Python.gitignore # Byte-compiled / optimized / DLL files __pycache__/ *.py[cod] *$py.class # C extensions *.so # Distribution / packaging .Python build/ develop-eggs/ dist/ downloads/ eggs/ .eggs/ lib/ lib64/ parts/ sdist/ var/ wheels/ share/python-wheels/ *.egg-info/ .installed.cfg *.egg MANIFEST # PyInstaller # Usually these files are written by a python script from a template # before PyInstaller builds the exe, so as to inject date/other infos into it. *.manifest *.spec # Installer logs pip-log.txt pip-delete-this-directory.txt # Unit test / coverage reports htmlcov/ .tox/ .nox/ .coverage .coverage.* .cache nosetests.xml coverage.xml *.cover *.py,cover .hypothesis/ .pytest_cache/ cover/ # Translations *.mo *.pot # Django stuff: *.log local_settings.py db.sqlite3 db.sqlite3-journal # Flask stuff: instance/ .webassets-cache # Scrapy stuff: .scrapy # Sphinx documentation docs/_build/ # PyBuilder .pybuilder/ target/ # Jupyter Notebook .ipynb_checkpoints # IPython profile_default/ ipython_config.py # pyenv # For a library or package, you might want to ignore these files since the code is # intended to run in multiple environments; otherwise, check them in: .python-version # pdm # Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control. #pdm.lock # pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it # in version control. # https://pdm.fming.dev/latest/usage/project/#working-with-version-control .pdm.toml .pdm-python .pdm-build/ # PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm __pypackages__/ # Celery stuff celerybeat-schedule celerybeat.pid # SageMath parsed files *.sage.py # Environments .env .venv env/ venv/ ENV/ env.bak/ venv.bak/ # Spyder project settings .spyderproject .spyproject # Rope project settings .ropeproject # mkdocs documentation /site # mypy .mypy_cache/ .dmypy.json dmypy.json # Pyre type checker .pyre/ # pytype static type analyzer .pytype/ # Cython debug symbols cython_debug/ # PyCharm # JetBrains specific template is maintained in a separate JetBrains.gitignore that can # be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore # and can be added to the global gitignore or merged into this file. For a more nuclear # option (not recommended) you can uncomment the following to ignore the entire idea folder. .idea/ # Visual Studio Code # Ignores the folder created by VS Code when changing workspace settings, doing debugger # configuration, etc. Can be commented out to share Workspace Settings within a team .vscode # Zed editor # Ignores the folder created when setting Project Settings in the Zed editor. Can be commented out # to share Project Settings within a team .zed

.mise.toml .nvim.lua storage # The rest is copied from https://github.com/github/gitignore/blob/main/Python.gitignore # Byte-compiled / optimized / DLL files __pycache__/ *.py[cod] *$py.class # C extensions *.so # Distribution / packaging .Python build/ develop-eggs/ dist/ downloads/ eggs/ .eggs/ lib/ lib64/ parts/ sdist/ var/ wheels/ share/python-wheels/ *.egg-info/ .installed.cfg *.egg MANIFEST # PyInstaller # Usually these files are written by a python script from a template # before PyInstaller builds the exe, so as to inject date/other infos into it. *.manifest *.spec # Installer logs pip-log.txt pip-delete-this-directory.txt # Unit test / coverage reports htmlcov/ .tox/ .nox/ .coverage .coverage.* .cache nosetests.xml coverage.xml *.cover *.py,cover .hypothesis/ .pytest_cache/ cover/ # Translations *.mo *.pot # Django stuff: *.log local_settings.py db.sqlite3 db.sqlite3-journal # Flask stuff: instance/ .webassets-cache # Scrapy stuff: .scrapy # Sphinx documentation docs/_build/ # PyBuilder .pybuilder/ target/ # Jupyter Notebook .ipynb_checkpoints # IPython profile_default/ ipython_config.py # pyenv # For a library or package, you might want to ignore these files since the code is # intended to run in multiple environments; otherwise, check them in: .python-version # pdm # Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control. #pdm.lock # pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it # in version control. # https://pdm.fming.dev/latest/usage/project/#working-with-version-control .pdm.toml .pdm-python .pdm-build/ # PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm __pypackages__/ # Celery stuff celerybeat-schedule celerybeat.pid # SageMath parsed files *.sage.py # Environments .env .venv env/ venv/ ENV/ env.bak/ venv.bak/ # Spyder project settings .spyderproject .spyproject # Rope project settings .ropeproject # mkdocs documentation /site # mypy .mypy_cache/ .dmypy.json dmypy.json # Pyre type checker .pyre/ # pytype static type analyzer .pytype/ # Cython debug symbols cython_debug/ # PyCharm # JetBrains specific template is maintained in a separate JetBrains.gitignore that can # be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore # and can be added to the global gitignore or merged into this file. For a more nuclear # option (not recommended) you can uncomment the following to ignore the entire idea folder. .idea/ # Visual Studio Code # Ignores the folder created by VS Code when changing workspace settings, doing debugger # configuration, etc. Can be commented out to share Workspace Settings within a team .vscode # Zed editor # Ignores the folder created when setting Project Settings in the Zed editor. Can be commented out # to share Project Settings within a team .zed

1# Apify Actors Development Guide 2 3Important: Before you begin, fill in the `generatedBy` property in the meta section of `.actor/actor.json`. Replace it with the tool and model you're currently using, such as "Claude Code with Claude Sonnet 4.5". This helps Apify monitor and improve AGENTS.md for specific AI tools and models. 4 5## What are Apify Actors? 6 7- Actors are serverless programs that run in the cloud. They're inspired by the UNIX philosophy - programs that do one thing well and can be easily combined to build complex systems. 8- Actors are programs packaged as Docker images that run in isolated containers 9 10## Core Concepts 11 12- Accept well-defined JSON input 13- Perform isolated tasks (web scraping, automation, data processing) 14- Produce structured JSON output to datasets and/or store data in key-value stores 15- Can run from seconds to hours or even indefinitely 16- Persist state and can be restarted 17 18## Do 19 20- accept well-defined JSON input and produce structured JSON output 21- use Apify SDK (`apify`) for code running ON Apify platform 22- validate input early with proper error handling and fail gracefully 23- use CheerioCrawler for static HTML content (10x faster than browsers) 24- use PlaywrightCrawler only for JavaScript-heavy sites and dynamic content 25- use router pattern (createCheerioRouter/createPlaywrightRouter) for complex crawls 26- implement retry strategies with exponential backoff for failed requests 27- use proper concurrency settings (HTTP: 10-50, Browser: 1-5) 28- set sensible defaults in `.actor/input_schema.json` for all optional fields 29- set up output schema in `.actor/output_schema.json` 30- clean and validate data before pushing to dataset 31- use semantic CSS selectors and fallback strategies for missing elements 32- respect robots.txt, ToS, and implement rate limiting with delays 33- check which tools (cheerio/playwright/crawlee) are installed before applying guidance 34- use `Actor.log` for logging (censors sensitive data) 35- implement readiness probe handler for standby Actors 36- handle the `aborting` event to gracefully shut down when Actor is stopped 37 38## Don't 39 40- do not rely on `Dataset.getInfo()` for final counts on Cloud platform 41- do not use browser crawlers when HTTP/Cheerio works (massive performance gains with HTTP) 42- do not hard code values that should be in input schema or environment variables 43- do not skip input validation or error handling 44- do not overload servers - use appropriate concurrency and delays 45- do not scrape prohibited content or ignore Terms of Service 46- do not store personal/sensitive data unless explicitly permitted 47- do not use deprecated options like `requestHandlerTimeoutMillis` on CheerioCrawler (v3.x) 48- do not use `additionalHttpHeaders` - use `preNavigationHooks` instead 49- do not assume that local storage is persistent or automatically synced to Apify Console - when running locally with `apify run`, the `storage/` directory is local-only and is NOT pushed to the Cloud 50- do not disable standby mode (`usesStandbyMode: false`) without explicit permission 51 52## Logging 53 54- **ALWAYS use `Actor.log` for logging** - This logger contains critical security logic including censoring sensitive data (Apify tokens, API keys, credentials) to prevent accidental exposure in logs 55 56### Available Log Levels 57 58The Apify Actor logger provides the following methods for logging: 59 60- `Actor.log.debug()` - Debug level logs (detailed diagnostic information) 61- `Actor.log.info()` - Info level logs (general informational messages) 62- `Actor.log.warning()` - Warning level logs (warning messages for potentially problematic situations) 63- `Actor.log.error()` - Error level logs (error messages for failures) 64- `Actor.log.exception()` - Exception level logs (for exceptions with stack traces) 65 66**Best practices:** 67 68- Use `Actor.log.debug()` for detailed operation-level diagnostics (inside functions) 69- Use `Actor.log.info()` for general informational messages (API requests, successful operations) 70- Use `Actor.log.warning()` for potentially problematic situations (validation failures, unexpected states) 71- Use `Actor.log.error()` for actual errors and failures 72- Use `Actor.log.exception()` for caught exceptions with stack traces 73 74## Graceful Abort Handling 75 76Handle the `aborting` event to terminate the Actor quickly when stopped by user or platform, minimizing costs especially for PPU/PPE+U billing. 77 78```python 79import asyncio 80 81async def on_aborting() -> None: 82 # Persist any state, do any cleanup you need, and terminate the Actor using `await Actor.exit()` explicitly as soon as possible 83 # This will help ensure that the Actor is doing best effort to honor any potential limits on costs of a single run set by the user 84 # Wait 1 second to allow Crawlee/SDK state persistence operations to complete 85 # This is a temporary workaround until SDK implements proper state persistence in the aborting event 86 await asyncio.sleep(1) 87 await Actor.exit() 88 89Actor.on('aborting', on_aborting) 90``` 91 92## Standby Mode 93 94- **NEVER disable standby mode (`usesStandbyMode: false`) in `.actor/actor.json` without explicit permission** - Actor Standby mode solves this problem by letting you have the Actor ready in the background, waiting for the incoming HTTP requests. In a sense, the Actor behaves like a real-time web server or standard API server instead of running the logic once to process everything in batch. Always keep `usesStandbyMode: true` unless there is a specific documented reason to disable it 95- **ALWAYS implement readiness probe handler for standby Actors** - Handle the `x-apify-container-server-readiness-probe` header at GET / endpoint to ensure proper Actor lifecycle management 96 97You can recognize a standby Actor by checking the `usesStandbyMode` property in `.actor/actor.json`. Only implement the readiness probe if this property is set to `true`. 98 99### Readiness Probe Implementation Example 100 101```python 102# Apify standby readiness probe 103from http.server import SimpleHTTPRequestHandler 104 105class GetHandler(SimpleHTTPRequestHandler): 106 def do_GET(self): 107 # Handle Apify standby readiness probe 108 if 'x-apify-container-server-readiness-probe' in self.headers: 109 self.send_response(200) 110 self.end_headers() 111 self.wfile.write(b'Readiness probe OK') 112 return 113 114 self.send_response(200) 115 self.end_headers() 116 self.wfile.write(b'Actor is ready') 117``` 118 119Key points: 120 121- Detect the `x-apify-container-server-readiness-probe` header in incoming requests 122- Respond with HTTP 200 status code for both readiness probe and normal requests 123- This enables proper Actor lifecycle management in standby mode 124 125## Commands 126 127```bash 128# Bootstrap & local development 129apify create [name] # Create new Actor project from a template 130apify init # Initialize Actor in current directory 131apify run # Run Actor locally with simulated platform env 132apify run --purge # Run after clearing previous local storage 133apify validate-schema # Validate .actor/input_schema.json 134 135# Authentication & account 136apify login # Authenticate account (token stored in ~/.apify) 137apify logout # Remove stored credentials 138apify info # Print currently authenticated account info 139 140# Deployment & remote execution 141apify push # Deploy Actor to platform per .actor/actor.json 142apify pull <actor> # Download Actor code from the platform 143apify call <actor> # Execute Actor remotely on the platform 144apify actors build <actor> # Create a new build of an Actor 145apify runs ls # List recent runs 146 147# Discovery (search Apify Store for community Actors) 148apify actors search "<query>" --user-agent <your-agent-name> 149apify actors info <actor> # Get details about a specific Actor 150 151# Secrets (referenced from actor.json via "@mySecret") 152apify secrets add <name> <value> # Store a secret locally; uploaded on push 153apify secrets ls # List stored secret keys 154 155# Direct API access 156apify api <endpoint> # Send an authenticated HTTP request to Apify API 157 158# Help 159apify help # List all commands 160apify <command> --help # Get help for a specific command 161``` 162 163Note: If no dedicated Actor exists for your target, search Apify Store for community options with `apify actors search "<query>" --user-agent <your-agent-name>` before building from scratch. 164 165Tip: Inside a running Actor, prefer the SDK (`Actor.get_input()`, `Actor.push_data()`, `Actor.set_value()`) over the equivalent `apify actor` runtime subcommands. 166 167## Apify Platform Environment 168 169When the Actor runs on the Apify platform, the API token is automatically available via the `APIFY_TOKEN` environment variable (note: the variable is `APIFY_TOKEN`, not `APIFY_API_TOKEN`). The Apify SDK reads it automatically, so you do not need to pass it explicitly. Locally, run `apify login` once and the SDK will use your stored credentials. 170 171## Safety and Permissions 172 173Allowed without prompt: 174 175- read files with `Actor.get_value()` 176- push data with `Actor.push_data()` 177- set values with `Actor.set_value()` 178- enqueue requests to RequestQueue 179- run locally with `apify run` 180 181Ask first: 182 183- npm/pip package installations 184- apify push (deployment to cloud) 185- proxy configuration changes (requires paid plan) 186- Dockerfile changes affecting builds 187- deleting datasets or key-value stores 188 189## Project Structure 190 191.actor/ 192├── actor.json # Actor config: name, version, env vars, runtime settings 193├── input_schema.json # Input validation & Console form definition 194└── output_schema.json # Specifies where an Actor stores its output 195src/ 196└── main.js # Actor entry point and orchestrator 197storage/ # Local-only storage for development (NOT synced to Cloud) 198├── datasets/ # Output items (JSON objects) 199├── key_value_stores/ # Files, config, INPUT 200└── request_queues/ # Pending crawl requests 201Dockerfile # Container image definition 202AGENTS.md # AI agent instructions (this file) 203 204## Local vs Cloud Storage 205 206When running locally with `apify run`, the Apify SDK emulates Cloud storage APIs using the local `storage/` directory. This local storage behaves differently from Cloud storage: 207 208- **Local storage is NOT persistent** - The `storage/` directory is meant for local development and testing only. Data stored there (datasets, key-value stores, request queues) exists only on your local disk. 209- **Local storage is NOT automatically pushed to Apify Console** - Running `apify run` does not upload any storage data to the Apify platform. The data stays local. 210- **Each local run may overwrite previous data** - The local `storage/` directory is reused between runs, but this is local-only behavior, not Cloud persistence. 211- **Cloud storage only works when running on Apify platform** - After deploying with `apify push` and running the Actor in the Cloud, storage calls (`Actor.push_data()`, `Actor.set_value()`, etc.) interact with real Apify Cloud storage, which is then visible in the Apify Console. 212- **To verify Actor output, deploy and run in Cloud** - Do not rely on local `storage/` contents as proof that data will appear in the Apify Console. Always test by deploying (`apify push`) and running the Actor on the platform. 213 214## Actor Input Schema 215 216The input schema defines the input parameters for an Actor. It's a JSON object comprising various field types supported by the Apify platform. 217 218### Structure 219 220```json 221{ 222 "title": "<INPUT-SCHEMA-TITLE>", 223 "type": "object", 224 "schemaVersion": 1, 225 "properties": {/* define input fields here */}, 226 "required": [] 227} 228``` 229 230### Example 231 232```json 233{ 234 "title": "E-commerce Product Scraper Input", 235 "type": "object", 236 "schemaVersion": 1, 237 "properties": { 238 "startUrls": { 239 "title": "Start URLs", 240 "type": "array", 241 "description": "URLs to start scraping from (category pages or product pages)", 242 "editor": "requestListSources", 243 "default": [{ "url": "https://example.com/category" }], 244 "prefill": [{ "url": "https://example.com/category" }] 245 }, 246 "followVariants": { 247 "title": "Follow Product Variants", 248 "type": "boolean", 249 "description": "Whether to scrape product variants (different colors, sizes)", 250 "default": true 251 }, 252 "maxRequestsPerCrawl": { 253 "title": "Max Requests per Crawl", 254 "type": "integer", 255 "description": "Maximum number of pages to scrape (0 = unlimited)", 256 "default": 1000, 257 "minimum": 0 258 }, 259 "proxyConfiguration": { 260 "title": "Proxy Configuration", 261 "type": "object", 262 "description": "Proxy settings for anti-bot protection", 263 "editor": "proxy", 264 "default": { "useApifyProxy": false } 265 }, 266 "locale": { 267 "title": "Locale", 268 "type": "string", 269 "description": "Language/country code for localized content", 270 "default": "cs", 271 "enum": ["cs", "en", "de", "sk"], 272 "enumTitles": ["Czech", "English", "German", "Slovak"] 273 } 274 }, 275 "required": ["startUrls"] 276} 277``` 278 279## Actor Output Schema 280 281The Actor output schema builds upon the schemas for the dataset and key-value store. It specifies where an Actor stores its output and defines templates for accessing that output. Apify Console uses these output definitions to display run results. 282 283### Structure 284 285```json 286{ 287 "actorOutputSchemaVersion": 1, 288 "title": "<OUTPUT-SCHEMA-TITLE>", 289 "properties": {/* define your outputs here */} 290} 291``` 292 293### Example 294 295```json 296{ 297 "actorOutputSchemaVersion": 1, 298 "title": "Output schema of the files scraper", 299 "properties": { 300 "files": { 301 "type": "string", 302 "title": "Files", 303 "template": "{{links.apiDefaultKeyValueStoreUrl}}/keys" 304 }, 305 "dataset": { 306 "type": "string", 307 "title": "Dataset", 308 "template": "{{links.apiDefaultDatasetUrl}}/items" 309 } 310 } 311} 312``` 313 314### Output Schema Template Variables 315 316- `links` (object) - Contains quick links to most commonly used URLs 317- `links.publicRunUrl` (string) - Public run url in format `https://console.apify.com/view/runs/:runId` 318- `links.consoleRunUrl` (string) - Console run url in format `https://console.apify.com/actors/runs/:runId` 319- `links.apiRunUrl` (string) - API run url in format `https://api.apify.com/v2/actor-runs/:runId` 320- `links.apiDefaultDatasetUrl` (string) - API url of default dataset in format `https://api.apify.com/v2/datasets/:defaultDatasetId` 321- `links.apiDefaultKeyValueStoreUrl` (string) - API url of default key-value store in format `https://api.apify.com/v2/key-value-stores/:defaultKeyValueStoreId` 322- `links.containerRunUrl` (string) - URL of a webserver running inside the run in format `https://<containerId>.runs.apify.net/` 323- `run` (object) - Contains information about the run same as it is returned from the `GET Run` API endpoint 324- `run.defaultDatasetId` (string) - ID of the default dataset 325- `run.defaultKeyValueStoreId` (string) - ID of the default key-value store 326 327## Dataset Schema Specification 328 329The dataset schema defines how your Actor's output data is structured, transformed, and displayed in the Output tab in the Apify Console. 330 331### Example 332 333Consider an example Actor that calls `Actor.pushData()` to store data into dataset: 334 335```python 336# Dataset push example (Python) 337import asyncio 338from datetime import datetime 339from apify import Actor 340 341async def main(): 342 await Actor.init() 343 344 # Actor code 345 await Actor.push_data({ 346 'numericField': 10, 347 'pictureUrl': 'https://www.google.com/images/branding/googlelogo/2x/googlelogo_color_92x30dp.png', 348 'linkUrl': 'https://google.com', 349 'textField': 'Google', 350 'booleanField': True, 351 'dateField': datetime.now().isoformat(), 352 'arrayField': ['#hello', '#world'], 353 'objectField': {}, 354 }) 355 356 # Exit successfully 357 await Actor.exit() 358 359if __name__ == '__main__': 360 asyncio.run(main()) 361``` 362 363To set up the Actor's output tab UI, reference a dataset schema file in `.actor/actor.json`: 364 365```json 366{ 367 "actorSpecification": 1, 368 "name": "book-library-scraper", 369 "title": "Book Library Scraper", 370 "version": "1.0.0", 371 "storages": { 372 "dataset": "./dataset_schema.json" 373 } 374} 375``` 376 377Then create the dataset schema in `.actor/dataset_schema.json`: 378 379```json 380{ 381 "actorSpecification": 1, 382 "fields": {}, 383 "views": { 384 "overview": { 385 "title": "Overview", 386 "transformation": { 387 "fields": [ 388 "pictureUrl", 389 "linkUrl", 390 "textField", 391 "booleanField", 392 "arrayField", 393 "objectField", 394 "dateField", 395 "numericField" 396 ] 397 }, 398 "display": { 399 "component": "table", 400 "properties": { 401 "pictureUrl": { 402 "label": "Image", 403 "format": "image" 404 }, 405 "linkUrl": { 406 "label": "Link", 407 "format": "link" 408 }, 409 "textField": { 410 "label": "Text", 411 "format": "text" 412 }, 413 "booleanField": { 414 "label": "Boolean", 415 "format": "boolean" 416 }, 417 "arrayField": { 418 "label": "Array", 419 "format": "array" 420 }, 421 "objectField": { 422 "label": "Object", 423 "format": "object" 424 }, 425 "dateField": { 426 "label": "Date", 427 "format": "date" 428 }, 429 "numericField": { 430 "label": "Number", 431 "format": "number" 432 } 433 } 434 } 435 } 436 } 437} 438``` 439 440### Structure 441 442```json 443{ 444 "actorSpecification": 1, 445 "fields": {}, 446 "views": { 447 "<VIEW_NAME>": { 448 "title": "string (required)", 449 "description": "string (optional)", 450 "transformation": { 451 "fields": ["string (required)"], 452 "unwind": ["string (optional)"], 453 "flatten": ["string (optional)"], 454 "omit": ["string (optional)"], 455 "limit": "integer (optional)", 456 "desc": "boolean (optional)" 457 }, 458 "display": { 459 "component": "table (required)", 460 "properties": { 461 "<FIELD_NAME>": { 462 "label": "string (optional)", 463 "format": "text|number|date|link|boolean|image|array|object (optional)" 464 } 465 } 466 } 467 } 468 } 469} 470``` 471 472**Dataset Schema Properties:** 473 474- `actorSpecification` (integer, required) - Specifies the version of dataset schema structure document (currently only version 1) 475- `fields` (JSONSchema object, required) - Schema of one dataset object (use JsonSchema Draft 2020-12 or compatible) 476- `views` (DatasetView object, required) - Object with API and UI views description 477 478**DatasetView Properties:** 479 480- `title` (string, required) - Visible in UI Output tab and API 481- `description` (string, optional) - Only available in API response 482- `transformation` (ViewTransformation object, required) - Data transformation applied when loading from Dataset API 483- `display` (ViewDisplay object, required) - Output tab UI visualization definition 484 485**ViewTransformation Properties:** 486 487- `fields` (string[], required) - Fields to present in output (order matches column order) 488- `unwind` (string[], optional) - Deconstructs nested children into parent object 489- `flatten` (string[], optional) - Transforms nested object into flat structure 490- `omit` (string[], optional) - Removes specified fields from output 491- `limit` (integer, optional) - Maximum number of results (default: all) 492- `desc` (boolean, optional) - Sort order (true = newest first) 493 494**ViewDisplay Properties:** 495 496- `component` (string, required) - Only `table` is available 497- `properties` (Object, optional) - Keys matching `transformation.fields` with ViewDisplayProperty values 498 499**ViewDisplayProperty Properties:** 500 501- `label` (string, optional) - Table column header 502- `format` (string, optional) - One of: `text`, `number`, `date`, `link`, `boolean`, `image`, `array`, `object` 503 504## Key-Value Store Schema Specification 505 506The key-value store schema organizes keys into logical groups called collections for easier data management. 507 508### Example 509 510Consider an example Actor that calls `Actor.setValue()` to save records into the key-value store: 511 512```python 513# Key-Value Store set example (Python) 514import asyncio 515from apify import Actor 516 517async def main(): 518 await Actor.init() 519 520 # Actor code 521 await Actor.set_value('document-1', 'my text data', content_type='text/plain') 522 523 image_id = '123' # example placeholder 524 image_buffer = b'...' # bytes buffer with image data 525 await Actor.set_value(f'image-{image_id}', image_buffer, content_type='image/jpeg') 526 527 # Exit successfully 528 await Actor.exit() 529 530if __name__ == '__main__': 531 asyncio.run(main()) 532``` 533 534To configure the key-value store schema, reference a schema file in `.actor/actor.json`: 535 536```json 537{ 538 "actorSpecification": 1, 539 "name": "data-collector", 540 "title": "Data Collector", 541 "version": "1.0.0", 542 "storages": { 543 "keyValueStore": "./key_value_store_schema.json" 544 } 545} 546``` 547 548Then create the key-value store schema in `.actor/key_value_store_schema.json`: 549 550```json 551{ 552 "actorKeyValueStoreSchemaVersion": 1, 553 "title": "Key-Value Store Schema", 554 "collections": { 555 "documents": { 556 "title": "Documents", 557 "description": "Text documents stored by the Actor", 558 "keyPrefix": "document-" 559 }, 560 "images": { 561 "title": "Images", 562 "description": "Images stored by the Actor", 563 "keyPrefix": "image-", 564 "contentTypes": ["image/jpeg"] 565 } 566 } 567} 568``` 569 570### Structure 571 572```json 573{ 574 "actorKeyValueStoreSchemaVersion": 1, 575 "title": "string (required)", 576 "description": "string (optional)", 577 "collections": { 578 "<COLLECTION_NAME>": { 579 "title": "string (required)", 580 "description": "string (optional)", 581 "key": "string (conditional - use key OR keyPrefix)", 582 "keyPrefix": "string (conditional - use key OR keyPrefix)", 583 "contentTypes": ["string (optional)"], 584 "jsonSchema": "object (optional)" 585 } 586 } 587} 588``` 589 590**Key-Value Store Schema Properties:** 591 592- `actorKeyValueStoreSchemaVersion` (integer, required) - Version of key-value store schema structure document (currently only version 1) 593- `title` (string, required) - Title of the schema 594- `description` (string, optional) - Description of the schema 595- `collections` (Object, required) - Object where each key is a collection ID and value is a Collection object 596 597**Collection Properties:** 598 599- `title` (string, required) - Collection title shown in UI tabs 600- `description` (string, optional) - Description appearing in UI tooltips 601- `key` (string, conditional\*) - Single specific key for this collection 602- `keyPrefix` (string, conditional\*) - Prefix for keys included in this collection 603- `contentTypes` (string[], optional) - Allowed content types for validation 604- `jsonSchema` (object, optional) - JSON Schema Draft 07 format for `application/json` content type validation 605 606\*Either `key` or `keyPrefix` must be specified for each collection, but not both. 607 608## Actor README 609 610**Always generate a README.md file as part of Actor development.** The README is the Actor's public landing page on Apify Store - it serves as SEO, first impression, documentation, and support page combined. 611 612### Required: Generate README automatically 613 614When building an Actor, always create a `README.md` in the project root. Do not wait for the user to ask for it. The README is a critical part of a complete Actor. 615 616### README structure 617 618Write in Markdown. Use H2 (`##`) for main sections (these become the table of contents) and H3 (`###`) for subsections. Do not use H1 - the Actor name is automatically the H1. Aim for at least 300 words. 619 620Include these sections in order: 621 6221. **What does [Actor name] do?** - 2-3 sentences explaining what it does, what data it extracts, and how to try it. Link to the target website. Mention Apify platform advantages (API access, scheduling, integrations, proxy rotation, monitoring). 6232. **Why use [Actor name]?** - Business use cases and benefits. 6243. **How to use [Actor name]** - Numbered step-by-step tutorial. Keep it simple and reassuring. 6254. **Input** - Describe input fields. Reference the Input tab. Optionally include a screenshot or JSON example of the input schema. 6265. **Output** - Show a simplified JSON output example. Mention "You can download the dataset in various formats such as JSON, HTML, CSV, or Excel." 6276. **Data table** - If the Actor extracts data, include a table of the main data fields it outputs. 6287. **Pricing / Cost estimation** - Set expectations on cost. Mention free tier limits if applicable. Frame as "How much does it cost to scrape [target site]?" 6298. **Tips or Advanced options** - How to optimize runs, limit compute units, improve speed or accuracy. 6309. **FAQ, disclaimers, and support** - Legality disclaimer for scrapers, known limitations, link to Issues tab for feedback, mention custom solution availability. 631 632### README best practices 633 634- Write SEO-friendly headings with relevant keywords (e.g., "How to scrape [site] data" not just "Tutorial") 635- Bold the most important words in the intro 636- The first 25% of the README matters most - front-load the value proposition 637- Match the tone to the target audience: simple language for no-code users, technical details for developers 638- Include a JSON output example showing 1-2 representative items 639- Reference these top Actors for README best practices: https://apify.com/apify/instagram-scraper and https://apify.com/compass/crawler-google-places 640- Embed YouTube video URLs on their own line (Apify Console auto-renders them) 641- Use HTML for image sizing if needed; CSS is not supported 642 643## MCP Tools 644 645### Apify MCP 646 647If the Apify MCP server is configured, use these tools for documentation: 648 649- `search-apify-docs` - Search documentation 650- `fetch-apify-docs` - Get full doc pages 651 652Otherwise, reference: `@https://mcp.apify.com/` 653 654### Playwright MCP (debugging) 655 656The Playwright MCP server is a useful tool for debugging Actors that interact with the web - it lets the agent drive a real browser to inspect pages, capture selectors, and reproduce issues. 657 658Install with the Claude Code CLI: 659 660```bash 661claude mcp add playwright npx @playwright/mcp@latest 662``` 663 664Or add it manually to your MCP config: 665 666```json 667{ 668 "mcpServers": { 669 "playwright": { 670 "command": "npx", 671 "args": ["@playwright/mcp@latest"] 672 } 673 } 674} 675``` 676 677## Resources 678 679- [docs.apify.com/llms.txt](https://docs.apify.com/llms.txt) - Quick reference 680- [docs.apify.com/llms-full.txt](https://docs.apify.com/llms-full.txt) - Complete docs 681- [crawlee.dev](https://crawlee.dev) - Crawlee documentation 682- [whitepaper.actor](https://raw.githubusercontent.com/apify/actor-whitepaper/refs/heads/master/README.md) - Complete Actor specification

# First, specify the base Docker image. # You can see the Docker images from Apify at https://hub.docker.com/r/apify/. # You can also use any other image from Docker Hub. FROM apify/actor-python:3.14 USER myuser # Second, copy just requirements.txt into the Actor image, # since it should be the only file that affects the dependency install in the next step, # in order to speed up the build COPY --chown=myuser:myuser requirements.txt ./ # Install the packages specified in requirements.txt, # Print the installed Python version, pip version # and all installed packages with their versions for debugging RUN echo "Python version:" \ && python --version \ && echo "Pip version:" \ && pip --version \ && echo "Installing dependencies:" \ && pip install -r requirements.txt \ && echo "All installed Python packages:" \ && pip freeze # Next, copy the remaining files and directories with the source code. # Since we do this after installing the dependencies, quick build will be really fast # for most source file changes. COPY --chown=myuser:myuser . ./ # Use compileall to ensure the runnability of the Actor Python code. RUN python -m compileall -q my_actor/ # Specify how to launch the source code of your Actor. CMD ["python", "-m", "my_actor"]

Business license leads scraper (US and Canada)

BBB Scraper — Ratings, Reviews, Complaints & Leads