SF Municipal Planning & Permit Notices Scraper

🏙️ SF Municipal Planning & Permit Notices Scraper is a powerful Apify Actor designed to discover, track, and collect municipal planning and permit record data directly from San Francisco's official Socrata open data API. This tool provides comprehensive SF Municipal Planning & Permit Notices intelligence including permit/record ID, project address, applicant name, record status, and a direct link to the official record for any search keyword. Whether you're building a real estate research dashboard, tracking development activity in a neighborhood, or conducting civic transparency research, the SF Municipal Planning & Permit Notices Scraper delivers actionable SF Municipal Planning & Permit Notices insights efficiently.

With direct Socrata API access, keyword-based full-text search, automatic pagination with global deduplication, and reliable Apify Dataset delivery, the SF Municipal Planning & Permit Notices Scraper ensures comprehensive SF Municipal Planning & Permit Notices coverage across any list of search keywords. It focuses on key SF Municipal Planning & Permit Notices signals including record status, applicant, and address, making it an essential tool for real estate and civic development research.

---

📋 Table of Contents

• Features
• Data Source
• How It Works
• Input
• Output
• Technical Stack
• Data Fields
• Use Cases
• Quick Start
• Configuration
• Performance
• Important Notes
• License & Legal

---

🔥 Features

• Direct Socrata API Access – Queries San Francisco's official Socrata open data endpoint directly for fast, structured SF Municipal Planning & Permit Notices data.
• Multi-Keyword Search – Accepts any number of search keywords and scans matching planning/permit records for each.
• Latest-Records Mode – Supports an empty keyword to pull the most recent records across the entire dataset.
• Full-Text Query Support – Uses Socrata's native $q full-text search parameter to match keywords against record content.
• Automatic Pagination – Pages through all available results using $limit/$offset until the configured maximum is reached.
• Global Deduplication – Tracks seen record IDs across all keywords to prevent duplicate entries in the output.
• Retry-Resilient Requests – Automatically retries on rate limits (429) and auth errors (401/403) with backoff delays.
• Residential Proxy Support – Apify residential proxy for reliable SF Municipal Planning & Permit Notices access.
• Configurable Result Cap – Control how many records are pulled per keyword.
• Incremental Dataset Push – Pushes each batch of results to the Dataset as it's collected, not just at the end.

---

📊 Data Source

San Francisco Open Data (Socrata) API

• Authority: The City and County of San Francisco's official open data platform (data.sfgov.org), via the Socrata Open Data API (SODA)
• Access Method: Browser-impersonated HTTP requests via curl_cffi, run through an executor for async compatibility
• Coverage: Any publicly listed SF Municipal Planning & Permit Notices record in the target dataset matching the given keyword
• Data: Structured JSON records, including record ID, project address, applicant, and status
• Access: Public API, no API key required
• Update Frequency: Reflects San Francisco's currently published planning/permit dataset at time of run

---

⚙️ How It Works

The SF Municipal Planning & Permit Notices Scraper accepts a list of search keywords (or an empty keyword for the latest records) and queries San Francisco's Socrata API for each, using the platform's native $q full-text search parameter. For each keyword, the Actor pages through all available results using $limit/$offset pagination sorted by record ID descending, retrying automatically on rate limits or transient auth errors with increasing backoff delays. Every record is normalized into a clean entry containing the permit/record ID, address, applicant, and status, with a direct link built to the official record page. A global set of seen record IDs prevents the same record from being saved twice, even across different keywords. Results are pushed to the Apify Dataset incrementally, batch by batch, as they're collected across all keywords.

Key Processing Steps:

1. Input Parsing – Accept a list of keywords (or a single keyword, or none for latest records) and a max result count
2. Proxy Setup – Configure Apify residential proxy for the impersonated HTTP session
3. Keyword Loop – Iterate through each SF Municipal Planning & Permit Notices search keyword
4. API Request – GET the Socrata endpoint with $limit, $offset, $order, and optional $q search parameters
5. Retry Handling – Back off and retry on rate limits (429) or auth errors (401/403)
6. Deduplication Check – Skip any record ID already seen in a previous keyword or page
7. Record Normalization – Extract permit ID, address, applicant, and status, and build the record link
8. Pagination – Continue requesting subsequent pages using offset increments until results or the max cap are exhausted
9. Incremental Push – Push each batch of normalized results to the Dataset as it's ready
10. Rate Limiting – Apply a short delay between paginated requests

Key Benefits:

• Pull SF Municipal Planning & Permit Notices data for any keyword without manually searching the city's data portal
• Track development activity by address, applicant, or project type
• Monitor multiple keywords or areas of interest in a single run
• Feed real estate research pipelines, civic dashboards, or watchdog tools
• Automate recurring SF Municipal Planning & Permit Notices checks as new records are published

---

📥 Input

The Actor accepts the following input parameters:

Field	Type	Default	Description
keywords	array or string	(none — latest records)	One or more search terms to scan for matching SF Municipal Planning & Permit Notices records.
maxResults	integer	100	Maximum number of records to collect per keyword.
proxyConfiguration	object	{"useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"]}	Apify proxy configuration for the search session.

Example Input:

{
  "keywords": ["Mission Street", "condominium conversion"],
  "maxResults": 100,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"]
  }
}

Fetch the Latest Records (No Keyword):

{
  "maxResults": 200
}

Search a Single Keyword:

{
  "keywords": "accessory dwelling unit"
}

---

📤 Output

The Actor pushes SF Municipal Planning & Permit Notices records with the following structure:

Field	Type	Description
search_keyword	string	The keyword that matched this result, or "LATEST_RECORDS" if none was provided
permit_id	string	The unique record identifier from San Francisco's planning/permit dataset
address	string	The project's street address
applicant	string	Name of the applicant associated with the record
status	string	Current status of the planning/permit record
link	string	Direct URL to view the record on San Francisco's open data portal
scraped_at	string	ISO 8601 timestamp of when the record was scraped

Example SF Municipal Planning & Permit Notices Record:

{
  "search_keyword": "condominium conversion",
  "permit_id": "2026-004521PRJ",
  "address": "1450 Mission Street, San Francisco, CA",
  "applicant": "Example Development LLC",
  "status": "UNDER REVIEW",
  "link": "https://data.sfgov.org/d/qvu5-m3a2?record_id=2026-004521PRJ",
  "scraped_at": "2026-08-16T09:00:00+00:00"
}

---

🧰 Technical Stack

• HTTP Client: curl_cffi with Chrome impersonation for reliable access to the Socrata API
• Concurrency: asyncio with an executor bridge for running the impersonated client asynchronously
• Query Syntax: Socrata SODA parameters ($limit, $offset, $order, $q) for pagination, sorting, and full-text search
• Deduplication: In-memory set tracking of seen record IDs across the full run
• Date Handling: datetime with timezone-aware UTC timestamps
• Proxy: Apify Proxy with RESIDENTIAL configuration by default
• Logging: Apify Actor logging system
• Platform: Apify Actor serverless environment

---

📊 Data Fields Explained

Record Identity

• permit_id: The unique identifier for the planning or permit record
• search_keyword: The keyword that surfaced this result, for traceability across multi-keyword runs

Project Details

• address: The street address associated with the project
• applicant: The name of the individual or organization that filed the record
• status: The record's current review or approval status

Access

• link: Direct link to view the full SF Municipal Planning & Permit Notices record on San Francisco's open data portal

---

🎯 Use Cases

• Real Estate Research – Track planning and permit activity for specific addresses or neighborhoods
• Development Monitoring – Watch for new construction, conversion, or renovation filings
• Journalism & Transparency – Source primary planning data for local development stories
• Competitive Intelligence – Monitor which developers or applicants are active in a given area
• Academic Research – Study municipal planning and permitting patterns at scale
• Community Advocacy – Stay informed on development proposals relevant to local neighborhoods

---

🚀 Quick Start

1. Prepare Input

Go to Apify Console and enter:

{
  "keywords": ["Mission Street"]
}

2. Run the Actor

Click Start. The Actor will:

• Query San Francisco's Socrata API for each keyword (or latest records if none given)
• Page through all matching results, retrying on rate limits automatically
• Deduplicate records across all keywords
• Push batches to the Dataset incrementally as they're collected

3. Monitor Progress

Console shows:

[1/1] Querying Socrata API for: 'Mission Street'
Saved 100 records for 'Mission Street' (Progress: 100/100)
🎉 Scraping complete! Successfully extracted 100 municipal planning records.

4. View & Download Results

• Results Tab: All SF Municipal Planning & Permit Notices records
• Export: JSON, CSV, Excel, or HTML
• Filter: By status or applicant
• API Access: Available via the Apify API

---

⚙️ Configuration

Single keyword search:

{
  "keywords": "accessory dwelling unit"
}

Multiple keywords in one run:

{
  "keywords": ["Mission Street", "condominium conversion", "seismic retrofit"],
  "maxResults": 200
}

Latest records, no keyword:

{
  "maxResults": 500
}

---

📈 Performance

Processing Speed

• One or more paginated requests per keyword, depending on total result volume
• 0.3 second delay between paginated requests to stay within fair-use expectations
• Automatic retry backoff (up to 3 attempts) on rate limits or transient errors

Resource Usage

• Memory: Low, since no browser instance is launched
• Network: Multiple lightweight JSON requests per keyword, scaling with result volume
• Proxy: One residential proxy tunnel shared across the impersonated session for the run

---

⚠️ Important Notes

Legal & Compliance

• Fair Use: Applies pacing delays between paginated requests to the public Socrata API
• Public Data Only: Retrieves only publicly published SF Municipal Planning & Permit Notices data already indexed by the city's open data portal
• Legal: Not legal advice — consult qualified professionals before using this data for compliance-sensitive decisions

Data Quality

• Freshness: Reflects San Francisco's currently published dataset at time of run
• Completeness: Field availability depends on what the city has published for each record; some fields may show "N/A" or "UNKNOWN"
• Accuracy: Sourced directly from the official San Francisco Socrata open data API
• Verification: Cross-check high-stakes records against the live data.sfgov.org portal

Best Practices

• Use specific keywords (street names, project types) to keep result volume manageable and relevant
• Omit the keyword to pull the latest records dataset-wide
• Re-run periodically to catch newly published or updated records
• Increase maxResults for broader keyword coverage, keeping pagination volume in mind
• Search multiple related keywords in one run to broaden neighborhood or project-type coverage

---

📄 License & Legal

Terms of Use:

• Use for legitimate research, real estate, and civic transparency purposes
• Respect the City and County of San Francisco's open data Terms of Use
• Use SF Municipal Planning & Permit Notices data responsibly and in compliance with applicable laws

Disclaimer: SF Municipal Planning & Permit Notices Scraper is provided as-is for research and analytics purposes. Users are responsible for compliance with San Francisco's open data terms and all applicable laws. This is not legal advice.

---

⚖️ SF Municipal Planning & Permit Notices Excellence

This Actor is optimized for SF Municipal Planning & Permit Notices research with:

• ✅ Direct, official Socrata API access — no HTML scraping fragility
• ✅ Retry-resilient requests with automatic backoff
• ✅ Global deduplication across multi-keyword runs
• ✅ Full pagination across all matching results
• ✅ Incremental Dataset push
• ✅ Production-ready code