# SF Municipal Planning & Permit Notices Scraper (`datapilot/sf-municipal-planning-permit-notices-scraper`) Actor

Search by keywords and extract permit ID, project address, applicant, status, record link, and timestamp. Supports pagination, multiple keywords, duplicate removal, Residential Proxy, retries, and Apify Dataset output.

- **URL**: https://apify.com/datapilot/sf-municipal-planning-permit-notices-scraper.md
- **Developed by:** [Data Pilot](https://apify.com/datapilot) (community)
- **Categories:** Other
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 result scrapes

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

🏙️ **SF Municipal Planning & Permit Notices** Scraper is a powerful Apify Actor designed to discover, track, and collect municipal planning and permit record data directly from San Francisco's official Socrata open data API. This tool provides comprehensive **SF Municipal Planning & Permit Notices** intelligence including permit/record ID, project address, applicant name, record status, and a direct link to the official record for any search keyword. Whether you're building a real estate research dashboard, tracking development activity in a neighborhood, or conducting civic transparency research, the SF Municipal Planning & Permit Notices Scraper delivers actionable **SF Municipal Planning & Permit Notices** insights efficiently.

With direct Socrata API access, keyword-based full-text search, automatic pagination with global deduplication, and reliable Apify Dataset delivery, the SF Municipal Planning & Permit Notices Scraper ensures comprehensive **SF Municipal Planning & Permit Notices** coverage across any list of search keywords. It focuses on key **SF Municipal Planning & Permit Notices** signals including record status, applicant, and address, making it an essential tool for real estate and civic development research.

***

### 📋 Table of Contents

- [Features](#-features)
- [Data Source](#-data-source)
- [How It Works](#-how-it-works)
- [Input](#-input)
- [Output](#-output)
- [Technical Stack](#-technical-stack)
- [Data Fields](#-data-fields-explained)
- [Use Cases](#-use-cases)
- [Quick Start](#-quick-start)
- [Configuration](#-configuration)
- [Performance](#-performance)
- [Important Notes](#-important-notes)
- [License & Legal](#-license--legal)

***

### 🔥 Features

- **Direct Socrata API Access** – Queries San Francisco's official Socrata open data endpoint directly for fast, structured **SF Municipal Planning & Permit Notices** data.
- **Multi-Keyword Search** – Accepts any number of search keywords and scans matching planning/permit records for each.
- **Latest-Records Mode** – Supports an empty keyword to pull the most recent records across the entire dataset.
- **Full-Text Query Support** – Uses Socrata's native `$q` full-text search parameter to match keywords against record content.
- **Automatic Pagination** – Pages through all available results using `$limit`/`$offset` until the configured maximum is reached.
- **Global Deduplication** – Tracks seen record IDs across all keywords to prevent duplicate entries in the output.
- **Retry-Resilient Requests** – Automatically retries on rate limits (429) and auth errors (401/403) with backoff delays.
- **Residential Proxy Support** – Apify residential proxy for reliable **SF Municipal Planning & Permit Notices** access.
- **Configurable Result Cap** – Control how many records are pulled per keyword.
- **Incremental Dataset Push** – Pushes each batch of results to the Dataset as it's collected, not just at the end.

***

### 📊 Data Source

#### **San Francisco Open Data (Socrata) API**

- **Authority**: The City and County of San Francisco's official open data platform (`data.sfgov.org`), via the Socrata Open Data API (SODA)
- **Access Method**: Browser-impersonated HTTP requests via `curl_cffi`, run through an executor for async compatibility
- **Coverage**: Any publicly listed **SF Municipal Planning & Permit Notices** record in the target dataset matching the given keyword
- **Data**: Structured JSON records, including record ID, project address, applicant, and status
- **Access**: Public API, no API key required
- **Update Frequency**: Reflects San Francisco's currently published planning/permit dataset at time of run

***

### ⚙️ How It Works

The SF Municipal Planning & Permit Notices Scraper accepts a list of search keywords (or an empty keyword for the latest records) and queries San Francisco's Socrata API for each, using the platform's native `$q` full-text search parameter. For each keyword, the Actor pages through all available results using `$limit`/`$offset` pagination sorted by record ID descending, retrying automatically on rate limits or transient auth errors with increasing backoff delays. Every record is normalized into a clean entry containing the permit/record ID, address, applicant, and status, with a direct link built to the official record page. A global set of seen record IDs prevents the same record from being saved twice, even across different keywords. Results are pushed to the Apify Dataset incrementally, batch by batch, as they're collected across all keywords.

**Key Processing Steps:**

1. **Input Parsing** – Accept a list of keywords (or a single keyword, or none for latest records) and a max result count
2. **Proxy Setup** – Configure Apify residential proxy for the impersonated HTTP session
3. **Keyword Loop** – Iterate through each **SF Municipal Planning & Permit Notices** search keyword
4. **API Request** – GET the Socrata endpoint with `$limit`, `$offset`, `$order`, and optional `$q` search parameters
5. **Retry Handling** – Back off and retry on rate limits (429) or auth errors (401/403)
6. **Deduplication Check** – Skip any record ID already seen in a previous keyword or page
7. **Record Normalization** – Extract permit ID, address, applicant, and status, and build the record link
8. **Pagination** – Continue requesting subsequent pages using offset increments until results or the max cap are exhausted
9. **Incremental Push** – Push each batch of normalized results to the Dataset as it's ready
10. **Rate Limiting** – Apply a short delay between paginated requests

**Key Benefits:**

- Pull **SF Municipal Planning & Permit Notices** data for any keyword without manually searching the city's data portal
- Track development activity by address, applicant, or project type
- Monitor multiple keywords or areas of interest in a single run
- Feed real estate research pipelines, civic dashboards, or watchdog tools
- Automate recurring **SF Municipal Planning & Permit Notices** checks as new records are published

***

### 📥 Input

The Actor accepts the following input parameters:

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `keywords` | array or string | *(none — latest records)* | One or more search terms to scan for matching **SF Municipal Planning & Permit Notices** records. |
| `maxResults` | integer | `100` | Maximum number of records to collect per keyword. |
| `proxyConfiguration` | object | `{"useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"]}` | Apify proxy configuration for the search session. |

**Example Input:**

```json
{
  "keywords": ["Mission Street", "condominium conversion"],
  "maxResults": 100,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"]
  }
}
```

**Fetch the Latest Records (No Keyword):**

```json
{
  "maxResults": 200
}
```

**Search a Single Keyword:**

```json
{
  "keywords": "accessory dwelling unit"
}
```

***

### 📤 Output

The Actor pushes **SF Municipal Planning & Permit Notices** records with the following structure:

| Field | Type | Description |
|-------|------|-------------|
| `search_keyword` | string | The keyword that matched this result, or `"LATEST_RECORDS"` if none was provided |
| `permit_id` | string | The unique record identifier from San Francisco's planning/permit dataset |
| `address` | string | The project's street address |
| `applicant` | string | Name of the applicant associated with the record |
| `status` | string | Current status of the planning/permit record |
| `link` | string | Direct URL to view the record on San Francisco's open data portal |
| `scraped_at` | string | ISO 8601 timestamp of when the record was scraped |

**Example SF Municipal Planning & Permit Notices Record:**

```json
{
  "search_keyword": "condominium conversion",
  "permit_id": "2026-004521PRJ",
  "address": "1450 Mission Street, San Francisco, CA",
  "applicant": "Example Development LLC",
  "status": "UNDER REVIEW",
  "link": "https://data.sfgov.org/d/qvu5-m3a2?record_id=2026-004521PRJ",
  "scraped_at": "2026-08-16T09:00:00+00:00"
}
```

***

### 🧰 Technical Stack

- **HTTP Client:** `curl_cffi` with Chrome impersonation for reliable access to the Socrata API
- **Concurrency:** `asyncio` with an executor bridge for running the impersonated client asynchronously
- **Query Syntax:** Socrata SODA parameters (`$limit`, `$offset`, `$order`, `$q`) for pagination, sorting, and full-text search
- **Deduplication:** In-memory `set` tracking of seen record IDs across the full run
- **Date Handling:** `datetime` with timezone-aware UTC timestamps
- **Proxy:** Apify Proxy with `RESIDENTIAL` configuration by default
- **Logging:** Apify Actor logging system
- **Platform:** Apify Actor serverless environment

***

### 📊 Data Fields Explained

#### **Record Identity**

- **permit\_id**: The unique identifier for the planning or permit record
- **search\_keyword**: The keyword that surfaced this result, for traceability across multi-keyword runs

#### **Project Details**

- **address**: The street address associated with the project
- **applicant**: The name of the individual or organization that filed the record
- **status**: The record's current review or approval status

#### **Access**

- **link**: Direct link to view the full **SF Municipal Planning & Permit Notices** record on San Francisco's open data portal

***

### 🎯 Use Cases

- **Real Estate Research** – Track planning and permit activity for specific addresses or neighborhoods
- **Development Monitoring** – Watch for new construction, conversion, or renovation filings
- **Journalism & Transparency** – Source primary planning data for local development stories
- **Competitive Intelligence** – Monitor which developers or applicants are active in a given area
- **Academic Research** – Study municipal planning and permitting patterns at scale
- **Community Advocacy** – Stay informed on development proposals relevant to local neighborhoods

***

### 🚀 Quick Start

#### **1. Prepare Input**

Go to Apify Console and enter:

```json
{
  "keywords": ["Mission Street"]
}
```

#### **2. Run the Actor**

Click **Start**. The Actor will:

- Query San Francisco's Socrata API for each keyword (or latest records if none given)
- Page through all matching results, retrying on rate limits automatically
- Deduplicate records across all keywords
- Push batches to the Dataset incrementally as they're collected

#### **3. Monitor Progress**

Console shows:

```
[1/1] Querying Socrata API for: 'Mission Street'
Saved 100 records for 'Mission Street' (Progress: 100/100)
🎉 Scraping complete! Successfully extracted 100 municipal planning records.
```

#### **4. View & Download Results**

- **Results Tab**: All **SF Municipal Planning & Permit Notices** records
- **Export**: JSON, CSV, Excel, or HTML
- **Filter**: By status or applicant
- **API Access**: Available via the Apify API

***

### ⚙️ Configuration

Single keyword search:

```json
{
  "keywords": "accessory dwelling unit"
}
```

Multiple keywords in one run:

```json
{
  "keywords": ["Mission Street", "condominium conversion", "seismic retrofit"],
  "maxResults": 200
}
```

Latest records, no keyword:

```json
{
  "maxResults": 500
}
```

***

### 📈 Performance

#### **Processing Speed**

- One or more paginated requests per keyword, depending on total result volume
- 0.3 second delay between paginated requests to stay within fair-use expectations
- Automatic retry backoff (up to 3 attempts) on rate limits or transient errors

#### **Resource Usage**

- Memory: Low, since no browser instance is launched
- Network: Multiple lightweight JSON requests per keyword, scaling with result volume
- Proxy: One residential proxy tunnel shared across the impersonated session for the run

***

### ⚠️ Important Notes

#### **Legal & Compliance**

- **Fair Use**: Applies pacing delays between paginated requests to the public Socrata API
- **Public Data Only**: Retrieves only publicly published **SF Municipal Planning & Permit Notices** data already indexed by the city's open data portal
- **Legal**: Not legal advice — consult qualified professionals before using this data for compliance-sensitive decisions

#### **Data Quality**

- **Freshness**: Reflects San Francisco's currently published dataset at time of run
- **Completeness**: Field availability depends on what the city has published for each record; some fields may show `"N/A"` or `"UNKNOWN"`
- **Accuracy**: Sourced directly from the official San Francisco Socrata open data API
- **Verification**: Cross-check high-stakes records against the live data.sfgov.org portal

#### **Best Practices**

- Use specific keywords (street names, project types) to keep result volume manageable and relevant
- Omit the keyword to pull the latest records dataset-wide
- Re-run periodically to catch newly published or updated records
- Increase `maxResults` for broader keyword coverage, keeping pagination volume in mind
- Search multiple related keywords in one run to broaden neighborhood or project-type coverage

***

### 📄 License & Legal

**Terms of Use:**

- Use for legitimate research, real estate, and civic transparency purposes
- Respect the City and County of San Francisco's open data Terms of Use
- Use **SF Municipal Planning & Permit Notices** data responsibly and in compliance with applicable laws

**Disclaimer:**
SF Municipal Planning & Permit Notices Scraper is provided as-is for research and analytics purposes. Users are responsible for compliance with San Francisco's open data terms and all applicable laws. This is not legal advice.

***

### ⚖️ SF Municipal Planning & Permit Notices Excellence

This Actor is optimized for **SF Municipal Planning & Permit Notices** research with:

- ✅ Direct, official Socrata API access — no HTML scraping fragility
- ✅ Retry-resilient requests with automatic backoff
- ✅ Global deduplication across multi-keyword runs
- ✅ Full pagination across all matching results
- ✅ Incremental Dataset push
- ✅ Production-ready code

# Actor input Schema

## `keywords` (type: `array`):

List of project names, street addresses, or descriptions to search (leave blank for latest records).

## `maxResults` (type: `integer`):

Maximum number of permit records to collect per search.

## `proxyConfiguration` (type: `object`):

Select Residential Proxy to prevent rate limits and IP blocks.

## Actor input object example

```json
{
  "keywords": [
    "market street"
  ],
  "maxResults": 100,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL"
    ]
  }
}
```

# Actor output Schema

## `crawlResults` (type: `string`):

No description

## `permitsOverview` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "keywords": [
        "market street"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("datapilot/sf-municipal-planning-permit-notices-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "keywords": ["market street"] }

# Run the Actor and wait for it to finish
run = client.actor("datapilot/sf-municipal-planning-permit-notices-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "keywords": [
    "market street"
  ]
}' |
apify call datapilot/sf-municipal-planning-permit-notices-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,datapilot/sf-municipal-planning-permit-notices-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Ck60buf94AmsXnqje/builds/hCgcn2CRdl37er7wB/openapi.json