# FormerGov Directory Scraper (`rl1987/formergov-scraper`) Actor Scrapes the formergov.com directory of former government and military professionals, with contact info (LinkedIn, website, email) and full advanced-search filtering. - **URL**: https://apify.com/rl1987/formergov-scraper.md - **Developed by:** [R.L.](https://apify.com/rl1987) (community) - **Categories:** Lead generation - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $1.50 / 1,000 profile scrapeds This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## FormerGov Directory Scraper **Extract former government and military professionals from the [FormerGov](https://formergov.com) directory — with their contact info (LinkedIn profile and website) and full professional background.** Mirror the site's advanced search with structured filters, or pull specific profiles by username. Built with Scrapy on the Apify platform, so you get scheduling, a versioned API, integrations, proxy rotation, and run monitoring out of the box. ### What does FormerGov Directory Scraper do? [FormerGov](https://formergov.com) is an apolitical directory that connects former federal, state, local, tribal, and foreign government and military professionals with people who need their expertise. This Actor turns that directory into structured data: for every matching person it returns their name, headline, location, employment history, sectors, practice areas, and — most importantly — their **contact points**: LinkedIn profile URL and personal/employer website. It talks directly to FormerGov's public JSON API, so it is fast and reliable. The same advanced-search filters available on the website (keyword, practice areas, sectors, functions, jurisdiction, position type, agency, location, and more) are exposed as Actor input, so you can target exactly the people you care about. ### Why use FormerGov Directory Scraper? - **Lead generation & business development** — build targeted lists of former officials by practice area, sector, or agency. - **Recruiting & executive search** — find candidates with specific government or military backgrounds. - **Expert sourcing** — locate expert witnesses, board advisors, speakers, or media commentators. - **Research & due diligence** — map the post-government careers of professionals in a field. - **CRM enrichment** — append LinkedIn and website contact points to existing records. ### How to use FormerGov Directory Scraper 1. On [formergov.com/directory](https://formergov.com/directory), enter a keyword and/or use **Advanced Search** to pick the filters you want (search type, practice areas, sectors, jurisdiction, …), then click **SEARCH**. The address bar now holds your search as a URL (e.g. `…/directory?type=combined&text=OSINT`). 2. Copy that URL from your browser's address bar. 3. Open the Actor in Apify Console, go to the **Input** tab, and paste that URL into **FormerGov URLs**. 4. (Optional) Set **Max profiles** to cap the run, and configure **Proxy**. 5. Click **Start**. When the run finishes, open the **Output** tab and export the data as JSON, CSV, Excel, or HTML. > Tip: paste `https://formergov.com` (or a bare `https://formergov.com/directory`) to scrape the **entire directory**, or paste individual profile URLs like `https://formergov.com/directory/brianlevine` to scrape just those people. ### Input The Actor takes a single field: a list of FormerGov URLs copied straight from your browser. No need to look up internal ids — the site already encodes every filter in the URL. | Field | Type | Description | |---|---|---| | `startUrls` | request list | FormerGov URLs to scrape. Each URL is one of: a **directory search URL** (e.g. `https://formergov.com/directory?type=combined§ors=`) whose filters drive the search; the **home page** `https://formergov.com` (or a bare `/directory`) to scrape the whole directory; or an **individual profile URL** `https://formergov.com/directory/`. `page`/`pageSize` in the URL are ignored — the Actor paginates for you. | | `maxItems` | integer | Max profiles to scrape (0 = no limit). | | `proxyConfiguration` | object | Proxy settings. Defaults to Apify Proxy (datacenter). If you see persistent `HTTP 403`, switch to **residential** groups. | If any individual profile URLs are present, only those are scraped (the search URL is ignored). #### Example input ```json { "startUrls": [ { "url": "https://formergov.com/directory?type=combined&practiceAreas=1e01b018-bd8a-4c1b-857d-3f93d613935a§ors=e97b9d61-7fd5-4565-8ca5-8f4d5a9dd56e" } ], "maxItems": 200 } ```` ### Output Each dataset item is one person. You can download the dataset in various formats such as JSON, HTML, CSV, or Excel. ```json { "username": "brianlevine", "profileUrl": "https://formergov.com/directory/brianlevine", "fullName": "Brian L Levine", "firstName": "Brian", "lastName": "Levine", "headline": "Leading at the intersection of law and technology", "city": "Washington", "state": "District of Columbia", "country": "United States", "linkedinUrl": "https://www.linkedin.com/in/brian-levine-cyberlaw", "websiteUrl": "https://www.ey.com/en_us/people/brian-levine", "websites": [ { "name": "LinkedIn", "url": "https://www.linkedin.com/in/brian-levine-cyberlaw" }, { "name": "Employer", "url": "https://www.ey.com/en_us/people/brian-levine" } ], "clearVerified": true, "currentTitle": "Founder and Executive Director", "currentEmployer": "FormerGov", "sectors": ["Consulting Services", "Public Services", "Legal", "Technology"], "practiceAreas": ["Artificial Intelligence", "Cybersecurity", "Privacy / Data Privacy"], "functions": ["Cybersecurity", "Data Privacy", "Technology"], "roles": [{ "title": "Managing Director", "employer": "EY Parthenon", "isCurrentRole": false, "isGovernmentRole": false }], "scrapedAt": "2026-06-25T12:00:00+00:00" } ``` #### Data fields | Field | Description | |---|---| | `username`, `profileUrl` | Directory handle and public profile link. | | `firstName`, `middleName`, `lastName`, `fullName` | Person's name. | | `headline` | Professional headline. | | `city`, `state`, `country` | Location. | | `linkedinUrl` | LinkedIn profile URL (contact). | | `websiteUrl` | Primary non-LinkedIn website (contact). | | `websites` | Full list of `{name, url}` links on the profile. | | `clearVerified` | Whether the member is identity-verified via CLEAR. | | `currentTitle`, `currentEmployer` | Current role. | | `sectors`, `practiceAreas`, `functions` | Expertise tags. | | `roles` | Structured employment history. | | `biography`, `education`, `certifications`, `languages`, `memberships`, `honorsAwards`, `publications` | Rich profile detail. | | `profilePicture` | Profile image URL. | | `scrapedAt` | ISO timestamp of extraction. | ### How much does it cost to scrape FormerGov? This Actor is **pay per result**: you are charged **US$0.0015 per profile** delivered to the dataset (US$1.50 per 1,000 profiles). You only pay for unique profiles actually returned — duplicates, not-found (404) profiles, and failed requests are not billed. For example, scraping the entire directory (~2,500 profiles) costs about **US$3.75**. Set **Max profiles** (`maxItems`) to cap how many rows a run can produce, and set your run's max-charge limit in the Console to stay within budget. ### Tips and advanced options - **Whole directory**: paste `https://formergov.com` (or a bare `https://formergov.com/directory`) — an unfiltered `type=combined` search that covers every profile (~2,500). - **Any filter**: anything the site's advanced search can express ends up in the URL (search type, practice areas, sectors, functions, jurisdiction, agency, combined second-leg filters, free text, …), so it works automatically — just build the search on formergov.com and copy the URL. - **Search result ceiling**: a single search returns at most **10,000** results (the backend's pagination window). The full directory is well under that, but for a filtered search exceeding 10,000, split it into narrower queries (e.g. by jurisdiction, sector, or state) to reach everything. The Actor logs a warning when a search exceeds this ceiling. - **Missing & blocked profiles**: some directory usernames have no public profile (HTTP 404) and are skipped. Profiles blocked by a temporary IP block (HTTP 403) are automatically retried at the end of the run, so transient blocks don't drop them. The run's final log line reports how many were scraped, not-found, and permanently failed. - **403 errors**: FormerGov has anti-bot protection that primarily blocks non-browser requests, which the Actor handles by sending browser headers, pacing requests (AutoThrottle), and retrying blocked requests on a fresh IP. Some datacenter IPs can also be rejected by reputation; if you see **persistent** `HTTP 403` warnings, switch the proxy to **residential** groups in the input. ### FAQ, disclaimers, and support **Is scraping FormerGov legal?** This Actor collects only publicly available profile information that FormerGov members choose to publish in the public directory. You are responsible for using the data in compliance with FormerGov's Terms of Service, applicable laws (including data-protection rules such as GDPR/CCPA), and for respecting individuals' privacy. Do not use the data for spam or any unlawful purpose. **Found a bug or need a custom field?** Open an issue on the Actor's **Issues** tab. Custom scraping solutions can be arranged on request. # Actor input Schema ## `startUrls` (type: `array`): Paste FormerGov URLs straight from your browser: - A **directory search URL** to scrape its results, e.g. `https://formergov.com/directory?type=combined&practiceAreas=1e01b018-bd8a-4c1b-857d-3f93d613935a§ors=e97b9d61-7fd5-4565-8ca5-8f4d5a9dd56e`. Build it on the site with the advanced search filters you want, then copy the address bar. - The **home page** `https://formergov.com` (or a bare `https://formergov.com/directory`) to scrape the entire directory. - An **individual profile URL** like `https://formergov.com/directory/brianlevine` to scrape just that person. You can mix several profile URLs, or give a single search/home-page URL. `page`/`pageSize` in the URL are ignored — the scraper paginates for you. ## `maxItems` (type: `integer`): Maximum number of profiles to scrape (0 = no limit). Note: deep pagination on the search API is windowed, so very large searches may not reach every result in one run — narrow the search with filters to get the rest. ## `proxyConfiguration` (type: `object`): Proxy servers used by the scraper. The site has anti-bot protection that mainly blocks non-browser requests (handled by the scraper's browser headers); blocked requests are retried on a fresh IP. If you hit persistent HTTP 403 warnings, some datacenter IPs are likely being rejected — switch to RESIDENTIAL proxy groups here. ## Actor input object example ```json { "startUrls": [ { "url": "https://formergov.com/directory?type=combined&practiceAreas=1e01b018-bd8a-4c1b-857d-3f93d613935a§ors=e97b9d61-7fd5-4565-8ca5-8f4d5a9dd56e" } ], "maxItems": 0, "proxyConfiguration": { "useApifyProxy": true } } ``` # Actor output Schema ## `profiles` (type: `string`): No description ## `datasetOverview` (type: `string`): No description ## `contacts` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ { "url": "https://formergov.com/directory?type=combined&practiceAreas=1e01b018-bd8a-4c1b-857d-3f93d613935a§ors=e97b9d61-7fd5-4565-8ca5-8f4d5a9dd56e" } ], "proxyConfiguration": { "useApifyProxy": true } }; // Run the Actor and wait for it to finish const run = await client.actor("rl1987/formergov-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": [{ "url": "https://formergov.com/directory?type=combined&practiceAreas=1e01b018-bd8a-4c1b-857d-3f93d613935a§ors=e97b9d61-7fd5-4565-8ca5-8f4d5a9dd56e" }], "proxyConfiguration": { "useApifyProxy": True }, } # Run the Actor and wait for it to finish run = client.actor("rl1987/formergov-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ { "url": "https://formergov.com/directory?type=combined&practiceAreas=1e01b018-bd8a-4c1b-857d-3f93d613935a§ors=e97b9d61-7fd5-4565-8ca5-8f4d5a9dd56e" } ], "proxyConfiguration": { "useApifyProxy": true } }' | apify call rl1987/formergov-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=rl1987/formergov-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "FormerGov Directory Scraper", "description": "Scrapes the formergov.com directory of former government and military professionals, with contact info (LinkedIn, website, email) and full advanced-search filtering.", "version": "0.1", "x-build-id": "kirPxNjGXflPjs6aF" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/rl1987~formergov-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-rl1987-formergov-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/rl1987~formergov-scraper/runs": { "post": { "operationId": "runs-sync-rl1987-formergov-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/rl1987~formergov-scraper/run-sync": { "post": { "operationId": "run-sync-rl1987-formergov-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "startUrls": { "title": "FormerGov URLs", "type": "array", "description": "Paste FormerGov URLs straight from your browser:\n\n- A **directory search URL** to scrape its results, e.g. `https://formergov.com/directory?type=combined&practiceAreas=1e01b018-bd8a-4c1b-857d-3f93d613935a§ors=e97b9d61-7fd5-4565-8ca5-8f4d5a9dd56e`. Build it on the site with the advanced search filters you want, then copy the address bar.\n- The **home page** `https://formergov.com` (or a bare `https://formergov.com/directory`) to scrape the entire directory.\n- An **individual profile URL** like `https://formergov.com/directory/brianlevine` to scrape just that person.\n\nYou can mix several profile URLs, or give a single search/home-page URL. `page`/`pageSize` in the URL are ignored — the scraper paginates for you.", "items": { "type": "object", "required": [ "url" ], "properties": { "url": { "type": "string", "title": "URL of a web page", "format": "uri" } } } }, "maxItems": { "title": "Max profiles", "minimum": 0, "type": "integer", "description": "Maximum number of profiles to scrape (0 = no limit). Note: deep pagination on the search API is windowed, so very large searches may not reach every result in one run — narrow the search with filters to get the rest.", "default": 0 }, "proxyConfiguration": { "title": "Proxy configuration", "type": "object", "description": "Proxy servers used by the scraper. The site has anti-bot protection that mainly blocks non-browser requests (handled by the scraper's browser headers); blocked requests are retried on a fresh IP. If you hit persistent HTTP 403 warnings, some datacenter IPs are likely being rejected — switch to RESIDENTIAL proxy groups here.", "default": { "useApifyProxy": true } } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```