# Open Payments Scraper (`automation-lab/open-payments-scraper`) Actor Extract official CMS Open Payments records, dataset metadata, and CSV URLs by year, payment type, NPI, company, and state. - **URL**: https://apify.com/automation-lab/open-payments-scraper.md - **Developed by:** [Stas Persiianenko](https://apify.com/automation-lab) (community) - **Categories:** Business - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing Pay per event This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Open Payments Scraper Extract CMS Open Payments payment records, dataset metadata, and CSV download URLs from the official public CMS Open Payments APIs. Open Payments Scraper helps compliance teams, healthcare analysts, journalists, pharma/medtech researchers, and sales operations teams turn the large CMS Open Payments catalog into clean Apify datasets. Use it to pull payments by reporting year, payment type, physician NPI, recipient name, company name, and state, or to discover official CSV files for bulk processing. ### What does Open Payments Scraper do? Open Payments Scraper connects to the public CMS Open Payments metadata and datastore APIs. It can: - ๐Ÿฅ Extract detailed general, research, and ownership payment records. - ๐Ÿ“… Select CMS reporting years such as 2024, 2023, or 2022. - ๐Ÿ”Ž Filter output by recipient NPI, recipient name, company name, and state. - ๐Ÿ“„ Return dataset metadata only for lightweight discovery. - ๐Ÿ”— Return official CMS CSV download URLs for bulk ETL workflows. - ๐Ÿงพ Preserve the raw CMS API record for auditability. ### Who is it for? Open Payments Scraper is useful for teams that need repeatable access to CMS payment transparency data. - Compliance teams monitoring physician-industry relationships. - Healthcare analytics teams building payment dashboards. - Pharma and medtech commercial teams researching customer segments. - Journalists investigating healthcare payments. - Academic researchers analyzing Open Payments trends. - Data engineers who need stable CSV URLs and dataset identifiers. ### Why use this actor? The CMS website and bulk files are powerful but large. This actor gives you a controlled Apify interface with limits, filters, and normalized output. Instead of manually navigating CMS pages, downloading huge CSV files, and writing one-off scripts, you can run a saved Apify task and export the results as JSON, CSV, Excel, or through the API. ### What data can you extract? | Field | Description | | --- | --- | | `recordType` | `payment`, `dataset_metadata`, or `csv_url` | | `datasetIdentifier` | CMS dataset UUID | | `datasetTitle` | CMS dataset title | | `datasetYear` | Reporting year when available | | `datasetType` | General, research, ownership, summary, or profile | | `recipientName` | Physician, entity, or hospital name | | `recipientNpi` | Covered recipient or investigator NPI | | `companyName` | Manufacturer, GPO, or submitting organization | | `paymentAmountUsd` | Payment amount in USD | | `paymentDate` | CMS payment date | | `paymentNature` | Nature of payment or transfer of value | | `csvDownloadUrl` | Official CMS CSV download URL | | `rawRecord` | Full original CMS API row | ### How much does it cost to scrape CMS Open Payments data? This actor uses pay-per-event pricing. - Start event: $0.005 per run. - Per result item: Free $0.000032381, Starter/BRONZE $0.000028158, Scale/SILVER $0.000021963, Business/GOLD $0.000016895, Platinum $0.000011263, Diamond $0.00001. - Each saved payment, metadata, or CSV URL row is charged as one result item. - Use `metadataOnly` or `csvUrlsOnly` for cheap discovery runs before bulk extraction. - Keep `maxItems` low while testing filters, then raise it for production runs. ### Quick start 1. Open the actor on Apify. 2. Choose one or more reporting years. 3. Choose dataset types such as `general`, `research`, or `ownership`. 4. Optionally enter an NPI, recipient name, company name, or state. 5. Set `maxItems`. 6. Run the actor. 7. Export the dataset or consume it through the Apify API. ### Input options #### `years` Array of CMS reporting years. Example: `[2024]`. #### `datasetTypes` Choose one or more dataset families: - `general` - `research` - `ownership` - `summary` - `profile` - `all` #### `datasetIdentifiers` Optional CMS dataset UUIDs. If you already know the exact dataset identifier, this overrides year/type selection. #### `recipientNpi` Filter by covered recipient or principal investigator NPI. #### `recipientName` Case-insensitive filter for physician, entity, or teaching hospital names. #### `companyName` Case-insensitive filter for manufacturer, GPO, or reporting entity names. #### `state` Two-letter state code such as `CA`, `NY`, or `TX`. #### `metadataOnly` Return one row per matching CMS dataset with title, identifier, modified date, source URL, and download links. #### `csvUrlsOnly` Return one row per matching dataset focused on official CSV download URLs. #### `maxItems` Maximum number of output rows to save. ### Example input ```json { "years": [2024], "datasetTypes": ["general"], "state": "CA", "maxItems": 100 } ```` ### Output example ```json { "recordType": "payment", "datasetIdentifier": "e6b17c6a-2534-4207-a4a1-6746a14911ff", "datasetTitle": "2024 General Payment Data", "datasetYear": "2024", "datasetType": "general", "recipientName": "Jane Smith", "recipientNpi": "1234567890", "recipientState": "CA", "companyName": "Example Medical Inc.", "paymentAmountUsd": 25.0, "paymentDate": "2024-05-10", "paymentNature": "Food and Beverage", "csvDownloadUrl": "https://download.cms.gov/openpayments/...csv" } ``` ### Tips for better results - Start with `metadataOnly: true` to see available datasets. - Use `csvUrlsOnly: true` when you want CMS bulk files instead of paginated rows. - Use exact NPI values for targeted provider checks. - Use state filters for regional compliance reviews. - Increase `maxItems` only after confirming your filters. ### Integrations Open Payments Scraper works with common Apify workflows: - Export to Google Sheets for compliance review. - Send dataset items to Make or Zapier. - Load JSONL output into Snowflake, BigQuery, or Postgres. - Schedule recurring monitoring tasks. - Combine with lead enrichment or healthcare-provider datasets. ### API usage #### Node.js ```js import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('automation-lab/open-payments-scraper').call({ years: [2024], datasetTypes: ['general'], maxItems: 100, }); console.log(run.defaultDatasetId); ``` #### Python ```python from apify_client import ApifyClient client = ApifyClient('MY-APIFY-TOKEN') run = client.actor('automation-lab/open-payments-scraper').call(run_input={ 'years': [2024], 'datasetTypes': ['general'], 'maxItems': 100, }) print(run['defaultDatasetId']) ``` #### cURL ```bash curl -X POST "https://api.apify.com/v2/acts/automation-lab~open-payments-scraper/runs?token=$APIFY_TOKEN" \ -H 'Content-Type: application/json' \ -d '{"years":[2024],"datasetTypes":["general"],"maxItems":100}' ``` ### MCP usage Use this actor from MCP-compatible clients through Apify MCP. MCP URL: ```text https://mcp.apify.com/?tools=automation-lab/open-payments-scraper ``` Claude Code setup: ```bash claude mcp add apify-open-payments "https://mcp.apify.com/?tools=automation-lab/open-payments-scraper" ``` MCP JSON configuration: ```json { "mcpServers": { "apify-open-payments": { "url": "https://mcp.apify.com/?tools=automation-lab/open-payments-scraper" } } } ``` Example prompts: - "Run Open Payments Scraper for 2024 general payments in California and summarize the largest payments." - "Find CMS Open Payments CSV URLs for 2024 research and ownership payment datasets." - "Extract 100 Open Payments records for a specific physician NPI." ### Data freshness The actor reads the live CMS Open Payments metadata API. CMS controls dataset publication schedules, modified dates, and CSV file paths. ### Legality CMS Open Payments data is public United States government transparency data. You are responsible for using it lawfully and for complying with your organization's privacy, compliance, and data-retention policies. ### FAQ #### Why did I get fewer rows than `maxItems`? Your filters may be narrow, or the selected dataset may have fewer matching rows within the bounded scan window. Try metadata discovery first, broaden filters, or use the official CSV URL for bulk processing. #### Why is `rawRecord` large? CMS records contain many columns. The actor keeps the raw row so analysts can audit values that are not normalized into top-level fields. #### Should I use CSV mode or payment mode? Use payment mode for API-ready samples and filtered exports. Use CSV URL mode for warehouse-scale ETL. ### Related scrapers Other Automation Lab actors can complement this data: - https://apify.com/automation-lab/website-contact-finder - https://apify.com/automation-lab/google-maps-lead-finder - https://apify.com/automation-lab/company-website-scraper ### Support If a CMS field changes or a dataset identifier stops working, open an issue on the actor page with the run ID and input used. # Actor input Schema ## `years` (type: `array`): CMS Open Payments reporting years to search. Leave empty to use the latest detailed general payment datasets available in metadata order. ## `datasetTypes` (type: `array`): Open Payments dataset families to include. Detailed payment types are general, research, and ownership; summary/profile datasets are also supported. ## `datasetIdentifiers` (type: `array`): Optional CMS metastore identifiers. When provided, these override year/type selection. ## `recipientNpi` (type: `string`): Filter detailed payment records by covered recipient or principal investigator NPI. ## `recipientName` (type: `string`): Case-insensitive name filter for physician, non-covered recipient entity, or teaching hospital names. ## `companyName` (type: `string`): Case-insensitive filter for the manufacturer, GPO, or submitting organization name. ## `state` (type: `string`): Two-letter recipient state filter, for example CA or NY. ## `metadataOnly` (type: `boolean`): Return matching dataset titles, identifiers, modified dates, data dictionary links, and CSV URLs without paging payment rows. ## `csvUrlsOnly` (type: `boolean`): Return one item per matching CMS dataset with CSV download URLs for bulk workflows. ## `maxItems` (type: `integer`): Maximum dataset items to save. Keep this low for filtered exploratory runs; raise it for bulk extraction. ## Actor input object example ```json { "years": [ 2024 ], "datasetTypes": [ "general" ], "datasetIdentifiers": [], "metadataOnly": false, "csvUrlsOnly": false, "maxItems": 20 } ``` # Actor output Schema ## `overview` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "years": [ 2024 ], "datasetTypes": [ "general" ], "datasetIdentifiers": [], "recipientNpi": "", "recipientName": "", "companyName": "", "state": "", "metadataOnly": false, "csvUrlsOnly": false, "maxItems": 20 }; // Run the Actor and wait for it to finish const run = await client.actor("automation-lab/open-payments-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`๐Ÿ’พ Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // ๐Ÿ“š Want to learn more ๐Ÿ“–? Go to โ†’ https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "years": [2024], "datasetTypes": ["general"], "datasetIdentifiers": [], "recipientNpi": "", "recipientName": "", "companyName": "", "state": "", "metadataOnly": False, "csvUrlsOnly": False, "maxItems": 20, } # Run the Actor and wait for it to finish run = client.actor("automation-lab/open-payments-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("๐Ÿ’พ Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # ๐Ÿ“š Want to learn more ๐Ÿ“–? Go to โ†’ https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "years": [ 2024 ], "datasetTypes": [ "general" ], "datasetIdentifiers": [], "recipientNpi": "", "recipientName": "", "companyName": "", "state": "", "metadataOnly": false, "csvUrlsOnly": false, "maxItems": 20 }' | apify call automation-lab/open-payments-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=automation-lab/open-payments-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Open Payments Scraper", "description": "Extract official CMS Open Payments records, dataset metadata, and CSV URLs by year, payment type, NPI, company, and state.", "version": "0.1", "x-build-id": "Cy8T2qQVPNxuXesR1" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/automation-lab~open-payments-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-automation-lab-open-payments-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/automation-lab~open-payments-scraper/runs": { "post": { "operationId": "runs-sync-automation-lab-open-payments-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/automation-lab~open-payments-scraper/run-sync": { "post": { "operationId": "run-sync-automation-lab-open-payments-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "years": { "title": "Reporting years", "type": "array", "description": "CMS Open Payments reporting years to search. Leave empty to use the latest detailed general payment datasets available in metadata order.", "items": { "type": "integer", "minimum": 2013, "maximum": 2030 } }, "datasetTypes": { "title": "Dataset/payment types", "uniqueItems": true, "type": "array", "description": "Open Payments dataset families to include. Detailed payment types are general, research, and ownership; summary/profile datasets are also supported.", "items": { "type": "string" }, "default": [ "general" ] }, "datasetIdentifiers": { "title": "Specific CMS dataset identifiers", "type": "array", "description": "Optional CMS metastore identifiers. When provided, these override year/type selection.", "items": { "type": "string" } }, "recipientNpi": { "title": "Recipient NPI", "type": "string", "description": "Filter detailed payment records by covered recipient or principal investigator NPI." }, "recipientName": { "title": "Recipient or physician name contains", "type": "string", "description": "Case-insensitive name filter for physician, non-covered recipient entity, or teaching hospital names." }, "companyName": { "title": "Company/reporting entity name contains", "type": "string", "description": "Case-insensitive filter for the manufacturer, GPO, or submitting organization name." }, "state": { "title": "Recipient state", "type": "string", "description": "Two-letter recipient state filter, for example CA or NY." }, "metadataOnly": { "title": "Return dataset metadata only", "type": "boolean", "description": "Return matching dataset titles, identifiers, modified dates, data dictionary links, and CSV URLs without paging payment rows.", "default": false }, "csvUrlsOnly": { "title": "Return CSV download URLs only", "type": "boolean", "description": "Return one item per matching CMS dataset with CSV download URLs for bulk workflows.", "default": false }, "maxItems": { "title": "Maximum output items", "minimum": 1, "maximum": 100000, "type": "integer", "description": "Maximum dataset items to save. Keep this low for filtered exploratory runs; raise it for bulk extraction.", "default": 20 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```