Company Search
Pricing
from $3.12 / 1,000 api requests
Tomba Companies lets you identify companies aligned with your ideal customer profile. Add filters to quickly surface qualified companies and proceed to find their contact information.
Pricing
from $3.12 / 1,000 api requests
Rating
0.0
(0)
Developer
Tomba io
Maintained by CommunityActor stats
0
Bookmarked
30
Total users
1
Monthly active users
6 days ago
Last modified
Categories
Share
1[settings]2idiomatic_version_file_enable_tools = []3
4[tools."github:apify/apify-cli"]5version = "latest"6bin = "apify"7
8[tools."github:apify/apify-cli".platforms]9linux-x64 = { asset_pattern = "apify-*-linux-x64" }10linux-arm64 = { asset_pattern = "apify-*-linux-arm64" }11macos-x64 = { asset_pattern = "apify-*-darwin-x64" }12macos-arm64 = { asset_pattern = "apify-*-darwin-arm64" }13windows-x64 = { asset_pattern = "apify-*-windows-x64.exe" }14windows-arm64 = { asset_pattern = "apify-*-windows-arm64.exe" }.prettierignore{ "printWidth": 120, "singleQuote": true, "tabWidth": 4}{ "extends": "./tsconfig.json", "compilerOptions": { "noEmit": true }, "include": ["./src/**/*", "./test/**/*"]}# configurations.idea.vscode.zed
# crawlee and apify storage foldersapify_storagecrawlee_storagestorage
# installed filesnode_modules
# git folder.git
# dist folderdisttest# This file tells Git which files shouldn't be added to source control
.idea.vscode.zedstorageapify_storagecrawlee_storagenode_modulesdisttsconfig.tsbuildinfo
# Added by Apify CLI.venv1# Apify Actors Development Guide2
3Important: Before you begin, fill in the `generatedBy` property in the meta section of `.actor/actor.json`. Replace it with the model you're currently using. This helps Apify monitor and improve AGENTS.md for specific LLM models.4
5## This Actor: Tomba Company Search6
7Maintainer notes for this Actor. The README is the end-user page shown on Apify Store and must not contain developer content (SDK snippets, secrets, monetization setup). The rest of this file is the generic Apify Actors guide.8
9### Scripts10
11```bash12npm install13npm run build # compile TypeScript to dist/14npm run lint # ESLint (src and test)15npm run format # Prettier16npm test # unit + end-to-end tests (node:test)17npm start # run locally with tsx18```19
20### Credentials21
22The Actor uses our Tomba account. Credentials come from environment variables, never from the input (there is no API key field):23
24| Variable | Description |25| -------------------- | -------------------------------------------------- |26| `TOMBA_API_KEY` | Tomba API key (`ta_…`) |27| `TOMBA_API_SECRET` | Tomba secret (`ts_…`) |28| `TOMBA_API_ENDPOINT` | Optional API base URL; only used by the test suite |2930`.actor/actor.json` maps the variables to Apify secrets:31
32```bash33apify secrets add tombaApiKey ta_xxxxxxxxxxxxxxxxxxxx34apify secrets add tombaApiSecret ts_xxxxxxxxxxxxxxxxxxxx35apify push36```37
38Run locally:39
40```bash41TOMBA_API_KEY=ta_… TOMBA_API_SECRET=ts_… npm start42```43
44If the variables are missing the run fails with "Actor is misconfigured" before any request is made.45
46### Pricing (pay per event)47
48In **Apify Console → Publication → Monetization**, choose **Pay per event** and add:49
50| Event | Price | Charged when |51| --------------- | -------- | ---------------------------------------------------------- |52| `tomba-request` | $0.00312 | Tomba returns a billable response for one page (see below) |5354Company Search is paged: each page (`meta.limit` companies) is a separate `POST /reveal/search` request and a separate event.55
56`isBillable()` in `src/tomba.ts` mirrors Tomba's billing:57
58| Tomba outcome | Charged |59| ---------------------------------------------------------------- | ------- |60| JSON with non-empty `data`, including a page with zero companies | Yes |61| Error status (4xx, 5xx, including 422 and 429) | No |62| Success with empty or null `data` | No |63| Success with an `errors` object | No |64| Non-JSON body (reported as 502) | No |65| Cache hit | No |6667There is no client-side rate limit or delay between pages; 429 and 5xx responses are retried with exponential backoff.68
69### Architecture70
71- `src/tomba.ts`: shared helper, identical in every Tomba Actor. It handles credentials, caching (per-Actor `tomba-cache-<actorId>` key-value store; falls back to an in-run cache if it can't be opened), retries with exponential backoff, pay-per-event charging, budget reservation, the concurrency pool and resume state. Do not edit it in one Actor only.72- `src/main.ts`: input handling, paging and output mapping.73 - The run fails early when there is neither a `query` nor `filters`.74 - The request body is `{ query, filters, _source, page }` (`source` input → `_source`; empty values are left out).75 - Pages are fetched sequentially starting at `page`; `maxConcurrency` has no effect here.76 - Paging stops at `maxResults`, at `meta.pages`, at a short page (`< meta.limit`), at a non-billable or empty page, or when the charge limit is reached.77 - One dataset item per company: the Tomba company fields plus `source`, `charged`, `cached`. When the search returns nothing, one item `{ query, page, source, charged, cached, error }` is saved.78 - Resume (batch runs only): finished pages (`TOMBA_STATE`) and the number of companies saved (`COMPANY_SEARCH_PROGRESS`) are persisted, so a resumed run continues at the next page and still honours `maxResults`. In Standby mode this progress is kept in memory for the request only.79- The `tomba` SDK v1.1.1 resolves every call to `{ data, rateLimit }`, where `data` is the response body. Its `.d.ts` types still declare the old return type, so always go through `callTomba()`.80
81### Tests82
83- `test/tomba.test.ts`: unit tests for the shared helper (identical in every Actor)84- `test/main.test.ts`: end-to-end tests that run `src/main.ts` against a local mock Tomba API (paging, billing, cache, retries, charge limit and resume, credentials, Standby HTTP API)85- `test/helpers.ts`: mock server and Actor runner (identical in every Actor)86
87Locally, the Apify SDK prices every event at $1 when `ACTOR_TEST_PAY_PER_EVENT=true`, so the tests use `maxTotalChargeUsd` as an event count.88
89### Standby mode (real-time API)90
91`.actor/actor.json` sets `usesStandbyMode: true` and `webServerSchema: ./web_server_schema.json` (OpenAPI 3).92
93- `src/standby.ts` (shared, identical in every Actor): `runActor()` runs a batch job, or, when `APIFY_META_ORIGIN=STANDBY`, starts an HTTP server on `Actor.config.get('containerPort')`.94 - `GET /` with the `x-apify-container-server-readiness-probe` header, or with no query: readiness / usage.95 - `GET /?…`: input built by `fromQuery()` in `src/main.ts` (`query`, `filters` as a JSON string, `source` as repeated or comma-separated values, `page`, `maxResults`). Invalid `filters` JSON is a `400`.96 - `POST /`: the same JSON input as a batch run.97 - Responses: `200 { items }`, `400` invalid input (`InputError`), `402` max charge limit reached, `404`, `405`.98- `run(input, ctx)` is shared by both modes: `ctx.push()` writes to the dataset in batch runs and to the HTTP response in Standby; `ctx.isDone()`/`ctx.markDone()` persist resume state only in batch runs. `COMPANY_SEARCH_PROGRESS` is only used in batch runs.99- `maxResults` is a local loop condition, never `stop()` (a global flag reserved for the charge limit).100- Caching and pay-per-event charging work the same in both modes.101
102Try it locally:103
104```bash105APIFY_META_ORIGIN=STANDBY ACTOR_WEB_SERVER_PORT=8080 TOMBA_API_KEY=ta_… TOMBA_API_SECRET=ts_… npm start106curl "localhost:8080/?query=software%20companies&maxResults=10"107```108
109### Key-value store schema110
111`.actor/key_value_store_schema.json` documents the default key-value store records (`INPUT`, `TOMBA_STATE`, `COMPANY_SEARCH_PROGRESS`). The cross-run cache lives in the separate named store `tomba-cache-<actorId>`, one per Actor: under limited permissions an Actor can only open named storages it created itself, so the Tomba Actors must not share one store. If the store can't be opened, the run logs a warning and caches for this run only.112
113### Memory114
115`defaultMemoryMbytes` is 256: the Actor only makes HTTP calls, so more memory just costs more.116
117## What are Apify Actors?118
119- Actors are serverless programs that run in the cloud. They're inspired by the UNIX philosophy - programs that do one thing well and can be easily combined to build complex systems.120- Actors are programs packaged as Docker images that run in isolated containers121
122## Core Concepts123
124- Accept well-defined JSON input125- Perform isolated tasks (web scraping, automation, data processing)126- Produce structured JSON output to datasets and/or store data in key-value stores127- Can run from seconds to hours or even indefinitely128- Persist state and can be restarted129
130## Do131
132- accept well-defined JSON input and produce structured JSON output133- use Apify SDK (`apify`) for code running ON Apify platform134- validate input early with proper error handling and fail gracefully135- use CheerioCrawler for static HTML content (10x faster than browsers)136- use PlaywrightCrawler only for JavaScript-heavy sites and dynamic content137- use router pattern (createCheerioRouter/createPlaywrightRouter) for complex crawls138- implement retry strategies with exponential backoff for failed requests139- use proper concurrency settings (HTTP: 10-50, Browser: 1-5)140- set sensible defaults in `.actor/input_schema.json` for all optional fields141- set up output schema in `.actor/output_schema.json`142- clean and validate data before pushing to dataset143- use semantic CSS selectors and fallback strategies for missing elements144- respect robots.txt, ToS, and implement rate limiting with delays when scraping (this Actor calls the Tomba API and needs no client-side rate limit)145- check which tools (cheerio/playwright/crawlee) are installed before applying guidance146
147## Don't148
149- do not rely on `Dataset.getInfo()` for final counts on Cloud platform150- do not use browser crawlers when HTTP/Cheerio works (massive performance gains with HTTP)151- do not hard code values that should be in input schema or environment variables152- do not skip input validation or error handling153- do not overload servers - use appropriate concurrency and delays154- do not scrape prohibited content or ignore Terms of Service155- do not store personal/sensitive data unless explicitly permitted156- do not use deprecated options like `requestHandlerTimeoutMillis` on CheerioCrawler (v3.x)157- do not use `additionalHttpHeaders` - use `preNavigationHooks` instead158
159## Commands160
161```bash162# Local development163apify run # Run Actor locally164
165# Authentication & deployment166apify login # Authenticate account167apify push # Deploy to Apify platform168
169# Help170apify help # List all commands171```172
173## Safety and Permissions174
175Allowed without prompt:176
177- read files with `Actor.getValue()`178- push data with `Actor.pushData()`179- set values with `Actor.setValue()`180- enqueue requests to RequestQueue181- run locally with `apify run`182
183Ask first:184
185- npm/pip package installations186- apify push (deployment to cloud)187- proxy configuration changes (requires paid plan)188- Dockerfile changes affecting builds189- deleting datasets or key-value stores190
191## Project Structure192
193.actor/194├── actor.json # Actor config: name, version, env vars, runtime settings195├── input_schema.json # Input validation & Console form definition196└── output_schema.json # Specifies where an Actor stores its output197src/198├── main.ts # Actor entry point: input, paging, output199└── tomba.ts # Shared Tomba helper (identical in every Tomba Actor)200test/ # node:test unit and end-to-end tests201storage/ # Local storage (mirrors Cloud during development)202├── datasets/ # Output items (JSON objects)203├── key_value_stores/ # Files, config, INPUT204└── request_queues/ # Pending crawl requests205Dockerfile # Container image definition206AGENTS.md # AI agent instructions (this file)207
208## Actor Input Schema209
210The input schema defines the input parameters for an Actor. It's a JSON object comprising various field types supported by the Apify platform.211
212### Structure213
214```json215{216 "title": "<INPUT-SCHEMA-TITLE>",217 "type": "object",218 "schemaVersion": 1,219 "properties": {220 /* define input fields here */221 },222 "required": []223}224```225
226### Example227
228```json229{230 "title": "E-commerce Product Scraper Input",231 "type": "object",232 "schemaVersion": 1,233 "properties": {234 "startUrls": {235 "title": "Start URLs",236 "type": "array",237 "description": "URLs to start scraping from (category pages or product pages)",238 "editor": "requestListSources",239 "default": [{ "url": "https://example.com/category" }],240 "prefill": [{ "url": "https://example.com/category" }]241 },242 "followVariants": {243 "title": "Follow Product Variants",244 "type": "boolean",245 "description": "Whether to scrape product variants (different colors, sizes)",246 "default": true247 },248 "maxRequestsPerCrawl": {249 "title": "Max Requests per Crawl",250 "type": "integer",251 "description": "Maximum number of pages to scrape (0 = unlimited)",252 "default": 1000,253 "minimum": 0254 },255 "proxyConfiguration": {256 "title": "Proxy Configuration",257 "type": "object",258 "description": "Proxy settings for anti-bot protection",259 "editor": "proxy",260 "default": { "useApifyProxy": false }261 },262 "locale": {263 "title": "Locale",264 "type": "string",265 "description": "Language/country code for localized content",266 "default": "cs",267 "enum": ["cs", "en", "de", "sk"],268 "enumTitles": ["Czech", "English", "German", "Slovak"]269 }270 },271 "required": ["startUrls"]272}273```274
275## Actor Output Schema276
277The Actor output schema builds upon the schemas for the dataset and key-value store. It specifies where an Actor stores its output and defines templates for accessing that output. Apify Console uses these output definitions to display run results.278
279### Structure280
281```json282{283 "actorOutputSchemaVersion": 1,284 "title": "<OUTPUT-SCHEMA-TITLE>",285 "properties": {286 /* define your outputs here */287 }288}289```290
291### Example292
293```json294{295 "actorOutputSchemaVersion": 1,296 "title": "Output schema of the files scraper",297 "properties": {298 "files": {299 "type": "string",300 "title": "Files",301 "template": "{{links.apiDefaultKeyValueStoreUrl}}/keys"302 },303 "dataset": {304 "type": "string",305 "title": "Dataset",306 "template": "{{links.apiDefaultDatasetUrl}}/items"307 }308 }309}310```311
312### Output Schema Template Variables313
314- `links` (object) - Contains quick links to most commonly used URLs315- `links.publicRunUrl` (string) - Public run url in format `https://console.apify.com/view/runs/:runId`316- `links.consoleRunUrl` (string) - Console run url in format `https://console.apify.com/actors/runs/:runId`317- `links.apiRunUrl` (string) - API run url in format `https://api.apify.com/v2/actor-runs/:runId`318- `links.apiDefaultDatasetUrl` (string) - API url of default dataset in format `https://api.apify.com/v2/datasets/:defaultDatasetId`319- `links.apiDefaultKeyValueStoreUrl` (string) - API url of default key-value store in format `https://api.apify.com/v2/key-value-stores/:defaultKeyValueStoreId`320- `links.containerRunUrl` (string) - URL of a webserver running inside the run in format `https://<containerId>.runs.apify.net/`321- `run` (object) - Contains information about the run same as it is returned from the `GET Run` API endpoint322- `run.defaultDatasetId` (string) - ID of the default dataset323- `run.defaultKeyValueStoreId` (string) - ID of the default key-value store324
325## Dataset Schema Specification326
327The dataset schema defines how your Actor's output data is structured, transformed, and displayed in the Output tab in the Apify Console.328
329### Example330
331Consider an example Actor that calls `Actor.pushData()` to store data into dataset:332
333```typescript334import { Actor } from 'apify';335// Initialize the JavaScript SDK336await Actor.init();337
338/**339 * Actor code340 */341await Actor.pushData({342 numericField: 10,343 pictureUrl: 'https://www.google.com/images/branding/googlelogo/2x/googlelogo_color_92x30dp.png',344 linkUrl: 'https://google.com',345 textField: 'Google',346 booleanField: true,347 dateField: new Date(),348 arrayField: ['#hello', '#world'],349 objectField: {},350});351
352// Exit successfully353await Actor.exit();354```355
356To set up the Actor's output tab UI, reference a dataset schema file in `.actor/actor.json`:357
358```json359{360 "actorSpecification": 1,361 "name": "book-library-scraper",362 "title": "Book Library Scraper",363 "version": "1.0.0",364 "storages": {365 "dataset": "./dataset_schema.json"366 }367}368```369
370Then create the dataset schema in `.actor/dataset_schema.json`:371
372```json373{374 "actorSpecification": 1,375 "fields": {},376 "views": {377 "overview": {378 "title": "Overview",379 "transformation": {380 "fields": [381 "pictureUrl",382 "linkUrl",383 "textField",384 "booleanField",385 "arrayField",386 "objectField",387 "dateField",388 "numericField"389 ]390 },391 "display": {392 "component": "table",393 "properties": {394 "pictureUrl": {395 "label": "Image",396 "format": "image"397 },398 "linkUrl": {399 "label": "Link",400 "format": "link"401 },402 "textField": {403 "label": "Text",404 "format": "text"405 },406 "booleanField": {407 "label": "Boolean",408 "format": "boolean"409 },410 "arrayField": {411 "label": "Array",412 "format": "array"413 },414 "objectField": {415 "label": "Object",416 "format": "object"417 },418 "dateField": {419 "label": "Date",420 "format": "date"421 },422 "numericField": {423 "label": "Number",424 "format": "number"425 }426 }427 }428 }429 }430}431```432
433### Structure434
435```json436{437 "actorSpecification": 1,438 "fields": {},439 "views": {440 "<VIEW_NAME>": {441 "title": "string (required)",442 "description": "string (optional)",443 "transformation": {444 "fields": ["string (required)"],445 "unwind": ["string (optional)"],446 "flatten": ["string (optional)"],447 "omit": ["string (optional)"],448 "limit": "integer (optional)",449 "desc": "boolean (optional)"450 },451 "display": {452 "component": "table (required)",453 "properties": {454 "<FIELD_NAME>": {455 "label": "string (optional)",456 "format": "text|number|date|link|boolean|image|array|object (optional)"457 }458 }459 }460 }461 }462}463```464
465**Dataset Schema Properties:**466
467- `actorSpecification` (integer, required) - Specifies the version of dataset schema structure document (currently only version 1)468- `fields` (JSONSchema object, required) - Schema of one dataset object (use JsonSchema Draft 2020-12 or compatible)469- `views` (DatasetView object, required) - Object with API and UI views description470
471**DatasetView Properties:**472
473- `title` (string, required) - Visible in UI Output tab and API474- `description` (string, optional) - Only available in API response475- `transformation` (ViewTransformation object, required) - Data transformation applied when loading from Dataset API476- `display` (ViewDisplay object, required) - Output tab UI visualization definition477
478**ViewTransformation Properties:**479
480- `fields` (string[], required) - Fields to present in output (order matches column order)481- `unwind` (string[], optional) - Deconstructs nested children into parent object482- `flatten` (string[], optional) - Transforms nested object into flat structure483- `omit` (string[], optional) - Removes specified fields from output484- `limit` (integer, optional) - Maximum number of results (default: all)485- `desc` (boolean, optional) - Sort order (true = newest first)486
487**ViewDisplay Properties:**488
489- `component` (string, required) - Only `table` is available490- `properties` (Object, optional) - Keys matching `transformation.fields` with ViewDisplayProperty values491
492**ViewDisplayProperty Properties:**493
494- `label` (string, optional) - Table column header495- `format` (string, optional) - One of: `text`, `number`, `date`, `link`, `boolean`, `image`, `array`, `object`496
497## Key-Value Store Schema Specification498
499The key-value store schema organizes keys into logical groups called collections for easier data management.500
501### Example502
503Consider an example Actor that calls `Actor.setValue()` to save records into the key-value store:504
505```typescript506import { Actor } from 'apify';507// Initialize the JavaScript SDK508await Actor.init();509
510/**511 * Actor code512 */513await Actor.setValue('document-1', 'my text data', { contentType: 'text/plain' });514
515await Actor.setValue(`image-${imageID}`, imageBuffer, { contentType: 'image/jpeg' });516
517// Exit successfully518await Actor.exit();519```520
521To configure the key-value store schema, reference a schema file in `.actor/actor.json`:522
523```json524{525 "actorSpecification": 1,526 "name": "data-collector",527 "title": "Data Collector",528 "version": "1.0.0",529 "storages": {530 "keyValueStore": "./key_value_store_schema.json"531 }532}533```534
535Then create the key-value store schema in `.actor/key_value_store_schema.json`:536
537```json538{539 "actorKeyValueStoreSchemaVersion": 1,540 "title": "Key-Value Store Schema",541 "collections": {542 "documents": {543 "title": "Documents",544 "description": "Text documents stored by the Actor",545 "keyPrefix": "document-"546 },547 "images": {548 "title": "Images",549 "description": "Images stored by the Actor",550 "keyPrefix": "image-",551 "contentTypes": ["image/jpeg"]552 }553 }554}555```556
557### Structure558
559```json560{561 "actorKeyValueStoreSchemaVersion": 1,562 "title": "string (required)",563 "description": "string (optional)",564 "collections": {565 "<COLLECTION_NAME>": {566 "title": "string (required)",567 "description": "string (optional)",568 "key": "string (conditional - use key OR keyPrefix)",569 "keyPrefix": "string (conditional - use key OR keyPrefix)",570 "contentTypes": ["string (optional)"],571 "jsonSchema": "object (optional)"572 }573 }574}575```576
577**Key-Value Store Schema Properties:**578
579- `actorKeyValueStoreSchemaVersion` (integer, required) - Version of key-value store schema structure document (currently only version 1)580- `title` (string, required) - Title of the schema581- `description` (string, optional) - Description of the schema582- `collections` (Object, required) - Object where each key is a collection ID and value is a Collection object583
584**Collection Properties:**585
586- `title` (string, required) - Collection title shown in UI tabs587- `description` (string, optional) - Description appearing in UI tooltips588- `key` (string, conditional) - Single specific key for this collection589- `keyPrefix` (string, conditional) - Prefix for keys included in this collection590- `contentTypes` (string[], optional) - Allowed content types for validation591- `jsonSchema` (object, optional) - JSON Schema Draft 07 format for `application/json` content type validation592
593Either `key` or `keyPrefix` must be specified for each collection, but not both.594
595## Apify MCP Tools596
597If MCP server is configured, use these tools for documentation:598
599- `search-apify-docs` - Search documentation600- `fetch-apify-docs` - Get full doc pages601
602Otherwise, reference: `@https://mcp.apify.com/`603
604## Resources605
606- [docs.apify.com/llms.txt](https://docs.apify.com/llms.txt) - Quick reference607- [docs.apify.com/llms-full.txt](https://docs.apify.com/llms-full.txt) - Complete docs608- [crawlee.dev](https://crawlee.dev) - Crawlee documentation609- [whitepaper.actor](https://raw.githubusercontent.com/apify/actor-whitepaper/refs/heads/master/README.md) - Complete Actor specification1import prettier from 'eslint-config-prettier';2
3import apify from '@apify/eslint-config/ts.js';4import globals from 'globals';5import tsEslint from 'typescript-eslint';6
7// eslint-disable-next-line import/no-default-export8export default [9 { ignores: ['**/dist', 'eslint.config.mjs'] },10 ...apify,11 prettier,12 {13 languageOptions: {14 parser: tsEslint.parser,15 parserOptions: {16 project: 'tsconfig.eslint.json',17 },18 globals: {19 ...globals.node,20 ...globals.jest,21 },22 },23 plugins: {24 '@typescript-eslint': tsEslint.plugin,25 },26 rules: {27 'no-console': 0,28 },29 },30 {31 // node:test's describe/it return promises by design.32 files: ['test/**/*.ts'],33 rules: {34 '@typescript-eslint/no-floating-promises': 0,35 },36 },37];root = true
[*]indent_style = spaceindent_size = 4charset = utf-8trim_trailing_whitespace = trueinsert_final_newline = trueend_of_line = lf{ "extends": "@apify/tsconfig", "compilerOptions": { "module": "NodeNext", "moduleResolution": "NodeNext", "target": "ES2022", "outDir": "dist", "noUnusedLocals": false, "skipLibCheck": true, "lib": ["DOM"] }, "include": ["./src/**/*"]}{ "name": "company-search", "version": "1.0.0", "type": "module", "description": "Find relevant companies at scale", "engines": { "node": ">=18.0.0" }, "dependencies": { "apify": "^3.7.2", "tomba": "^1.1.1" }, "devDependencies": { "@apify/eslint-config": "^1.0.0", "@apify/tsconfig": "^0.1.1", "@types/node": "^22.15.32", "ajv": "^8.20.0", "eslint": "^9.29.0", "eslint-config-prettier": "^10.1.5", "globals": "^16.2.0", "prettier": "^3.5.3", "standard-version": "^9.5.0", "tsx": "^4.20.3", "typescript": "^5.8.3", "typescript-eslint": "^8.34.1" }, "scripts": { "start": "npm run start:dev", "start:prod": "node dist/main.js", "start:dev": "tsx src/main.ts", "build": "tsc", "lint": "eslint src test", "lint:fix": "eslint src test --fix", "format": "prettier --write .", "format:check": "prettier --check .", "release": "standard-version && git push --follow-tags origin main", "test": "node --import tsx --test test/*.test.ts" }, "author": "It's not you it's me", "license": "ISC"}# Specify the base Docker image. You can read more about# the available images at https://docs.apify.com/sdk/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node:22 AS builder
# Check preinstalled packagesRUN npm ls crawlee apify puppeteer playwright
# Copy just package.json and package-lock.json# to speed up the build using Docker layer cache.COPY package*.json ./
# Install all dependencies. Don't audit to speed up the installation.RUN npm install --include=dev --audit=false
# Next, copy the source files using the user set# in the base image.COPY . ./
# Install all dependencies and build the project.# Don't audit to speed up the installation.RUN npm run build
# Create final imageFROM apify/actor-node:22
# Check preinstalled packagesRUN npm ls crawlee apify puppeteer playwright
# Copy just package.json and package-lock.json# to speed up the build using Docker layer cache.COPY package*.json ./
# Install NPM packages, skip optional and development dependencies to# keep the image small. Avoid logging too much and print the dependency# tree for debuggingRUN npm --quiet set progress=false \ && npm install --omit=dev --omit=optional \ && echo "Installed NPM packages:" \ && (npm list --omit=dev --all || true) \ && echo "Node.js version:" \ && node --version \ && echo "NPM version:" \ && npm --version \ && rm -r ~/.npm
# Copy built JS files from builder imageCOPY /usr/src/app/dist ./dist
# Next, copy the remaining files and directories with the source code.# Since we do this after NPM install, quick build will be really fast# for most source file changes.COPY . ./
# Run the image.CMD npm run start:prod --silent{ "actorSpecification": 1, "fields": { "$schema": "http://json-schema.org/draft-07/schema#", "type": "object", "properties": { "name": { "type": ["string", "null"], "description": "The name of the company" }, "description": { "type": ["string", "null"], "description": "Company description" }, "country": { "type": ["string", "null"], "description": "Country where the company is located" }, "state": { "type": ["string", "null"], "description": "State or region where the company is located" }, "city": { "type": ["string", "null"], "description": "City where the company is located" }, "street_address": { "type": ["string", "null"], "description": "Street address of the company" }, "postal_code": { "type": ["string", "null"], "description": "Postal code of the company location" }, "industry": { "type": ["string", "null"], "description": "Industry sector of the company" }, "company_size": { "type": ["string", "null"], "description": "Employee count range" }, "type": { "type": ["string", "null"], "description": "Type of company" }, "founded": { "type": ["string", "null"], "description": "Year the company was founded" }, "website_url": { "type": ["string", "null"], "description": "Company website URL" }, "total_emails": { "type": ["number", "null"], "description": "Total number of email addresses found for this company" }, "revenue": { "type": ["string", "null"], "description": "Company revenue range" }, "phone_number": { "type": ["boolean", "string", "null"], "description": "Company phone number" }, "linkedin_url": { "type": ["string", "null"], "description": "Company LinkedIn profile URL" }, "facebook_url": { "type": ["string", "null"], "description": "Company Facebook page URL" }, "twitter_url": { "type": ["string", "null"], "description": "Company Twitter profile URL" }, "total_similar": { "type": ["number", "null"], "description": "Number of similar companies found" }, "source": { "type": "string", "description": "Data source identifier" }, "charged": { "type": "boolean", "description": "Whether this request was charged (pay-per-event)" }, "cached": { "type": "boolean", "description": "Whether the result came from the cache (free)" }, "query": { "type": ["string", "null"], "description": "Search query (only on error items)" }, "page": { "type": "integer", "description": "Page that failed (only on error items)" }, "error": { "type": ["string", "null"], "description": "Error message if the search failed or returned no companies" } } }}{ "openapi": "3.0.0", "info": { "title": "Tomba Company Search API", "version": "1.0.0", "description": "Real-time API (Apify Standby mode). Find companies matching a natural language query and/or advanced filters. Each page of results costs $0.00312 (a search matching no companies counts as one page); invalid searches, errors and cached results are free." }, "paths": { "/": { "get": { "summary": "Search companies", "description": "Without query parameters, returns a short usage note (also used as the readiness probe).", "parameters": [ { "name": "query", "in": "query", "description": "Natural language search, e.g. `technology companies in san francisco`. Provide `query`, `filters`, or both.", "required": false, "schema": { "type": "string" }, "example": "technology companies in san francisco" }, { "name": "filters", "in": "query", "description": "Advanced filters as a JSON object (URL-encoded), in the same format as the `filters` input: each filter has `include` and/or `exclude` lists. Available filters: `company`, `location_country`, `location_city`, `location_state`, `industry`, `size`, `type`, `keywords`, `founded`, `technologies`, `similar`, `revenue`, `sic`, `naics`.", "required": false, "schema": { "type": "string" }, "example": "{\"industry\":{\"include\":[\"software\"]},\"location_country\":{\"include\":[\"US\"]}}" }, { "name": "source", "in": "query", "description": "Fields to return for each company. Repeat the parameter or separate values with commas. Leave empty for all fields.", "required": false, "schema": { "type": "string" }, "example": "name,website_url,industry" }, { "name": "page", "in": "query", "description": "Page to start from (default 1).", "required": false, "schema": { "type": "integer", "minimum": 1, "default": 1 } }, { "name": "maxResults", "in": "query", "description": "Maximum number of companies to return (default 100). Each page of results is billed once.", "required": false, "schema": { "type": "integer", "minimum": 1, "maximum": 100000, "default": 100 }, "example": 20 } ], "responses": { "200": { "$ref": "#/components/responses/Results" }, "400": { "$ref": "#/components/responses/Error" }, "402": { "$ref": "#/components/responses/Error" } } }, "post": { "summary": "Search companies using a JSON input", "description": "Accepts the same JSON input as a normal Actor run. Provide `query`, `filters`, or both.", "requestBody": { "required": true, "content": { "application/json": { "schema": { "type": "object", "properties": { "query": { "type": "string", "description": "Natural language query to search for companies (e.g., 'technology companies in san francisco')", "example": "technology companies in san francisco" }, "filters": { "type": "object", "description": "Advanced filters to refine your company search", "properties": { "company": { "type": "object", "description": "Filter by company name", "properties": { "include": { "type": "array", "description": "Company names to include", "items": { "type": "string" } }, "exclude": { "type": "array", "description": "Company names to exclude", "items": { "type": "string" } } } }, "location_country": { "type": "object", "description": "Filter by country using 2-letter ISO country codes (e.g., US, GB, FR, CA, DE)", "properties": { "include": { "type": "array", "description": "Countries to include", "items": { "type": "string" } }, "exclude": { "type": "array", "description": "Countries to exclude", "items": { "type": "string" } } } }, "location_city": { "type": "object", "description": "Filter by city", "properties": { "include": { "type": "array", "description": "Cities to include", "items": { "type": "string" } }, "exclude": { "type": "array", "description": "Cities to exclude", "items": { "type": "string" } } } }, "location_state": { "type": "object", "description": "Filter by state/region", "properties": { "include": { "type": "array", "description": "States to include", "items": { "type": "string" } }, "exclude": { "type": "array", "description": "States to exclude", "items": { "type": "string" } } } }, "industry": { "type": "object", "description": "Filter by industry", "properties": { "include": { "type": "array", "description": "Industries to include", "items": { "type": "string" } }, "exclude": { "type": "array", "description": "Industries to exclude", "items": { "type": "string" } } } }, "size": { "type": "object", "description": "Filter by company size (e.g., '1-10', '11-50', '51-200', '201-500', '501-1000', '1000+')", "properties": { "include": { "type": "array", "description": "Sizes to include", "items": { "type": "string" } }, "exclude": { "type": "array", "description": "Sizes to exclude", "items": { "type": "string" } } } }, "type": { "type": "object", "description": "Filter by company type", "properties": { "include": { "type": "array", "description": "Types to include", "items": { "type": "string" } }, "exclude": { "type": "array", "description": "Types to exclude", "items": { "type": "string" } } } }, "keywords": { "type": "object", "description": "Filter by keywords", "properties": { "include": { "type": "array", "description": "Keywords to include", "items": { "type": "string" } }, "exclude": { "type": "array", "description": "Keywords to exclude", "items": { "type": "string" } } } }, "founded": { "type": "object", "description": "Filter by founding year", "properties": { "include": { "type": "array", "description": "Years to include", "items": { "type": "string" } }, "exclude": { "type": "array", "description": "Years to exclude", "items": { "type": "string" } } } }, "technologies": { "type": "object", "description": "Filter by technologies used", "properties": { "include": { "type": "array", "description": "Technologies to include", "items": { "type": "string" } }, "exclude": { "type": "array", "description": "Technologies to exclude", "items": { "type": "string" } } } }, "similar": { "type": "object", "description": "Filter by similar companies", "properties": { "include": { "type": "array", "description": "Similar companies to include", "items": { "type": "string" } }, "exclude": { "type": "array", "description": "Similar companies to exclude", "items": { "type": "string" } } } }, "revenue": { "type": "object", "description": "Filter by revenue range", "properties": { "include": { "type": "array", "description": "Revenue ranges to include", "items": { "type": "string" } }, "exclude": { "type": "array", "description": "Revenue ranges to exclude", "items": { "type": "string" } } } }, "sic": { "type": "object", "description": "Filter by SIC (Standard Industrial Classification) code", "properties": { "include": { "type": "array", "description": "SIC codes to include", "items": { "type": "string" } }, "exclude": { "type": "array", "description": "SIC codes to exclude", "items": { "type": "string" } } } }, "naics": { "type": "object", "description": "Filter by NAICS (North American Industry Classification System) code", "properties": { "include": { "type": "array", "description": "NAICS codes to include", "items": { "type": "string" } }, "exclude": { "type": "array", "description": "NAICS codes to exclude", "items": { "type": "string" } } } } }, "example": { "industry": { "include": ["software"] }, "size": { "include": ["51-200"] } } }, "source": { "type": "array", "description": "Specify which fields to include in the response. Leave empty for all fields.", "items": { "type": "string" } }, "page": { "type": "integer", "description": "Page number to start from (default: 1)", "minimum": 1, "default": 1 }, "maxResults": { "type": "integer", "description": "Maximum number of companies to return (default: 100)", "minimum": 1, "maximum": 100000, "default": 100 }, "maxConcurrency": { "type": "integer", "description": "How many Tomba requests run in parallel. There is no rate limit, so raise it for large lists.", "minimum": 1, "maximum": 50, "default": 10 }, "maxRetries": { "type": "integer", "description": "How many times to retry a request after a network error, 429 or 5xx response (exponential backoff). Failed requests are never charged.", "minimum": 0, "maximum": 10, "default": 3 }, "useCache": { "type": "boolean", "description": "Reuse results from previous runs for identical requests. Cached results are free.", "default": true }, "cacheTtlHours": { "type": "integer", "description": "How long cached results stay valid. 0 disables the cross-run cache.", "minimum": 0, "maximum": 720, "default": 24 } } } } } }, "responses": { "200": { "$ref": "#/components/responses/Results" }, "400": { "$ref": "#/components/responses/Error" }, "402": { "$ref": "#/components/responses/Error" } } } } }, "components": { "responses": { "Results": { "description": "One row per company, in the same format as the dataset items.", "content": { "application/json": { "schema": { "type": "object", "properties": { "items": { "type": "array", "items": { "$ref": "#/components/schemas/Company" } } } } } } }, "Error": { "description": "Invalid input (400) or the run's max charge limit was reached (402).", "content": { "application/json": { "schema": { "type": "object", "properties": { "error": { "type": "string" } } } } } } }, "schemas": { "Company": { "type": "object", "properties": { "name": { "type": "string", "nullable": true, "description": "The name of the company" }, "description": { "type": "string", "nullable": true, "description": "Company description" }, "country": { "type": "string", "nullable": true, "description": "Country where the company is located" }, "state": { "type": "string", "nullable": true, "description": "State or region where the company is located" }, "city": { "type": "string", "nullable": true, "description": "City where the company is located" }, "street_address": { "type": "string", "nullable": true, "description": "Street address of the company" }, "postal_code": { "type": "string", "nullable": true, "description": "Postal code of the company location" }, "industry": { "type": "string", "nullable": true, "description": "Industry sector of the company" }, "company_size": { "type": "string", "nullable": true, "description": "Employee count range" }, "type": { "type": "string", "nullable": true, "description": "Type of company" }, "founded": { "type": "string", "nullable": true, "description": "Year the company was founded" }, "website_url": { "type": "string", "nullable": true, "description": "Company website URL" }, "total_emails": { "type": "number", "nullable": true, "description": "Total number of email addresses found for this company" }, "revenue": { "type": "string", "nullable": true, "description": "Company revenue range" }, "phone_number": { "type": "string", "nullable": true, "description": "Company phone number" }, "linkedin_url": { "type": "string", "nullable": true, "description": "Company LinkedIn profile URL" }, "facebook_url": { "type": "string", "nullable": true, "description": "Company Facebook page URL" }, "twitter_url": { "type": "string", "nullable": true, "description": "Company Twitter profile URL" }, "total_similar": { "type": "number", "nullable": true, "description": "Number of similar companies found" }, "source": { "type": "string", "description": "Data source identifier" }, "charged": { "type": "boolean", "description": "Whether this request was charged (pay-per-event)" }, "cached": { "type": "boolean", "description": "Whether the result came from the cache (free)" }, "query": { "type": "string", "nullable": true, "description": "Search query (only on error items)" }, "page": { "type": "integer", "description": "Page that failed (only on error items)" }, "error": { "type": "string", "nullable": true, "description": "Error message if the search failed or returned no companies" } } } } }}{ "actorOutputSchemaVersion": 1, "title": "Tomba Company Search Output", "description": "Output from Tomba Company Search Actor - comprehensive company profiles with firmographics and contact information", "type": "object", "properties": { "results": { "type": "string", "title": "Company Information Results", "description": "Detailed company profiles with firmographics, contact details, and social media information", "template": "{{links.apiDefaultDatasetUrl}}/items" } }}{ "actorSpecification": 1, "name": "company-search", "title": "Tomba Company-Search", "description": "Tomba integration for Company Search API.", "version": "1.0", "buildTag": "latest", "meta": { "templateId": "ts-empty" }, "dockerfile": "../Dockerfile", "input": "./input_schema.json", "output": "./output_schema.json", "storages": { "dataset": "./dataset_schema.json", "keyValueStore": "./key_value_store_schema.json" }, "environmentVariables": { "TOMBA_API_KEY": "@tombaApiKey", "TOMBA_API_SECRET": "@tombaApiSecret" }, "usesStandbyMode": true, "webServerSchema": "./web_server_schema.json", "defaultMemoryMbytes": 256}{ "actorKeyValueStoreSchemaVersion": 1, "title": "Tomba Company Search storage", "description": "Records the Actor keeps in its default key-value store. Results are in the dataset.", "collections": { "input": { "title": "Input", "description": "The input of this run.", "key": "INPUT", "contentTypes": ["application/json"] }, "progress": { "title": "Progress", "description": "Result pages already processed, so a migrated or restarted run continues at the next page without charging again.", "key": "TOMBA_STATE", "contentTypes": ["application/json"], "jsonSchema": { "$schema": "http://json-schema.org/draft-07/schema#", "type": "object", "properties": { "done": { "type": "object", "description": "Processed pages (`page:<number>`) mapped to true", "additionalProperties": { "type": "boolean" } } }, "required": ["done"] } }, "companiesSaved": { "title": "Companies saved", "description": "Number of companies saved so far, so a resumed run still honours maxResults.", "key": "COMPANY_SEARCH_PROGRESS", "contentTypes": ["application/json"], "jsonSchema": { "$schema": "http://json-schema.org/draft-07/schema#", "type": "object", "properties": { "pushed": { "type": "integer", "minimum": 0, "description": "Companies saved to the dataset" } }, "required": ["pushed"] } } }}{ "title": "Tomba Company Search Input", "description": "Input schema for Tomba Company Search Actor - Find relevant companies aligned with your ideal customer profile using advanced filtering options.", "type": "object", "schemaVersion": 1, "properties": { "query": { "title": "Search Query", "type": "string", "description": "Natural language query to search for companies (e.g., 'technology companies in san francisco')", "editor": "textfield", "example": "technology companies in san francisco", "prefill": "technology companies in san francisco" }, "filters": { "title": "Company Filters", "type": "object", "description": "Advanced filters to refine your company search", "editor": "json", "prefill": { "location_city": { "include": ["san francisco", "new york"] }, "industry": { "include": ["technology", "software"] }, "size": { "include": ["51-200", "201-500"] } }, "properties": { "company": { "title": "Company Name", "type": "object", "description": "Filter by company name", "editor": "json", "properties": { "include": { "title": "Include", "type": "array", "description": "Company names to include", "editor": "json" }, "exclude": { "title": "Exclude", "type": "array", "description": "Company names to exclude", "editor": "json" } } }, "location_country": { "title": "Country", "type": "object", "description": "Filter by country using 2-letter ISO country codes (e.g., US, GB, FR, CA, DE)", "editor": "json", "properties": { "include": { "title": "Include", "type": "array", "description": "Countries to include", "editor": "json" }, "exclude": { "title": "Exclude", "type": "array", "description": "Countries to exclude", "editor": "json" } } }, "location_city": { "title": "City", "type": "object", "description": "Filter by city", "editor": "json", "properties": { "include": { "title": "Include", "type": "array", "description": "Cities to include", "editor": "json" }, "exclude": { "title": "Exclude", "type": "array", "description": "Cities to exclude", "editor": "json" } } }, "location_state": { "title": "State/Region", "type": "object", "description": "Filter by state/region", "editor": "json", "properties": { "include": { "title": "Include", "type": "array", "description": "States to include", "editor": "json" }, "exclude": { "title": "Exclude", "type": "array", "description": "States to exclude", "editor": "json" } } }, "industry": { "title": "Industry", "type": "object", "description": "Filter by industry", "editor": "json", "properties": { "include": { "title": "Include", "type": "array", "description": "Industries to include", "editor": "json" }, "exclude": { "title": "Exclude", "type": "array", "description": "Industries to exclude", "editor": "json" } } }, "size": { "title": "Company Size", "type": "object", "description": "Filter by company size (e.g., '1-10', '11-50', '51-200', '201-500', '501-1000', '1000+')", "editor": "json", "properties": { "include": { "title": "Include", "type": "array", "description": "Sizes to include", "editor": "json" }, "exclude": { "title": "Exclude", "type": "array", "description": "Sizes to exclude", "editor": "json" } } }, "type": { "title": "Company Type", "type": "object", "description": "Filter by company type", "editor": "json", "properties": { "include": { "title": "Include", "type": "array", "description": "Types to include", "editor": "json" }, "exclude": { "title": "Exclude", "type": "array", "description": "Types to exclude", "editor": "json" } } }, "keywords": { "title": "Keywords", "type": "object", "description": "Filter by keywords", "editor": "json", "properties": { "include": { "title": "Include", "type": "array", "description": "Keywords to include", "editor": "json" }, "exclude": { "title": "Exclude", "type": "array", "description": "Keywords to exclude", "editor": "json" } } }, "founded": { "title": "Founded Year", "type": "object", "description": "Filter by founding year", "editor": "json", "properties": { "include": { "title": "Include", "type": "array", "description": "Years to include", "editor": "json" }, "exclude": { "title": "Exclude", "type": "array", "description": "Years to exclude", "editor": "json" } } }, "technologies": { "title": "Technologies", "type": "object", "description": "Filter by technologies used", "editor": "json", "properties": { "include": { "title": "Include", "type": "array", "description": "Technologies to include", "editor": "json" }, "exclude": { "title": "Exclude", "type": "array", "description": "Technologies to exclude", "editor": "json" } } }, "similar": { "title": "Similar Companies", "type": "object", "description": "Filter by similar companies", "editor": "json", "properties": { "include": { "title": "Include", "type": "array", "description": "Similar companies to include", "editor": "json" }, "exclude": { "title": "Exclude", "type": "array", "description": "Similar companies to exclude", "editor": "json" } } }, "revenue": { "title": "Revenue Range", "type": "object", "description": "Filter by revenue range", "editor": "json", "properties": { "include": { "title": "Include", "type": "array", "description": "Revenue ranges to include", "editor": "json" }, "exclude": { "title": "Exclude", "type": "array", "description": "Revenue ranges to exclude", "editor": "json" } } }, "sic": { "title": "SIC Code", "type": "object", "description": "Filter by SIC (Standard Industrial Classification) code", "editor": "json", "properties": { "include": { "title": "Include", "type": "array", "description": "SIC codes to include", "editor": "json" }, "exclude": { "title": "Exclude", "type": "array", "description": "SIC codes to exclude", "editor": "json" } } }, "naics": { "title": "NAICS Code", "type": "object", "description": "Filter by NAICS (North American Industry Classification System) code", "editor": "json", "properties": { "include": { "title": "Include", "type": "array", "description": "NAICS codes to include", "editor": "json" }, "exclude": { "title": "Exclude", "type": "array", "description": "NAICS codes to exclude", "editor": "json" } } } } }, "source": { "title": "Source Fields", "type": "array", "description": "Specify which fields to include in the response. Leave empty for all fields.", "editor": "json", "items": { "type": "string" }, "example": ["name", "website_url", "industry", "city", "country"] }, "page": { "title": "Starting Page", "type": "integer", "description": "Page number to start from (default: 1)", "editor": "number", "default": 1, "minimum": 1, "prefill": 1 }, "maxResults": { "title": "Maximum Results", "type": "integer", "description": "Maximum number of companies to return (default: 100)", "editor": "number", "default": 100, "minimum": 1, "maximum": 100000, "prefill": 50 }, "maxConcurrency": { "title": "Max concurrency", "type": "integer", "description": "How many Tomba requests run in parallel. There is no rate limit, so raise it for large lists.", "minimum": 1, "maximum": 50, "default": 10, "editor": "number", "sectionCaption": "Run options", "sectionDescription": "Concurrency, retry and cache settings. The defaults work well for most runs." }, "maxRetries": { "title": "Max retries", "type": "integer", "description": "How many times to retry a request after a network error, 429 or 5xx response (exponential backoff). Failed requests are never charged.", "minimum": 0, "maximum": 10, "default": 3, "editor": "number" }, "useCache": { "title": "Use cache", "type": "boolean", "description": "Reuse results from previous runs for identical requests. Cached results are free.", "default": true, "editor": "checkbox" }, "cacheTtlHours": { "title": "Cache TTL (hours)", "type": "integer", "description": "How long cached results stay valid. 0 disables the cross-run cache.", "minimum": 0, "maximum": 720, "default": 24, "editor": "number", "unit": "hours" } }}1// Shared test harness for the Tomba Apify Actors.2// Keep this file identical across all Actors.3import { spawn } from 'node:child_process';4import { readFileSync } from 'node:fs';5import { mkdir, mkdtemp, readdir, readFile, rm, writeFile } from 'node:fs/promises';6import type { IncomingMessage, Server, ServerResponse } from 'node:http';7import { createServer, request as httpRequest } from 'node:http';8import type { AddressInfo } from 'node:net';9import { tmpdir } from 'node:os';10import { join } from 'node:path';11
12import { Ajv } from 'ajv';13
14export interface MockRequest {15 method: string;16 path: string;17 query: Record<string, string>;18 headers: IncomingMessage['headers'];19 body: unknown;20}21
22export interface MockResponse {23 status?: number;24 body?: unknown;25 /** Raw (non-JSON) body. */26 raw?: string;27 headers?: Record<string, string>;28}29
30export type MockHandler = (req: MockRequest) => MockResponse | Promise<MockResponse>;31
32export interface MockServer {33 url: string;34 requests: MockRequest[];35 close: () => Promise<void>;36}37
38/** Start a local HTTP server that impersonates the Tomba API. */39export async function startMockTomba(handler: MockHandler): Promise<MockServer> {40 const requests: MockRequest[] = [];41 const server: Server = createServer((req: IncomingMessage, res: ServerResponse) => {42 const chunks: Buffer[] = [];43 req.on('data', (chunk: Buffer) => chunks.push(chunk));44 req.on('end', async () => {45 const url = new URL(req.url ?? '/', 'http://localhost');46 const text = Buffer.concat(chunks).toString();47 let body: unknown = text;48 try {49 body = text ? JSON.parse(text) : undefined;50 } catch {51 // keep raw text52 }53 const request: MockRequest = {54 method: req.method ?? 'GET',55 path: url.pathname,56 query: Object.fromEntries(url.searchParams),57 headers: req.headers,58 body,59 };60 requests.push(request);61 try {62 const out = await handler(request);63 const status = out.status ?? 200;64 if (out.raw !== undefined) {65 res.writeHead(status, { 'content-type': 'text/html', ...out.headers });66 res.end(out.raw);67 } else {68 res.writeHead(status, { 'content-type': 'application/json', ...out.headers });69 res.end(JSON.stringify(out.body ?? {}));70 }71 } catch (err) {72 res.writeHead(500, { 'content-type': 'application/json' });73 res.end(JSON.stringify({ errors: { message: (err as Error).message } }));74 }75 });76 });77 await new Promise<void>((resolve) => {78 server.listen(0, '127.0.0.1', resolve);79 });80 const { port } = server.address() as AddressInfo;81 return {82 url: `http://127.0.0.1:${port}`,83 requests,84 close: async () =>85 new Promise<void>((resolve) => {86 server.close(() => resolve());87 }),88 };89}90
91export interface RunOptions {92 input: Record<string, unknown>;93 /** Mock server URL; omit to run without an endpoint override. */94 endpoint?: string;95 /** Reuse a storage directory (to test cache and resume). */96 storageDir?: string;97 /** Keep the default storages from the previous run in `storageDir`. */98 keepStorage?: boolean;99 /** Max total charge in USD (locally every event costs $1). */100 maxTotalChargeUsd?: number;101 /** Set to false to run without Tomba credentials. */102 withCredentials?: boolean;103 timeoutMs?: number;104}105
106export interface ChargeLogEntry {107 eventName: string;108 chargedCount?: number;109 [key: string]: unknown;110}111
112export interface RunResult {113 code: number | null;114 output: string;115 items: Record<string, unknown>[];116 charges: ChargeLogEntry[];117 storageDir: string;118 /** Number of charged events per event name. */119 chargeCounts: Record<string, number>;120}121
122/** Run the Actor (`src/main.ts`) in a child process against local storage. */123export async function runActor(options: RunOptions): Promise<RunResult> {124 const storageDir = options.storageDir ?? (await mkdtemp(join(tmpdir(), 'tomba-actor-test-')));125 const inputDir = join(storageDir, 'key_value_stores', 'default');126 await mkdir(inputDir, { recursive: true });127 await writeFile(join(inputDir, 'INPUT.json'), JSON.stringify(options.input));128
129 const env: NodeJS.ProcessEnv = {130 PATH: process.env.PATH,131 HOME: process.env.HOME,132 CRAWLEE_STORAGE_DIR: storageDir,133 CRAWLEE_PURGE_ON_START: options.keepStorage ? '0' : '1',134 APIFY_LOG_LEVEL: 'INFO',135 ACTOR_TEST_PAY_PER_EVENT: 'true',136 ACTOR_USE_CHARGING_LOG_DATASET: 'true',137 };138 if (options.withCredentials !== false) {139 env.TOMBA_API_KEY = 'ta_test_key';140 env.TOMBA_API_SECRET = 'ts_test_secret';141 }142 if (options.endpoint) env.TOMBA_API_ENDPOINT = options.endpoint;143 if (options.maxTotalChargeUsd !== undefined) env.ACTOR_MAX_TOTAL_CHARGE_USD = String(options.maxTotalChargeUsd);144
145 const { code, output } = await new Promise<{ code: number | null; output: string }>((resolve, reject) => {146 const child = spawn(process.execPath, ['--import', 'tsx', 'src/main.ts'], { env, cwd: process.cwd() });147 let out = '';148 child.stdout.on('data', (d: Buffer) => {149 out += d.toString();150 });151 child.stderr.on('data', (d: Buffer) => {152 out += d.toString();153 });154 const timer = setTimeout(() => {155 child.kill('SIGKILL');156 reject(new Error(`Actor run timed out\n${out}`));157 }, options.timeoutMs ?? 60_000);158 child.on('close', (c) => {159 clearTimeout(timer);160 resolve({ code: c, output: out });161 });162 });163
164 const items = await readDataset(storageDir, 'default');165 assertMatchesDatasetSchema(items);166 const charges = (await readDataset(storageDir, 'charging_log')) as ChargeLogEntry[];167 const chargeCounts: Record<string, number> = {};168 for (const c of charges) chargeCounts[c.eventName] = (chargeCounts[c.eventName] ?? 0) + (c.chargedCount ?? 1);169
170 return { code, output, items, charges, storageDir, chargeCounts };171}172
173export async function readDataset(storageDir: string, name: string): Promise<Record<string, unknown>[]> {174 const dir = join(storageDir, 'datasets', name);175 let files: string[];176 try {177 files = (await readdir(dir)).filter((f) => f.endsWith('.json')).sort();178 } catch {179 return [];180 }181 return Promise.all(files.map(async (f) => JSON.parse(await readFile(join(dir, f), 'utf8'))));182}183
184export async function removeStorage(storageDir: string): Promise<void> {185 await rm(storageDir, { recursive: true, force: true });186}187
188/** Total number of charged events across all event names. */189export function totalCharges(result: RunResult): number {190 return Object.values(result.chargeCounts).reduce((a, b) => a + b, 0);191}192
193export interface StandbyActor {194 /** Base URL of the Actor's HTTP server. */195 url: string;196 storageDir: string;197 /** GET or POST a request to the Actor and parse the JSON response. */198 call: (199 path: string,200 init?: { method?: string; body?: unknown; headers?: Record<string, string> },201 ) => Promise<{ status: number; body: Record<string, unknown> }>;202 /** Stop the Actor and return its log output and charges. */203 stop: () => Promise<{ output: string; chargeCounts: Record<string, number> }>;204}205
206async function freePort(): Promise<number> {207 const server = createServer();208 await new Promise<void>((resolve) => {209 server.listen(0, '127.0.0.1', resolve);210 });211 const { port } = server.address() as AddressInfo;212 await new Promise<void>((resolve) => {213 server.close(() => resolve());214 });215 return port;216}217
218async function httpCall(219 url: string,220 init: { method?: string; body?: unknown; headers?: Record<string, string> } = {},221): Promise<{ status: number; body: Record<string, unknown> }> {222 let payload: string | undefined;223 if (typeof init.body === 'string') payload = init.body;224 else if (init.body !== undefined) payload = JSON.stringify(init.body);225 return new Promise((resolve, reject) => {226 const req = httpRequest(227 url,228 {229 method: init.method ?? (payload ? 'POST' : 'GET'),230 headers: { 'content-type': 'application/json', ...init.headers },231 },232 (res) => {233 const chunks: Buffer[] = [];234 res.on('data', (c: Buffer) => chunks.push(c));235 res.on('end', () => {236 const text = Buffer.concat(chunks).toString();237 let body: Record<string, unknown> = {};238 try {239 body = JSON.parse(text);240 } catch {241 body = { raw: text };242 }243 resolve({ status: res.statusCode ?? 0, body });244 });245 },246 );247 req.on('error', reject);248 if (payload) req.write(payload);249 req.end();250 });251}252
253/** Start the Actor in Standby mode (real-time HTTP API) and wait until it is ready. */254export async function startStandbyActor(255 options: { endpoint?: string; maxTotalChargeUsd?: number; withCredentials?: boolean } = {},256): Promise<StandbyActor> {257 const storageDir = await mkdtemp(join(tmpdir(), 'tomba-standby-test-'));258 const port = await freePort();259 const env: NodeJS.ProcessEnv = {260 PATH: process.env.PATH,261 HOME: process.env.HOME,262 CRAWLEE_STORAGE_DIR: storageDir,263 APIFY_LOG_LEVEL: 'INFO',264 APIFY_META_ORIGIN: 'STANDBY',265 ACTOR_WEB_SERVER_PORT: String(port),266 ACTOR_TEST_PAY_PER_EVENT: 'true',267 ACTOR_USE_CHARGING_LOG_DATASET: 'true',268 };269 if (options.withCredentials !== false) {270 env.TOMBA_API_KEY = 'ta_test_key';271 env.TOMBA_API_SECRET = 'ts_test_secret';272 }273 if (options.endpoint) env.TOMBA_API_ENDPOINT = options.endpoint;274 if (options.maxTotalChargeUsd !== undefined) env.ACTOR_MAX_TOTAL_CHARGE_USD = String(options.maxTotalChargeUsd);275
276 const child = spawn(process.execPath, ['--import', 'tsx', 'src/main.ts'], { env, cwd: process.cwd() });277 let output = '';278 child.stdout.on('data', (d: Buffer) => {279 output += d.toString();280 });281 child.stderr.on('data', (d: Buffer) => {282 output += d.toString();283 });284 const exited = new Promise<void>((resolve) => {285 child.on('close', () => resolve());286 });287
288 const url = `http://127.0.0.1:${port}`;289 const deadline = Date.now() + 30_000;290 for (;;) {291 if (child.exitCode !== null) throw new Error(`Standby Actor exited early\n${output}`);292 try {293 const res = await httpCall(`${url}/`, { headers: { 'x-apify-container-server-readiness-probe': '1' } });294 if (res.status === 200) break;295 } catch {296 // not listening yet297 }298 if (Date.now() > deadline) {299 child.kill('SIGKILL');300 throw new Error(`Standby Actor did not become ready\n${output}`);301 }302 await new Promise((r) => {303 setTimeout(r, 100);304 });305 }306
307 return {308 url,309 storageDir,310 call: async (path, init) => httpCall(`${url}${path}`, init),311 stop: async () => {312 child.kill('SIGTERM');313 await exited;314 const charges = (await readDataset(storageDir, 'charging_log')) as ChargeLogEntry[];315 const chargeCounts: Record<string, number> = {};316 for (const c of charges)317 chargeCounts[c.eventName] = (chargeCounts[c.eventName] ?? 0) + (c.chargedCount ?? 1);318 await removeStorage(storageDir);319 return { output, chargeCounts };320 },321 };322}323
324let datasetValidator: ReturnType<Ajv['compile']> | undefined;325
326/**327 * Apify validates every pushed item against `.actor/dataset_schema.json` and fails the run on a mismatch,328 * while local runs don't. Check the items the same way so schema mismatches fail the tests.329 */330export function assertMatchesDatasetSchema(items: Record<string, unknown>[]): void {331 if (!datasetValidator) {332 const { fields } = JSON.parse(readFileSync(join(process.cwd(), '.actor/dataset_schema.json'), 'utf8'));333 delete fields.$schema;334 datasetValidator = new Ajv({ allErrors: true, strict: false }).compile(fields);335 }336 for (const [index, item] of items.entries()) {337 if (!datasetValidator(item)) {338 const errors = (datasetValidator.errors ?? [])339 .map((e) => `${e.instancePath || '/'} ${e.message}`)340 .join('; ');341 throw new Error(342 `Dataset item ${index} doesn't match .actor/dataset_schema.json: ${errors}\n${JSON.stringify(item)}`,343 );344 }345 }346}347
348/**349 * Validate an input against `.actor/input_schema.json` (Apify validates the input before a run or API call350 * starts). Returns the error messages; empty when the input is accepted.351 */352export function inputSchemaErrors(input: Record<string, unknown>): string[] {353 const schema = JSON.parse(readFileSync(join(process.cwd(), '.actor/input_schema.json'), 'utf8'));354 const validate = new Ajv({ allErrors: true, strict: false }).compile({355 type: 'object',356 properties: schema.properties,357 required: schema.required ?? [],358 });359 return validate(input) ? [] : (validate.errors ?? []).map((e) => `${e.instancePath || '/'} ${e.message}`);360}1// Unit tests for src/tomba.ts. Keep this file identical across all Actors.2import assert from 'node:assert/strict';3import { mkdtemp } from 'node:fs/promises';4import { tmpdir } from 'node:os';5import { join } from 'node:path';6import { after, before, describe, it } from 'node:test';7
8process.env.CRAWLEE_STORAGE_DIR = await mkdtemp(join(tmpdir(), 'tomba-unit-'));9process.env.ACTOR_TEST_PAY_PER_EVENT = 'true';10process.env.TOMBA_API_KEY = 'ta_unit';11process.env.TOMBA_API_SECRET = 'ts_unit';12
13const { Actor } = await import('apify');14const { TombaException } = await import('tomba');15const T = await import('../src/tomba.js');16
17const ok = (body: unknown) => async () => ({ data: body, rateLimit: {} });18const httpError = (status: number, body: unknown = {}, headers: Record<string, string> = {}) =>19 new TombaException('error', { status, headers }, body);20const charged = (event = T.EVENT_REQUEST) => Actor.getChargingManager().getChargedEventCount(event);21
22let seq = 0;23const params = () => ({ n: ++seq });24
25before(async () => {26 await Actor.init();27 await T.setupTomba({ maxRetries: 1, useCache: true, cacheTtlHours: 1, maxConcurrency: 4 });28});29
30after(async () => {31 await Actor.exit({ exit: false });32});33
34describe('isBillable', () => {35 const cases: [string, unknown, boolean][] = [36 ['data array', { data: [{ a: 1 }] }, true],37 ['data object', { data: { email: 'a@b.co' } }, true],38 ['negative answer (object with empty list)', { data: { organization: {}, emails: [] } }, true],39 ['negative answer (undeliverable)', { data: { email: { status: 'invalid' } } }, true],40 ['empty array', { data: [] }, false],41 ['empty object', { data: {} }, false],42 ['object with only null values', { data: { person: null, company: null } }, false],43 ['negative answer (email null, other fields set)', { data: { email: null, first_name: 'John' } }, true],44 ['null data', { data: null }, false],45 ['missing data', { meta: {} }, false],46 ['errors object', { data: { a: 1 }, errors: { message: 'x' } }, false],47 ['string body', '<html>', false],48 ['null body', null, false],49 ['array body', [1, 2], false],50 ];51 for (const [name, body, expected] of cases) {52 it(name, () => assert.equal(T.isBillable(body), expected));53 }54});55
56describe('normalizers', () => {57 it('normalizeDomain strips protocol, www, path and case', () => {58 assert.equal(T.normalizeDomain(' https://www.Stripe.com/about?x=1 '), 'stripe.com');59 assert.equal(T.normalizeDomain('http://tomba.io'), 'tomba.io');60 assert.equal(T.normalizeDomain('sub.example.org'), 'sub.example.org');61 });62
63 it('normalizeEmail trims and lowercases', () => {64 assert.equal(T.normalizeEmail(' John@Stripe.COM '), 'john@stripe.com');65 });66
67 it('unique keeps first occurrence and drops empty keys', () => {68 assert.deepEqual(T.unique(['a', 'b', 'a', '', 'c']), ['a', 'b', 'c']);69 assert.deepEqual(70 T.unique([{ id: 1 }, { id: 2 }, { id: 1 }], (x) => String(x.id)),71 [{ id: 1 }, { id: 2 }],72 );73 });74});75
76describe('phone data', () => {77 it('counts phone numbers in phone_data', () => {78 assert.equal(T.phoneDataCount({ phone_data: [{ number: '+1' }, { number: '+2' }] }), 2);79 assert.equal(T.phoneDataCount({ phone_data: [] }), 0);80 assert.equal(T.phoneDataCount({ phone_number: true }), 0);81 assert.equal(T.phoneDataCount(null), 0);82 assert.equal(T.phoneDataCount('x'), 0);83 });84
85 it('detects records with phone data', () => {86 assert.equal(T.hasPhoneData({ phone_data: [{ number: '+1' }] }), true);87 assert.equal(T.hasPhoneData({ phone_data: [] }), false);88 assert.equal(T.hasPhoneData({}), false);89 });90
91 it('prices phone data at 5 credits', () => {92 assert.equal(T.PHONE_CREDITS, 5);93 });94});95
96describe('cacheStoreName', () => {97 it("is unique per Actor so limited-permission runs never open another Actor's store", () => {98 const previous = process.env.ACTOR_ID;99 try {100 process.env.ACTOR_ID = 'AbC123xyz';101 assert.equal(T.cacheStoreName(), 'tomba-cache-AbC123xyz');102 delete process.env.ACTOR_ID;103 assert.equal(T.cacheStoreName(), 'tomba-cache');104 } finally {105 if (previous === undefined) delete process.env.ACTOR_ID;106 else process.env.ACTOR_ID = previous;107 }108 });109});110
111describe('setupTomba without cache permissions', () => {112 it('keeps working with an in-run cache when the cache store cannot be opened', async () => {113 const original = Actor.openKeyValueStore.bind(Actor);114 Actor.openKeyValueStore = async () => {115 throw new Error('Insufficient permissions for the key-value store.');116 };117 try {118 await T.setupTomba({ maxRetries: 1, useCache: true, cacheTtlHours: 1, maxConcurrency: 4 });119 const p = { cachePermissionTest: true };120 let calls = 0;121 const fn = async () => {122 calls++;123 return { data: { data: [1] }, rateLimit: {} };124 };125 const first = await T.callTomba('perm', p, fn);126 const second = await T.callTomba('perm', p, fn);127 assert.equal(first.charged, true);128 assert.equal(second.cached, true);129 assert.equal(calls, 1);130 } finally {131 Actor.openKeyValueStore = original;132 await T.setupTomba({ maxRetries: 1, useCache: true, cacheTtlHours: 1, maxConcurrency: 4 });133 }134 });135});136
137describe('runPool', () => {138 it('processes every item with bounded concurrency', async () => {139 let active = 0;140 let peak = 0;141 const seen: number[] = [];142 await T.runPool(143 Array.from({ length: 20 }, (_, i) => i),144 async (item) => {145 active++;146 peak = Math.max(peak, active);147 await new Promise((r) => {148 setTimeout(r, 5);149 });150 seen.push(item);151 active--;152 },153 3,154 );155 assert.equal(seen.length, 20);156 assert.deepEqual(157 [...seen].sort((a, b) => a - b),158 Array.from({ length: 20 }, (_, i) => i),159 );160 assert.ok(peak <= 3, `peak concurrency ${peak}`);161 });162
163 it('handles an empty list', async () => {164 await T.runPool([], async () => assert.fail('should not run'));165 });166});167
168describe('callTomba', () => {169 it('charges a response with data and unwraps the SDK { data, rateLimit } shape', async () => {170 const start = charged();171 const res = await T.callTomba('test', params(), ok({ data: [{ name: 'React' }] }));172 assert.equal(res.status, 200);173 assert.equal(res.charged, true);174 assert.equal(res.cached, false);175 assert.deepEqual(res.data, [{ name: 'React' }]);176 assert.equal(charged(), start + 1);177 });178
179 it('accepts an SDK that returns the body directly', async () => {180 const res = await T.callTomba('test', params(), async () => ({ data: { x: 1 } }));181 assert.equal(res.charged, true);182 assert.deepEqual(res.data, { x: 1 });183 });184
185 it('charges a negative answer', async () => {186 const res = await T.callTomba('test', params(), ok({ data: { organization: {}, emails: [] } }));187 assert.equal(res.charged, true);188 });189
190 it('serves a repeated request from cache without charging', async () => {191 const p = params();192 let calls = 0;193 const fn = async () => {194 calls++;195 return { data: { data: { v: 1 } }, rateLimit: {} };196 };197 await T.callTomba('test', p, fn);198 const start = charged();199 const res = await T.callTomba('test', { ...p }, fn);200 assert.equal(res.cached, true);201 assert.equal(res.charged, false);202 assert.deepEqual(res.data, { v: 1 });203 assert.equal(calls, 1);204 assert.equal(charged(), start);205 });206
207 it('uses a different cache entry for different params or endpoints', async () => {208 const p = params();209 await T.callTomba('a', p, ok({ data: [1] }));210 const other = await T.callTomba('b', p, ok({ data: [2] }));211 assert.equal(other.cached, false);212 assert.deepEqual(other.data, [2]);213 });214
215 it('does not charge or cache empty data', async () => {216 const p = params();217 const start = charged();218 const res = await T.callTomba('test', p, ok({ data: [] }));219 assert.equal(res.charged, false);220 assert.equal(res.status, 200);221 const again = await T.callTomba('test', p, ok({ data: [] }));222 assert.equal(again.cached, false);223 assert.equal(charged(), start);224 });225
226 it('does not charge a success with an errors object', async () => {227 const res = await T.callTomba('test', params(), ok({ data: { a: 1 }, errors: { message: 'Bad thing' } }));228 assert.equal(res.charged, false);229 assert.equal(res.error, 'Bad thing');230 });231
232 it('reports a non-JSON body as 502 without charging', async () => {233 const res = await T.callTomba('test', params(), ok('<html>oops</html>'));234 assert.equal(res.status, 502);235 assert.equal(res.charged, false);236 });237
238 it('does not retry or charge a 422', async () => {239 let calls = 0;240 const res = await T.callTomba('test', params(), async () => {241 calls++;242 throw httpError(422, { errors: { message: 'Invalid domain' } });243 });244 assert.equal(calls, 1);245 assert.equal(res.status, 422);246 assert.equal(res.charged, false);247 assert.match(res.error ?? '', /Invalid domain/);248 });249
250 it('does not retry SDK input validation errors', async () => {251 let calls = 0;252 const res = await T.callTomba('test', params(), async () => {253 calls++;254 throw new TombaException('Domain is required.');255 });256 assert.equal(calls, 1);257 assert.equal(res.status, 400);258 assert.equal(res.charged, false);259 });260
261 it('retries a 429 honoring Retry-After, then charges the success', async () => {262 let calls = 0;263 const res = await T.callTomba('test', params(), async () => {264 calls++;265 if (calls === 1) throw httpError(429, {}, { 'retry-after': '1' });266 return { data: { data: { ok: true } }, rateLimit: {} };267 });268 assert.equal(calls, 2);269 assert.equal(res.charged, true);270 });271
272 it('retries network errors', async () => {273 let calls = 0;274 const res = await T.callTomba('test', params(), async () => {275 calls++;276 if (calls === 1) throw new TombaException('connect ECONNREFUSED 127.0.0.1:9');277 return { data: { data: [1] }, rateLimit: {} };278 });279 assert.equal(calls, 2);280 assert.equal(res.charged, true);281 });282
283 it('gives up after maxRetries on 5xx without charging', async () => {284 let calls = 0;285 const res = await T.callTomba('test', params(), async () => {286 calls++;287 throw httpError(503);288 });289 assert.equal(calls, 2); // 1 attempt + maxRetries (1)290 assert.equal(res.status, 503);291 assert.equal(res.charged, false);292 });293
294 it('charges the given event name', async () => {295 const start = charged(T.EVENT_PHONE_FINDER);296 await T.callTomba('phone', params(), ok({ data: { phone: '+1' } }), T.EVENT_PHONE_FINDER);297 assert.equal(charged(T.EVENT_PHONE_FINDER), start + 1);298 });299
300 it('charges a fixed count of events', async () => {301 const start = charged();302 const res = await T.callTomba('test', params(), ok({ data: [1] }), T.EVENT_REQUEST, 3);303 assert.equal(res.chargedCount, 3);304 assert.equal(charged(), start + 3);305 });306
307 it('charges a count computed from the response body', async () => {308 const start = charged();309 const res = await T.callTomba(310 'test',311 params(),312 ok({ data: { items: [1, 2, 3, 4] } }),313 T.EVENT_REQUEST,314 (body) => (body.data as { items: unknown[] }).items.length,315 );316 assert.equal(res.chargedCount, 4);317 assert.equal(charged(), start + 4);318 });319
320 it('does not charge a count for a non-billable response', async () => {321 const start = charged();322 const res = await T.callTomba('test', params(), ok({ data: [] }), T.EVENT_REQUEST, 5);323 assert.equal(res.charged, false);324 assert.equal(charged(), start);325 });326
327 it('tracks stats', () => {328 assert.ok(T.stats.requests > 0);329 assert.ok(T.stats.charged > 0);330 assert.ok(T.stats.cached > 0);331 assert.ok(T.stats.retries > 0);332 assert.ok(T.stats.failed > 0);333 });334});1// End-to-end tests: run the Actor against a mock Tomba API.2import assert from 'node:assert/strict';3import { after, afterEach, describe, it } from 'node:test';4
5import type { MockHandler, MockRequest, MockServer } from './helpers.js';6import { removeStorage, runActor, startMockTomba, startStandbyActor, totalCharges } from './helpers.js';7
8const PAGE_SIZE = 10;9
10function company(n: number) {11 return {12 name: `Company ${n}`,13 description: 'Leading provider of innovative software solutions',14 country: 'US',15 state: 'California',16 street_address: '123 Market Street',17 postal_code: '94103',18 industry: 'Software',19 company_size: '51-200',20 type: 'privately held',21 founded: '2015',22 city: 'San Francisco',23 website_url: `company${n}.com`,24 total_emails: 150,25 revenue: '$10M-$50M',26 phone_number: '+1-415-555-0100',27 linkedin_url: `https://www.linkedin.com/company/company${n}`,28 facebook_url: `https://www.facebook.com/company${n}`,29 twitter_url: `https://twitter.com/company${n}`,30 total_similar: 25,31 };32}33
34/** Number of companies matching each test query (default 25). */35const TOTALS: Record<string, number> = { big: 60, exact: 20, none: 0, nopages: 15 };36
37interface SearchRequest {38 query?: string;39 filters?: Record<string, unknown>;40 _source?: string[];41 page?: number;42}43
44/** A realistic paged Companies Search response. */45function searchPage(params: SearchRequest) {46 const query = params.query ?? 'filters';47 const total = TOTALS[query] ?? 25;48 const page = params.page ?? 1;49 const first = (page - 1) * PAGE_SIZE;50 const companies = Array.from({ length: Math.max(0, Math.min(PAGE_SIZE, total - first)) }, (_, i) =>51 company(first + i + 1),52 );53 const pages = Math.ceil(total / PAGE_SIZE);54 return {55 success: true,56 data: { companies },57 meta: {58 total,59 page,60 limit: PAGE_SIZE,61 ...(query === 'nopages' ? {} : { pages }),62 ...(params.filters ? { filters: params.filters } : {}),63 },64 };65}66
67const body = (req: MockRequest) => req.body as SearchRequest;68
69/** Default Tomba behaviour. */70const tomba: MockHandler = (req) => {71 assert.equal(req.method, 'POST');72 assert.equal(req.path, '/reveal/search');73 const params = body(req);74 if (params.query === 'empty') return { body: { data: null } };75 if (params.query === 'invalid') return { status: 422, body: { errors: { message: 'Invalid query' } } };76 if (params.query === 'html') return { raw: '<html>Bad gateway</html>' };77 return { body: searchPage(params) };78};79
80const servers: MockServer[] = [];81const dirs: string[] = [];82
83async function mock(handler: MockHandler = tomba): Promise<MockServer> {84 const server = await startMockTomba(handler);85 servers.push(server);86 return server;87}88
89async function run(...args: Parameters<typeof runActor>) {90 const result = await runActor(...args);91 dirs.push(result.storageDir);92 return result;93}94
95const pagesRequested = (server: MockServer) => server.requests.map((r) => body(r).page);96
97afterEach(async () => {98 await Promise.all(servers.splice(0).map(async (s) => s.close()));99});100
101after(async () => {102 await Promise.all(dirs.map(removeStorage));103});104
105describe('company-search', () => {106 it('returns one item per company and charges one event per billable page', async () => {107 const server = await mock();108 const result = await run({ input: { query: 'tech', maxResults: 10 }, endpoint: server.url });109
110 assert.equal(result.code, 0, result.output);111 assert.equal(result.items.length, 10);112 assert.deepEqual(result.items[0], {113 ...company(1),114 source: 'tomba_company_search',115 charged: true,116 cached: false,117 });118 assert.deepEqual(server.requests[0].body, { query: 'tech', page: 1 });119 assert.deepEqual(result.chargeCounts, { 'tomba-request': 1 });120 });121
122 it('sends the built-in credentials to Tomba', async () => {123 const server = await mock();124 await run({ input: { query: 'tech', maxResults: 1 }, endpoint: server.url });125 assert.equal(server.requests[0].headers['x-tomba-key'], 'ta_test_key');126 assert.equal(server.requests[0].headers['x-tomba-secret'], 'ts_test_secret');127 });128
129 it('trims the query and drops empty filters and fields', async () => {130 const server = await mock();131 await run({132 input: { query: ' tech ', filters: {}, source: [], maxResults: 1 },133 endpoint: server.url,134 });135 assert.deepEqual(server.requests[0].body, { query: 'tech', page: 1 });136 });137
138 it('passes filters, fields and the start page to Tomba', async () => {139 const server = await mock();140 const filters = { location_country: { include: ['US'] }, size: { exclude: ['1-10'] } };141 const result = await run({142 input: { filters, source: ['name', 'website_url'], page: 2, maxResults: 5 },143 endpoint: server.url,144 });145 assert.deepEqual(server.requests[0].body, { filters, _source: ['name', 'website_url'], page: 2 });146 assert.equal(result.items[0].name, 'Company 11');147 });148
149 it('pages until maxResults is reached, one request and one charge per page', async () => {150 const server = await mock();151 const result = await run({ input: { query: 'tech', maxResults: 15 }, endpoint: server.url });152 assert.deepEqual(pagesRequested(server), [1, 2]);153 assert.equal(result.items.length, 15);154 assert.equal(result.items[14].name, 'Company 15');155 assert.deepEqual(result.chargeCounts, { 'tomba-request': 2 });156 });157
158 it('stops paging at the last page reported by Tomba', async () => {159 const server = await mock();160 const result = await run({ input: { query: 'exact', maxResults: 1000 }, endpoint: server.url });161 // 20 companies, 2 full pages: meta.pages stops paging without requesting an empty page 3.162 assert.deepEqual(pagesRequested(server), [1, 2]);163 assert.equal(result.items.length, 20);164 assert.equal(totalCharges(result), 2);165 });166
167 it('stops paging at a short page', async () => {168 const server = await mock();169 const result = await run({ input: { query: 'nopages', maxResults: 1000 }, endpoint: server.url });170 assert.deepEqual(pagesRequested(server), [1, 2]);171 assert.equal(result.items.length, 15);172 assert.equal(totalCharges(result), 2);173 });174
175 it('keeps the companies it already paid for when a later page fails', async () => {176 const server = await mock(async (req) =>177 body(req).page === 2 ? { status: 422, body: { errors: { message: 'Bad page' } } } : tomba(req),178 );179 const result = await run({ input: { query: 'tech', maxResults: 100 }, endpoint: server.url });180 assert.equal(result.code, 0, result.output);181 assert.deepEqual(pagesRequested(server), [1, 2]);182 assert.equal(result.items.length, 10);183 assert.ok(result.items.every((i) => i.charged === true && i.error === undefined));184 assert.equal(totalCharges(result), 1);185 });186
187 it('charges a search with no matching companies as a negative answer', async () => {188 const server = await mock();189 const result = await run({ input: { query: 'none' }, endpoint: server.url });190 assert.equal(server.requests.length, 1);191 assert.deepEqual(result.items, [192 {193 query: 'none',194 page: 1,195 source: 'tomba_company_search',196 charged: true,197 cached: false,198 error: 'No companies found',199 },200 ]);201 assert.deepEqual(result.chargeCounts, { 'tomba-request': 1 });202 });203
204 it('does not charge empty data', async () => {205 const server = await mock();206 const result = await run({ input: { query: 'empty' }, endpoint: server.url });207 assert.equal(result.code, 0, result.output);208 assert.equal(result.items.length, 1);209 assert.equal(result.items[0].charged, false);210 assert.equal(result.items[0].error, 'No companies found');211 assert.equal(totalCharges(result), 0);212 });213
214 it('does not charge Tomba error statuses and does not retry them', async () => {215 const server = await mock();216 const result = await run({ input: { query: 'invalid' }, endpoint: server.url });217 assert.equal(result.code, 0, result.output);218 assert.equal(server.requests.length, 1);219 assert.equal(result.items[0].charged, false);220 assert.match(String(result.items[0].error), /422: Invalid query/);221 assert.equal(totalCharges(result), 0);222 });223
224 it('does not charge a non-JSON body', async () => {225 const server = await mock();226 const result = await run({ input: { query: 'html' }, endpoint: server.url });227 assert.equal(result.items[0].charged, false);228 assert.match(String(result.items[0].error), /Invalid response/);229 assert.equal(totalCharges(result), 0);230 });231
232 it('retries 429 and 5xx responses, then charges the success once', async () => {233 let calls = 0;234 const server = await mock(async (req) => {235 calls++;236 if (calls === 1)237 return {238 status: 429,239 body: { errors: { message: 'Too many requests' } },240 headers: { 'retry-after': '1' },241 };242 if (calls === 2) return { status: 503, body: {} };243 return tomba(req);244 });245 const result = await run({ input: { query: 'tech', maxResults: 10, maxRetries: 3 }, endpoint: server.url });246 assert.equal(server.requests.length, 3);247 assert.equal(result.items.length, 10);248 assert.equal(result.items[0].charged, true);249 assert.deepEqual(result.chargeCounts, { 'tomba-request': 1 });250 });251
252 it('serves repeated runs from the cache for free', async () => {253 const server = await mock();254 const input = { query: 'tech', maxResults: 20 };255 const first = await run({ input, endpoint: server.url });256 assert.equal(totalCharges(first), 2);257 const second = await run({ input, endpoint: server.url, storageDir: first.storageDir });258
259 assert.equal(server.requests.length, 2);260 assert.equal(second.items.length, 20);261 assert.ok(second.items.every((i) => i.cached === true && i.charged === false));262 assert.equal(totalCharges(second), 0);263 });264
265 it('calls Tomba again when the cache is disabled', async () => {266 const server = await mock();267 const input = { query: 'tech', maxResults: 10, useCache: false };268 const first = await run({ input, endpoint: server.url });269 await run({ input, endpoint: server.url, storageDir: first.storageDir });270 assert.equal(server.requests.length, 2);271 });272
273 it('stops at the max charge limit and resumes without reprocessing', async () => {274 const server = await mock();275 const input = { query: 'big', maxResults: 100, maxConcurrency: 1, useCache: false };276
277 // Locally every event costs $1, so a $2 budget allows two billable pages.278 const first = await run({ input, endpoint: server.url, maxTotalChargeUsd: 2 });279 assert.equal(first.code, 0, first.output);280 assert.equal(totalCharges(first), 2);281 assert.deepEqual(pagesRequested(server), [1, 2]);282 assert.equal(first.items.length, 20);283
284 const second = await run({ input, endpoint: server.url, storageDir: first.storageDir, keepStorage: true });285 assert.equal(second.code, 0, second.output);286 assert.deepEqual(pagesRequested(server), [1, 2, 3, 4, 5, 6]);287 assert.equal(second.items.length, 60);288 assert.equal(new Set(second.items.map((i) => i.name)).size, 60);289 // The charging log is kept with the storage: 2 events from the first run + 4 new ones.290 assert.equal(totalCharges(second), 6);291 });292
293 it('keeps maxResults across a resumed run', async () => {294 const server = await mock();295 const input = { query: 'big', maxResults: 25, useCache: false };296 const first = await run({ input, endpoint: server.url, maxTotalChargeUsd: 2 });297 assert.equal(first.items.length, 20);298 const second = await run({ input, endpoint: server.url, storageDir: first.storageDir, keepStorage: true });299 assert.deepEqual(pagesRequested(server), [1, 2, 3]);300 assert.equal(second.items.length, 25);301 });302
303 it('fetches the pages of a search one after another', async () => {304 let active = 0;305 let peak = 0;306 const server = await mock(async (req) => {307 active++;308 peak = Math.max(peak, active);309 await new Promise((r) => {310 setTimeout(r, 50);311 });312 active--;313 return tomba(req);314 });315 await run({ input: { query: 'big', maxResults: 60, maxConcurrency: 10 }, endpoint: server.url });316 assert.equal(server.requests.length, 6);317 assert.equal(peak, 1);318 });319
320 it('fails without Tomba credentials and never calls the API', async () => {321 const server = await mock();322 const result = await run({ input: { query: 'tech' }, endpoint: server.url, withCredentials: false });323 assert.notEqual(result.code, 0);324 assert.match(result.output, /misconfigured/);325 assert.doesNotMatch(result.output, /ta_test_key|ts_test_secret/);326 assert.equal(server.requests.length, 0);327 });328
329 it('fails without a query or filters and never calls the API', async () => {330 const server = await mock();331 const result = await run({ input: { query: ' ', filters: {} }, endpoint: server.url });332 assert.notEqual(result.code, 0);333 assert.match(result.output, /query/);334 assert.equal(server.requests.length, 0);335 });336});337
338describe('company-search standby (real-time API)', () => {339 it('answers the readiness probe and a bare GET with usage info', async () => {340 const server = await mock();341 const actor = await startStandbyActor({ endpoint: server.url });342 try {343 const probe = await actor.call('/', { headers: { 'x-apify-container-server-readiness-probe': '1' } });344 assert.equal(probe.status, 200);345 const usage = await actor.call('/');346 assert.equal(usage.status, 200);347 assert.match(String(usage.body.usage), /GET/);348 assert.equal(server.requests.length, 0);349 } finally {350 await actor.stop();351 }352 });353
354 it('searches from GET query parameters and charges one event per page', async () => {355 const server = await mock();356 const actor = await startStandbyActor({ endpoint: server.url });357 let stopped;358 try {359 const filters = encodeURIComponent(JSON.stringify({ industry: { include: ['software'] } }));360 const res = await actor.call(`/?query=tech&filters=${filters}&source=name,website_url&maxResults=15`);361 assert.equal(res.status, 200);362 const items = res.body.items as Record<string, unknown>[];363 assert.equal(items.length, 15);364 assert.equal(items[14].name, 'Company 15');365 assert.deepEqual(server.requests[0].body, {366 query: 'tech',367 filters: { industry: { include: ['software'] } },368 _source: ['name', 'website_url'],369 page: 1,370 });371 assert.deepEqual(pagesRequested(server), [1, 2]);372 } finally {373 stopped = await actor.stop();374 }375 assert.deepEqual(stopped.chargeCounts, { 'tomba-request': 2 });376 });377
378 it('accepts a POST with the same JSON input as a normal run', async () => {379 const server = await mock();380 const actor = await startStandbyActor({ endpoint: server.url });381 try {382 const res = await actor.call('/', {383 body: { filters: { size: { include: ['51-200'] } }, page: 2, maxResults: 5 },384 });385 assert.equal(res.status, 200);386 const items = res.body.items as Record<string, unknown>[];387 assert.equal(items.length, 5);388 assert.equal(items[0].name, 'Company 11');389 assert.deepEqual(pagesRequested(server), [2]);390 } finally {391 await actor.stop();392 }393 });394
395 it('serves repeated requests from the cache for free', async () => {396 const server = await mock();397 const actor = await startStandbyActor({ endpoint: server.url });398 let stopped;399 try {400 await actor.call('/?query=tech&maxResults=10');401 const second = await actor.call('/?query=tech&maxResults=10');402 assert.ok((second.body.items as Record<string, unknown>[]).every((i) => i.cached === true));403 assert.equal(server.requests.length, 1);404 } finally {405 stopped = await actor.stop();406 }407 assert.deepEqual(stopped.chargeCounts, { 'tomba-request': 1 });408 });409
410 it('keeps serving after a request hits maxResults', async () => {411 const server = await mock();412 const actor = await startStandbyActor({ endpoint: server.url });413 try {414 const first = await actor.call('/?query=big&maxResults=5');415 assert.equal((first.body.items as unknown[]).length, 5);416 const second = await actor.call('/?query=big&maxResults=25');417 assert.equal((second.body.items as unknown[]).length, 25);418 assert.deepEqual(pagesRequested(server), [1, 2, 3]);419 } finally {420 await actor.stop();421 }422 });423
424 it('rejects invalid input with 400 and unknown paths with 404', async () => {425 const server = await mock();426 const actor = await startStandbyActor({ endpoint: server.url });427 try {428 assert.equal((await actor.call('/', { body: {} })).status, 400);429 assert.equal((await actor.call('/', { body: 'not json' })).status, 400);430 assert.equal((await actor.call('/?query=tech&maxResults=abc')).status, 400);431 const badFilters = await actor.call('/?filters=notjson');432 assert.equal(badFilters.status, 400);433 assert.match(String(badFilters.body.error), /filters/);434 assert.equal((await actor.call(`/?filters=${encodeURIComponent('[1]')}`)).status, 400);435 assert.equal((await actor.call('/?page=1')).status, 400);436 assert.equal((await actor.call('/nope')).status, 404);437 assert.equal((await actor.call('/', { method: 'DELETE' })).status, 405);438 assert.equal(server.requests.length, 0);439 } finally {440 await actor.stop();441 }442 });443
444 it('returns 402 once the max charge limit is reached', async () => {445 const server = await mock();446 const actor = await startStandbyActor({ endpoint: server.url, maxTotalChargeUsd: 1 });447 try {448 const first = await actor.call('/?query=tech&maxResults=30');449 assert.equal(first.status, 200);450 assert.equal((first.body.items as unknown[]).length, 10);451 const second = await actor.call('/?query=other');452 assert.equal(second.status, 402);453 assert.equal(server.requests.length, 1);454 } finally {455 await actor.stop();456 }457 });458});1// Shared runner for the Tomba Apify Actors: batch runs and Standby (real-time HTTP API) mode.2// Keep this file identical across all Actors.3import type { IncomingMessage, ServerResponse } from 'node:http';4import { createServer } from 'node:http';5
6import { Actor, log } from 'apify';7
8import type { RunOptions } from './tomba.js';9import { isStopped, logSummary, setupTomba, useRunState } from './tomba.js';10
11/** Invalid input: a failed batch run, or `400 Bad Request` in Standby mode. */12export class InputError extends Error {}13
14export type Item = Record<string, unknown>;15
16export interface RunContext {17 /** Save result items (dataset in batch runs, HTTP response in Standby mode). */18 push: (items: Item | Item[]) => Promise<void>;19 /** True when this input was already processed (batch runs resume after a migration or restart). */20 isDone: (key: string) => boolean;21 /** Record a processed input so a resumed run skips it. */22 markDone: (key: string) => void;23 standby: boolean;24}25
26/** Processes one Actor input. Throws InputError for invalid input. */27export type ActorRun<I> = (input: I, ctx: RunContext) => Promise<void>;28
29export interface ActorDefinition<I> {30 /** Name used in logs. */31 title: string;32 run: ActorRun<I>;33 /** Builds the Actor input from `GET /?…` query parameters in Standby mode. */34 fromQuery: (query: URLSearchParams) => I;35 /** Number of inputs, for the batch summary log. */36 count?: (input: I) => number;37}38
39const MAX_BODY_BYTES = 1_000_000;40
41export function isStandby(): boolean {42 return Actor.config.get('metaOrigin') === 'STANDBY';43}44
45/** Entry point: runs the Actor as a batch job, or as an HTTP server in Standby mode. */46export async function runActor<I extends RunOptions>(definition: ActorDefinition<I>): Promise<void> {47 await Actor.init();48
49 if (isStandby()) {50 await setupTomba();51 await startStandbyServer(definition);52 return;53 }54
55 const input = ((await Actor.getInput<I>()) ?? {}) as I;56 await setupTomba(input);57 const state = await useRunState();58 const startedAt = Date.now();59 try {60 await definition.run(input, {61 push: async (items) => Actor.pushData(items),62 isDone: (key) => Boolean(state.done[key]),63 markDone: (key) => {64 state.done[key] = true;65 },66 standby: false,67 });68 } catch (err) {69 if (err instanceof InputError) {70 await Actor.fail(err.message);71 return;72 }73 throw err;74 }75 logSummary(definition.title, definition.count?.(input) ?? 0, startedAt);76 await Actor.exit();77}78
79async function startStandbyServer<I>(definition: ActorDefinition<I>): Promise<void> {80 const server = createServer((req, res) => {81 handleRequest(definition, req, res).catch((err: Error) => {82 log.exception(err, 'Standby request failed');83 send(res, 500, { error: 'Internal error' });84 });85 });86 const port = Actor.config.get('containerPort');87 await new Promise<void>((resolve) => {88 server.listen(port, resolve);89 });90 log.info(`${definition.title} is ready for real-time requests`, { port });91}92
93async function handleRequest<I>(definition: ActorDefinition<I>, req: IncomingMessage, res: ServerResponse) {94 if (req.headers['x-apify-container-server-readiness-probe']) {95 send(res, 200, { status: 'ready' });96 return;97 }98
99 const url = new URL(req.url ?? '/', 'http://localhost');100 if (url.pathname !== '/') {101 send(res, 404, { error: 'Not found. Use GET / with query parameters or POST / with a JSON input.' });102 return;103 }104
105 let input: I;106 try {107 if (req.method === 'GET') {108 if ([...url.searchParams.keys()].length === 0) {109 send(res, 200, {110 status: 'ready',111 usage: 'GET /?<parameters> or POST / with the same JSON input as a normal run.',112 });113 return;114 }115 input = definition.fromQuery(url.searchParams);116 } else if (req.method === 'POST') {117 input = (await readJson(req)) as I;118 } else {119 send(res, 405, { error: 'Method not allowed. Use GET or POST.' });120 return;121 }122 } catch (err) {123 send(res, 400, { error: (err as Error).message });124 return;125 }126
127 if (isStopped()) {128 send(res, 402, { error: 'Max charge limit reached for this Actor run.' });129 return;130 }131
132 const items: Item[] = [];133 try {134 await definition.run(input, {135 push: async (pushed) => {136 items.push(...(Array.isArray(pushed) ? pushed : [pushed]));137 },138 isDone: () => false,139 markDone: () => undefined,140 standby: true,141 });142 } catch (err) {143 if (err instanceof InputError) {144 send(res, 400, { error: err.message });145 return;146 }147 throw err;148 }149 send(res, 200, { items });150}151
152async function readJson(req: IncomingMessage): Promise<unknown> {153 const chunks: Buffer[] = [];154 let size = 0;155 for await (const chunk of req) {156 size += (chunk as Buffer).length;157 if (size > MAX_BODY_BYTES) throw new InputError('Request body is too large (max 1 MB).');158 chunks.push(chunk as Buffer);159 }160 const text = Buffer.concat(chunks).toString('utf8').trim();161 if (!text) return {};162 try {163 const body = JSON.parse(text);164 if (typeof body !== 'object' || body === null || Array.isArray(body)) {165 throw new InputError('Request body must be a JSON object.');166 }167 return body;168 } catch (err) {169 if (err instanceof InputError) throw err;170 throw new InputError('Request body must be valid JSON.');171 }172}173
174function send(res: ServerResponse, status: number, body: unknown): void {175 if (res.headersSent) return;176 res.writeHead(status, { 'content-type': 'application/json; charset=utf-8' });177 res.end(JSON.stringify(body));178}179
180// --- Query string helpers for `fromQuery` ---------------------------------------------------181
182/** Values of a list parameter: repeated (`?domain=a&domain=b`) and/or comma-separated (`?domains=a,b`). */183export function queryList(query: URLSearchParams, ...names: string[]): string[] {184 return names185 .flatMap((name) => query.getAll(name))186 .flatMap((value) => value.split(','))187 .map((value) => value.trim())188 .filter(Boolean);189}190
191export function queryString(query: URLSearchParams, name: string): string | undefined {192 const value = query.get(name)?.trim();193 return value || undefined;194}195
196export function queryBool(query: URLSearchParams, name: string): boolean | undefined {197 const value = query.get(name)?.trim().toLowerCase();198 if (value === undefined || value === '') return undefined;199 if (['1', 'true', 'yes'].includes(value)) return true;200 if (['0', 'false', 'no'].includes(value)) return false;201 throw new InputError(`Query parameter "${name}" must be true or false.`);202}203
204export function queryInt(query: URLSearchParams, name: string): number | undefined {205 const value = query.get(name)?.trim();206 if (value === undefined || value === '') return undefined;207 const number = Number(value);208 if (!Number.isInteger(number)) throw new InputError(`Query parameter "${name}" must be an integer.`);209 return number;210}1// Shared Tomba helper for the Tomba Apify Actors.2// Keep this file identical across all Actors.3import { createHash } from 'node:crypto';4
5import type { KeyValueStore } from 'apify';6import { Actor, log } from 'apify';7import { TombaClient, TombaException } from 'tomba';8
9/** Pay-per-event event names (configured in Apify Console → Monetization). */10export const EVENT_REQUEST = 'tomba-request';11export const EVENT_PHONE_FINDER = 'phone-finder-request';12
13/**14 * Tomba credits added per phone number (or per address with phone data, depending on the endpoint)15 * returned when `enrich_mobile=true`. Every credit is charged as one `tomba-request` event.16 */17export const PHONE_CREDITS = 5;18
19/** Number of phone numbers in a record's `phone_data` array (0 when missing or empty). */20export function phoneDataCount(record: unknown): number {21 if (!isObject(record)) return 0;22 return Array.isArray(record.phone_data) ? record.phone_data.length : 0;23}24
25/** True when a record has at least one phone number in `phone_data`. */26export function hasPhoneData(record: unknown): boolean {27 return phoneDataCount(record) > 0;28}29
30export interface RunOptions {31 maxConcurrency?: number;32 maxRetries?: number;33 useCache?: boolean;34 cacheTtlHours?: number;35}36
37export interface TombaCallResult {38 /** Raw JSON body returned by Tomba (`{ data, meta, ... }`). */39 body?: Record<string, unknown>;40 /** Shortcut to `body.data`. */41 data?: unknown;42 /** HTTP status (0 for network errors, 502 for a non-JSON body). */43 status: number;44 charged: boolean;45 /** Number of events charged for this call (0 when not charged). */46 chargedCount?: number;47 cached: boolean;48 /** True when the call was not made because the user's max charge limit was reached. */49 skipped?: boolean;50 error?: string;51}52
53export const stats = {54 requests: 0,55 charged: 0,56 cached: 0,57 failed: 0,58 retries: 0,59};60
61const CACHE_STORE_PREFIX = 'tomba-cache';62const BASE_DELAY_MS = 1000;63const MAX_DELAY_MS = 30000;64const NETWORK_ERROR = /ECONN|ETIMEDOUT|ENOTFOUND|EAI_AGAIN|EPIPE|socket hang up|timeout|network/i;65
66let client: TombaClient | undefined;67let options: Required<RunOptions> = { maxConcurrency: 10, maxRetries: 3, useCache: true, cacheTtlHours: 24 };68let cacheStore: KeyValueStore | undefined;69const memoryCache = new Map<string, Record<string, unknown>>();70const inFlight = new Map<string, number>();71let stopped = false;72
73/** Initialise the Tomba client from the Actor's environment variables and apply the run options. */74export async function setupTomba(runOptions: RunOptions = {}): Promise<TombaClient> {75 const key = process.env.TOMBA_API_KEY;76 const secret = process.env.TOMBA_API_SECRET;77 if (!key || !secret) {78 await Actor.fail('Actor is misconfigured: Tomba credentials are missing. Please contact the Actor developer.');79 throw new Error('unreachable');80 }81
82 options = {83 maxConcurrency: clamp(runOptions.maxConcurrency ?? 10, 1, 50),84 maxRetries: clamp(runOptions.maxRetries ?? 3, 0, 10),85 useCache: runOptions.useCache ?? true,86 cacheTtlHours: Math.max(0, runOptions.cacheTtlHours ?? 24),87 };88
89 client = new TombaClient();90 // Verification and enrichment can take over a minute, so keep the SDK's 120 s timeout.91 client.setKey(key).setSecret(secret).setTimeout(120_000);92 // Only used by the test suite to point the client at a local mock server.93 if (process.env.TOMBA_API_ENDPOINT) client.setEndpoint(process.env.TOMBA_API_ENDPOINT);94
95 if (options.useCache && options.cacheTtlHours > 0) {96 try {97 cacheStore = await Actor.openKeyValueStore(cacheStoreName());98 } catch (err) {99 // E.g. "Insufficient permissions" under limited permissions: never fail the run because of the cache.100 log.warning('Cross-run cache is unavailable; results are cached for this run only.', {101 error: (err as Error).message,102 });103 cacheStore = undefined;104 }105 }106
107 return client;108}109
110/**111 * Name of the cross-run cache store. It is per Actor: with limited permissions an Actor can only open112 * named storages it created itself, so the Tomba Actors must not share one store.113 */114export function cacheStoreName(): string {115 const actorId = Actor.getEnv().actorId ?? process.env.ACTOR_ID;116 return actorId ? `${CACHE_STORE_PREFIX}-${actorId}` : CACHE_STORE_PREFIX;117}118
119/** The Tomba client created by setupTomba(). */120export function getClient(): TombaClient {121 if (!client) throw new Error('setupTomba() must be called first');122 return client;123}124
125export function getOptions(): Required<RunOptions> {126 return options;127}128
129/** True once the user's max-charge budget is exhausted; workers should stop picking up new items. */130export function isStopped(): boolean {131 return stopped;132}133
134export function stop(): void {135 stopped = true;136}137
138/**139 * Tomba billing rules: charge only when Tomba returns a JSON body without an `errors` object140 * and with a non-empty `data` (a negative answer such as an undeliverable verification141 * or a domain search with no emails is still a non-empty `data` object).142 */143export function isBillable(body: unknown): boolean {144 if (!isObject(body)) return false;145 if (body.errors !== undefined && body.errors !== null) return false;146 const { data } = body;147 if (data === null || data === undefined) return false;148 if (Array.isArray(data)) return data.length > 0;149 // An object whose values are all null (e.g. `{ person: null, company: null }`) is an empty result.150 if (isObject(data)) return Object.values(data).some((v) => v !== null && v !== undefined);151 return data !== '' && data !== false;152}153
154/**155 * Perform one Tomba API call with caching, retry and pay-per-event charging.156 *157 * @param endpoint short endpoint name used for the cache key (e.g. 'technology')158 * @param params request parameters used for the cache key159 * @param fn the SDK call160 * @param eventName pay-per-event name to charge when the response is billable161 * @param count events to charge for a billable response: a number, or a function of the response body162 */163export async function callTomba(164 endpoint: string,165 params: Record<string, unknown>,166 fn: (tomba: TombaClient) => Promise<unknown>,167 eventName: string = EVENT_REQUEST,168 count: number | ((body: Record<string, unknown>) => number) = 1,169): Promise<TombaCallResult> {170 if (!client) throw new Error('setupTomba() must be called first');171
172 const cacheKey = hashKey(endpoint, params);173 const cachedBody = await readCache(cacheKey);174 if (cachedBody) {175 stats.cached++;176 return { body: cachedBody, data: cachedBody.data, status: 200, charged: false, cached: true };177 }178
179 // Reserve budget before calling Tomba so concurrent workers never exceed the user's max charge.180 // A fixed count is known up front, so reserve all of it; a body-dependent count reserves at least 1.181 const reserved = typeof count === 'number' ? Math.max(1, Math.ceil(count)) : 1;182 if (!reserve(eventName, reserved)) {183 stop();184 return { status: 0, charged: false, cached: false, skipped: true, error: 'Max charge limit reached' };185 }186
187 try {188 for (let attempt = 0; ; attempt++) {189 try {190 stats.requests++;191 const body = unwrap(await fn(client));192
193 if (!isObject(body)) {194 stats.failed++;195 return { status: 502, charged: false, cached: false, error: 'Invalid response from Tomba' };196 }197
198 if (!isBillable(body)) {199 const error = errorMessage(body) ?? undefined;200 return { body, data: body.data, status: 200, charged: false, cached: false, error };201 }202
203 const units = Math.max(1, Math.ceil(typeof count === 'function' ? count(body) : count));204 const chargedCount = await charge(eventName, units);205 const charged = chargedCount > 0;206 await writeCache(cacheKey, body);207 return { body, data: body.data, status: 200, charged, chargedCount, cached: false };208 } catch (err) {209 const { status, message, retryAfter } = parseError(err);210 const retryable = status === 0 || status === 429 || status >= 500;211 if (retryable && attempt < options.maxRetries) {212 stats.retries++;213 const delay = retryAfter ?? Math.min(MAX_DELAY_MS, BASE_DELAY_MS * 2 ** attempt);214 const jitter = Math.random() * 250;215 log.warning(`Tomba ${endpoint} returned ${status || 'network error'}, retrying in ${delay}ms`, {216 attempt: attempt + 1,217 });218 await sleep(delay + jitter);219 continue;220 }221 stats.failed++;222 return { status, charged: false, cached: false, error: message };223 }224 }225 } finally {226 release(eventName, reserved);227 }228}229
230/**231 * Run `worker` over `items` with bounded concurrency. Stops picking new items once the charge limit232 * is reached (`isStopped()`) or `shouldStop()` returns true (e.g. maxResults reached for this run).233 */234export async function runPool<T>(235 items: T[],236 worker: (item: T, index: number) => Promise<void>,237 concurrency = options.maxConcurrency,238 shouldStop: () => boolean = () => false,239): Promise<void> {240 let next = 0;241 const runners = Array.from({ length: Math.min(concurrency, items.length) }, async () => {242 while (!stopped && !shouldStop() && next < items.length) {243 const index = next++;244 await worker(items[index], index);245 }246 });247 await Promise.all(runners);248}249
250/** Persisted run state used to resume after a migration, restart or resurrection. */251export async function useRunState(): Promise<{ done: Record<string, true> }> {252 return Actor.useState<{ done: Record<string, true> }>('TOMBA_STATE', { done: {} });253}254
255export function logSummary(title: string, total: number, startedAt: number): void {256 const seconds = Math.round((Date.now() - startedAt) / 1000);257 log.info(`${title} finished`, {258 inputs: total,259 tombaRequests: stats.requests,260 charged: stats.charged,261 cacheHits: stats.cached,262 failed: stats.failed,263 retries: stats.retries,264 seconds,265 stoppedByChargeLimit: stopped,266 });267}268
269export function normalizeDomain(value: string): string {270 return value271 .trim()272 .toLowerCase()273 .replace(/^https?:\/\//, '')274 .replace(/^www\./, '')275 .replace(/\/.*$/, '');276}277
278export function normalizeEmail(value: string): string {279 return value.trim().toLowerCase();280}281
282export function unique<T>(items: T[], key: (item: T) => string = String): T[] {283 const seen = new Set<string>();284 return items.filter((item) => {285 const k = key(item);286 if (!k || seen.has(k)) return false;287 seen.add(k);288 return true;289 });290}291
292// ---------------------------------------------------------------------------293
294async function charge(eventName: string, count: number): Promise<number> {295 const result = await Actor.charge({ eventName, count });296 if (result.chargedCount > 0) stats.charged += result.chargedCount;297 if (result.eventChargeLimitReached) {298 log.info('Max charge limit reached, finishing current items and stopping.');299 stop();300 }301 return result.chargedCount;302}303
304function reserve(eventName: string, units: number): boolean {305 if (stopped) return false;306 const manager = Actor.getChargingManager();307 const remaining = manager.calculateMaxEventChargeCountWithinLimit(eventName);308 const current = inFlight.get(eventName) ?? 0;309 if (remaining - current < units) return false;310 inFlight.set(eventName, current + units);311 return true;312}313
314function release(eventName: string, units: number): void {315 inFlight.set(eventName, Math.max(0, (inFlight.get(eventName) ?? units) - units));316}317
318async function readCache(key: string): Promise<Record<string, unknown> | undefined> {319 if (!options.useCache) return undefined;320 const hit = memoryCache.get(key);321 if (hit) return hit;322 if (!cacheStore) return undefined;323 const entry = await cacheStore.getValue<{ savedAt: number; body: Record<string, unknown> }>(key);324 if (!entry) return undefined;325 if (Date.now() - entry.savedAt > options.cacheTtlHours * 3600_000) return undefined;326 memoryCache.set(key, entry.body);327 return entry.body;328}329
330async function writeCache(key: string, body: Record<string, unknown>): Promise<void> {331 if (!options.useCache) return;332 memoryCache.set(key, body);333 if (cacheStore) {334 try {335 await cacheStore.setValue(key, { savedAt: Date.now(), body });336 } catch (err) {337 log.debug('Failed to write cache entry', { error: (err as Error).message });338 }339 }340}341
342function hashKey(endpoint: string, params: Record<string, unknown>): string {343 const sorted = Object.keys(params)344 .sort()345 .map((k) => [k, params[k]]);346 return `${endpoint}-${createHash('sha256').update(JSON.stringify(sorted)).digest('hex').slice(0, 40)}`;347}348
349/** tomba >= 1.1.0 resolves to `{ data: body, rateLimit }`; older versions resolve to the body itself. */350function unwrap(response: unknown): unknown {351 if (isObject(response) && 'rateLimit' in response && 'data' in response) return response.data;352 return response;353}354
355function parseError(err: unknown): { status: number; message: string; retryAfter?: number } {356 if (err instanceof TombaException) {357 const response = err.code as { status?: number; headers?: Record<string, string> } | undefined;358 // Without an HTTP response it is either a network error (retryable, status 0)359 // or the SDK's own input validation (not retryable, treated as 400).360 const networkError = NETWORK_ERROR.test(err.message ?? '');361 let status = networkError ? 0 : 400;362 if (typeof response?.status === 'number') status = response.status;363 const retryAfterHeader = Number(response?.headers?.['retry-after']);364 const retryAfter =365 Number.isFinite(retryAfterHeader) && retryAfterHeader > 0 ? retryAfterHeader * 1000 : undefined;366 const message = errorMessage(err.response) ?? err.message ?? 'Tomba request failed';367 return { status, message: status ? `${status}: ${message}` : message, retryAfter };368 }369 return { status: 0, message: err instanceof Error ? err.message : String(err) };370}371
372function errorMessage(body: unknown): string | undefined {373 if (!isObject(body)) return typeof body === 'string' && body.length < 300 ? body : undefined;374 const errors = body.errors as Record<string, unknown> | undefined;375 if (isObject(errors) && typeof errors.message === 'string') return errors.message;376 return undefined;377}378
379function isObject(value: unknown): value is Record<string, unknown> {380 return typeof value === 'object' && value !== null && !Array.isArray(value);381}382
383function clamp(value: number, min: number, max: number): number {384 return Math.min(max, Math.max(min, Math.floor(value)));385}386
387async function sleep(ms: number): Promise<void> {388 return new Promise((resolve) => {389 setTimeout(resolve, ms);390 });391}1import { Actor, log } from 'apify';2import type { CompanyFilters } from 'tomba';3import { Reveal } from 'tomba';4
5import { InputError, queryInt, queryList, queryString, runActor } from './standby.js';6import type { RunOptions } from './tomba.js';7import { callTomba, getClient, isBillable } from './tomba.js';8
9interface ActorInput extends RunOptions {10 query?: string;11 filters?: CompanyFilters;12 source?: string[];13 page?: number;14 maxResults?: number;15}16
17interface SearchBody {18 data?: { companies?: Record<string, unknown>[] };19 meta?: { total?: number; page?: number; limit?: number; pages?: number };20}21
22const SOURCE = 'tomba_company_search';23
24/** `filters` query parameter: a JSON object, e.g. `{"industry":{"include":["software"]}}`. */25function parseFilters(value: string | undefined): CompanyFilters | undefined {26 if (value === undefined) return undefined;27 let filters: unknown;28 try {29 filters = JSON.parse(value);30 } catch {31 throw new InputError('Query parameter "filters" must be a JSON object.');32 }33 if (typeof filters !== 'object' || filters === null || Array.isArray(filters)) {34 throw new InputError('Query parameter "filters" must be a JSON object.');35 }36 return filters as CompanyFilters;37}38
39await runActor<ActorInput>({40 title: 'Company Search',41 count: () => 1,42 fromQuery: (query) => {43 const source = queryList(query, 'source');44 return {45 query: queryString(query, 'query'),46 filters: parseFilters(queryString(query, 'filters')),47 source: source.length ? source : undefined,48 page: queryInt(query, 'page'),49 maxResults: queryInt(query, 'maxResults'),50 };51 },52 run: async (input, { push, isDone, markDone, standby }) => {53 const query = input.query?.trim() || undefined;54 const filters = input.filters && Object.keys(input.filters).length > 0 ? input.filters : undefined;55 if (!query && !filters) {56 throw new InputError('Input must contain a search "query", "filters", or both.');57 }58 const { source: fields, page: startPage = 1, maxResults = 100 } = input;59 const reveal = new Reveal(getClient());60
61 // Resume (batch runs only): pages already processed are skipped, and the number of companies62 // already pushed is kept. In Standby mode the progress only lives for this request.63 const progress = standby64 ? { pushed: 0 }65 : await Actor.useState<{ pushed: number }>('COMPANY_SEARCH_PROGRESS', { pushed: 0 });66 if (progress.pushed > 0) {67 log.info(`Resuming: ${progress.pushed} companies already saved.`);68 }69
70 let pagesRequested = 0;71 if (!standby) log.info('Searching companies', { query, filters, maxResults });72
73 // One query: pages are fetched sequentially, each page is one billable request.74 for (let page = Math.max(1, startPage); progress.pushed < maxResults; page++) {75 if (isDone(`page:${page}`)) continue;76
77 const params = { query, filters, _source: fields?.length ? fields : undefined, page };78 const res = await callTomba('companies-search', params, async () => reveal.companiesSearch(params));79 if (res.skipped) break;80 pagesRequested++;81
82 const body = res.body as SearchBody | undefined;83 const companies = Array.isArray(body?.data?.companies) ? body.data.companies : [];84
85 if (!isBillable(res.body) || companies.length === 0) {86 if (progress.pushed === 0) {87 await push({88 query: query ?? null,89 page,90 source: SOURCE,91 charged: res.charged,92 cached: res.cached,93 error: res.error ?? 'No companies found',94 });95 }96 log.info(`Page ${page}: ${res.error ?? 'no companies found'}`);97 markDone(`page:${page}`);98 break;99 }100
101 const items = companies.slice(0, Math.max(0, maxResults - progress.pushed)).map((company) => ({102 ...company,103 source: SOURCE,104 charged: res.charged,105 cached: res.cached,106 }));107 await push(items);108 progress.pushed += items.length;109 markDone(`page:${page}`);110
111 const totalPages = body?.meta?.pages;112 log.info(113 `Page ${page}${typeof totalPages === 'number' ? ` of ${totalPages}` : ''}: ${items.length} companies${res.cached ? ' (cached)' : ''}`,114 { total: body?.meta?.total },115 );116
117 const pageSize = body?.meta?.limit;118 if (typeof totalPages === 'number' && page >= totalPages) break;119 // A short page is the last one: don't pay for an extra empty page.120 if (typeof pageSize === 'number' && companies.length < pageSize) break;121 }122
123 log.info(`Total companies saved: ${progress.pushed}`, { pagesRequested });124 },125});name: Test and push latest version
on: push: branches: - main - master pull_request: workflow_dispatch:
concurrency: group: ${{ github.workflow }}-${{ github.ref }} cancel-in-progress: true
jobs: test: name: Format, lint, build and test runs-on: ubuntu-latest steps: - uses: actions/checkout@v4
- name: Set up Node.js uses: actions/setup-node@v4 with: node-version: 22 cache: npm
- name: Install dependencies run: npm ci
- name: Check formatting run: npm run format:check
- name: Lint run: npm run lint
- name: Build run: npm run build
- name: Test run: npm test
push: name: Push Actor to Apify needs: test # Only publish from the default branch, never from pull requests. if: github.event_name != 'pull_request' && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/master') runs-on: ubuntu-latest steps: - uses: actions/checkout@v4
# .actor/actor.json maps TOMBA_API_KEY / TOMBA_API_SECRET to the Apify CLI secrets # @tombaApiKey / @tombaApiSecret, which `apify push` reads from ~/.apify/secrets.json. - name: Provide Tomba secrets to the Apify CLI env: TOMBA_API_KEY: ${{ secrets.TOMBA_API_KEY }} TOMBA_API_SECRET: ${{ secrets.TOMBA_API_SECRET }} run: | if [ -z "$TOMBA_API_KEY" ] || [ -z "$TOMBA_API_SECRET" ]; then echo "::error::Add the TOMBA_API_KEY and TOMBA_API_SECRET repository secrets." exit 1 fi mkdir -p ~/.apify node -e 'require("fs").writeFileSync(require("os").homedir() + "/.apify/secrets.json", JSON.stringify({ tombaApiKey: process.env.TOMBA_API_KEY, tombaApiSecret: process.env.TOMBA_API_SECRET }))'
- name: Push Actor to Apify id: push uses: apify/push-actor-action@v1 with: token: ${{ secrets.APIFY_TOKEN }}
- name: Print build details run: | echo "Build ${{ steps.push.outputs.build-id }} finished with status ${{ steps.push.outputs.build-status }}"