DomainScope URL Categorizer — Apify Actor

A thin Apify Actor wrapper around the ../ service ("DomainScope"), deployed publicly at https://domainscope.scrapetheworld.org. It does not reimplement any categorization logic — it forwards urls to the existing REST API and streams the results into the Actor's dataset.

How it decides which endpoint to call

Input	Endpoint(s) used
1 URL	POST /api/v1/categorize (auth optional)
2..jobThreshold URLs (default 25)	POST /api/v1/categorize/bulk (auth required)
more than jobThreshold URLs	POST /api/v1/jobs/bulk-categorize, polled via GET /jobs/{id}/status, fetched via GET /jobs/{id}/download (auth required)

The async job path uses a random Idempotency-Key per run so a retried submission cannot double-queue (and double-bill) the same URLs.

Input

See .actor/input_schema.json. Key fields:

• urls (required) — list of URLs/domains to categorize.
• apiKey — Bearer token for the DomainScope API. Required for more than one URL. If left blank, the Actor falls back to the DOMAINSCOPE_API_KEY environment variable (see "Shared service key" below), then to the anonymous tier (single URL only, 10 requests/hour).
• apiBaseUrl — defaults to https://domainscope.scrapetheworld.org/api/v1. Override for local/staging testing.
• jobThreshold, pollIntervalSecs, maxPollMinutes — tune the sync-vs-async cutover and how long the Actor waits on a large job.

Output

One dataset row per input URL, shaped like model.CategoryResult / model.CategoryResultWithError from the API (../openapi/openapi.yaml): url, category, sub_category, industry, country, business_model, target_market, audience_profile, content_focus, entity_type, city, summary, and error when categorization failed for that URL.

Billing

The Actor charges one url-categorized event (via Actor.charge()) per successfully categorized URL — rows with an error are pushed to the dataset but not charged, mirroring the API's own bulk/job semantics ("a per-URL failure does not fail the request").

Actor.charge() is a documented no-op outside a pay-per-event-enabled run (e.g. apify run locally), so local development works without any billing setup. To publish this as a monetized Actor on Apify Store:

1. Push the Actor to Apify (apify push) and open it in the Apify Console.
2. Under Publication → Monetization, choose Pay per event and add a url-categorized event with whatever price you want to charge — the event name must match the string used in src/main.js.
3. Retrieve the actor's chargeable event names via the Get Actor API endpoint to confirm the config took effect before going live.

Shared service key vs. bring-your-own key

Two ways to run this in production, pick one:

• BYO key (default, no extra setup) — each caller supplies their own DomainScope API key via the apiKey input field. They're billed by DomainScope's own credit system; the Actor is unmetered (or metered separately via Apify PPE on top, if you want to charge for the convenience wrapper itself).
• Shared service key — mint one API key for the Actor itself (POST /api/v1/api-keys against the DomainScope API, see ../API_DOCS.md), store it as an Apify Actor secret environment variable named DOMAINSCOPE_API_KEY (Console → Actor → Settings → Environment variables → mark as secret), and leave apiKey blank in the input. End users never see the key; you bill them entirely through Apify's pay-per-event pricing instead of DomainScope credits.

Either way, no changes are needed on the DomainScope API side — it already has a dedicated API-key rate-limit tier (6000 req/min by default, src/internal/middleware/ratelimit.go) separate from the anonymous and per-user tiers, so a shared service key comfortably absorbs Actor traffic without needing IP allowlisting (which wouldn't work for Apify's cloud workers anyway — they have no fixed egress IP range).

Local development

cd apify-actor
npm install
cp .env.example .env   # fill in DOMAINSCOPE_API_KEY for multi-URL testing
export $(grep -v '^#' .env | xargs)

# Provide input via storage/key_value_stores/default/INPUT.json, or:
echo '{"urls": ["https://example.com", "https://anthropic.com"]}' \
  > storage/key_value_stores/default/INPUT.json

npm run start:dev

Or with the Apify CLI (npm i -g apify-cli):

$apify run --input '{"urls": ["https://example.com"]}'

Deploying

apify login
apify push

This builds Dockerfile on Apify's infrastructure and creates/updates the Actor under your Apify account. See Apify's Actor deployment docs for CI-based alternatives.