WHOIS / RDAP Domain Scraper & Change Monitor

Run authoritative bulk WHOIS/RDAP domain lookups and monitor expiry, renewals, registrar transfers, nameservers, DNSSEC, DNS and TLS changes across scheduled runs.

This Actor turns a domain watchlist into authoritative, normalized observations and exact change events. It discovers each registry through the official IANA RDAP bootstrap, stores only the last complete successful state in Apify KVS, and never exposes registrant personal data.

Unlike a raw WHOIS scraper, it produces history-ready rows with explicit SUCCESS, PARTIAL, NOT_FOUND, RATE_LIMITED and failure semantics. A bad check never overwrites the last known good state, and a single 404 never becomes a false domain-drop alert.

What it is for

• audit a single domain or a portfolio of up to 1,000 inputs;
• detect registration renewals and expiry-date changes;
• alert once when 90/30/14/7/1-day expiry thresholds are crossed;
• detect registrar, registry nameserver, EPP status and DNSSEC changes;
• optionally observe public A, AAAA, MX, NS and CAA records;
• optionally monitor standard port-443 SSL/TLS certificate expiry and renewal;
• feed scheduled runs into webhooks, Slack/email automation, a SIEM or a CMDB.

This is not a WHOIS text parser, domain-availability oracle, vulnerability scanner or ownership/registrant data extractor.

Quick start: WHOIS/RDAP lookup

{
  "domains": [{"domain": "example.com"}],
  "mode": "snapshot",
  "outputMode": "all",
  "includeDns": false,
  "includeTls": false
}

The Actor accepts bare domains, subdomains and HTTP(S) URLs. A pinned Public Suffix List converts all of these to the same registration:

example.com
www.example.com
https://www.example.com/path
  -> example.com

IDNs are normalized to lowercase A-label/punycode form. foo.example.co.uk becomes example.co.uk; suffixes are never parsed with a hand-written last-two- labels rule.

Monitor a 100-domain portfolio

Use the same input on every scheduled run and keep mode: monitor:

{
  "domains": [
    {"domain": "example.com", "externalId": "portfolio-001"},
    {"domain": "example.org", "externalId": "client-17"}
  ],
  "mode": "monitor",
  "outputMode": "changesOnly",
  "includeDns": true,
  "includeTls": true,
  "expiryWarningDays": [90, 30, 14, 7, 1]
}

Extend domains to 100 or 1,000 objects. Duplicate normalized domains are looked up once but still receive one result row per input/external ID. One bad domain never aborts the batch.

outputMode: changesOnly suppresses the redundant UNCHANGED object inside events, while all keeps it. Both deliberately emit one status row per input so an empty dataset can never disguise a failed source or an unchanged portfolio. Trigger alerts only for non-empty events; baseline, failure and change classifications remain explicit in the main row.

Expiry monitoring without alert spam

Warnings fire only when the observed remaining days cross a configured threshold:

31 days -> 29 days
event: EXPIRY_WARNING
warning_code: EXPIRY_WARNING_30D

The same warning is not repeated on the next daily run. When RDAP reports a later expiration date, the Actor emits both DOMAIN_RENEWED (with exact before, after and extension_days) and EXPIRY_DATE_CHANGED, then resets the applicable warning state.

Registrar transfer example

{
  "change_type": "CHANGED",
  "events": [
    {
      "type": "REGISTRAR_CHANGED",
      "before": {"name": "Registrar A", "iana_id": "1"},
      "after": {"name": "Registrar B", "iana_id": "2"}
    }
  ]
}

Registrar comparison uses the normalized registrar-role RDAP entity and its IANA Registrar ID when available. Registrant/admin/technical vCards are dropped before any state or Dataset record is built.

Changes to registry-published created_at and domain-level updated_at are reported exactly as REGISTRATION_DATE_CHANGED and RDAP_LAST_CHANGED; the RDAP database refresh timestamp is intentionally not used as domain state.

Nameserver change example

Registry delegation and live DNS are separate signals:

{
  "events": [
    {
      "type": "NAMESERVERS_CHANGED",
      "before": ["ns1.old.example", "ns2.old.example"],
      "after": ["ns1.new.example", "ns2.new.example"]
    },
    {
      "type": "NS_CHANGED",
      "before": ["ns1.old.example", "ns2.old.example"],
      "after": ["ns1.new.example", "ns2.new.example"]
    }
  ]
}

NAMESERVERS_CHANGED comes from RDAP. NS_CHANGED comes from the optional DNS observation.

Renewal detected

{
  "type": "DOMAIN_RENEWED",
  "before": "2027-04-01T00:00:00Z",
  "after": "2028-04-01T00:00:00Z",
  "extension_days": 366
}

A change in RDAP updated_at alone is never treated as a renewal.

Safe NOT_FOUND handling

The statuses are:

• SUCCESS — RDAP and every requested DNS/TLS surface were verified;
• PARTIAL — RDAP succeeded but a requested DNS/TLS surface did not;
• NOT_FOUND — the authoritative RDAP service returned HTTP 404;
• UNSUPPORTED_TLD — current IANA bootstrap has no HTTPS RDAP route;
• RATE_LIMITED — the authoritative registry still returned 429 after bounded retries;
• FAILED — transport, HTTP, schema, input or unexpected isolated failure.

NOT_FOUND never means source failure. For a previously known domain, the first authoritative 404 emits NOT_FOUND_PENDING. Only a second independent successful authoritative confirmation emits DOMAIN_EXPIRED_OR_DROPPED. A timeout, 429, 500, malformed JSON or bootstrap failure never counts as confirmation and never overwrites the last good state.

DNS and SSL/TLS certificate scope

With includeDns, the Actor queries only public A, AAAA, MX, NS and CAA. It does not dump TXT records. With includeTls, it makes one normal TLS connection to port 443 and returns:

• SHA-256 certificate fingerprint;
• subject hostname and issuer;
• valid_from, valid_to, days_to_expiry;
• SAN count and deterministic hostname match.

There is no port scan, cipher scan, CVE lookup, SSL Labs clone or risk score. Before TLS, resolved IPs are checked and private/reserved destinations are rejected to prevent SSRF.

TLS is disabled by default because a valid registered domain does not necessarily serve HTTPS on its bare hostname. Enable it only when that is part of the state you want to monitor; a requested TLS failure is deliberately PARTIAL, not silently ignored.

Rejected malformed inputs use an opaque placeholder in output. The original invalid string is never echoed because it may itself contain credentials or tokens.

Run daily or weekly

First run the Actor once with the final monitor input. Then create an Apify Schedule in Console, or call the official schedule endpoint. This daily example uses placeholders and an Authorization header so no token appears in a URL:

curl -X POST "https://api.apify.com/v2/schedules" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "daily-domain-rdap-monitor",
    "isEnabled": true,
    "isExclusive": true,
    "cronExpression": "0 8 * * *",
    "timezone": "UTC",
    "actions": [{
      "type": "RUN_ACTOR",
      "actorId": "YOUR_ACTOR_ID",
      "runInput": {
        "body": "{\"domains\":[{\"domain\":\"example.com\",\"externalId\":\"brand-main\"}],\"mode\":\"monitor\",\"outputMode\":\"changesOnly\",\"includeDns\":true,\"includeTls\":true}",
        "contentType": "application/json; charset=utf-8"
      }
    }]
  }'

For a weekly portfolio review, change the expression to 0 8 * * 1.

Webhook event workflow

Attach an Actor-run-succeeded webhook in Apify Console to your HTTPS endpoint. The webhook tells the receiver which run finished; read that run's default Dataset, then keep rows where change_type == "CHANGED". Every such row is already an alert payload with source_id, external_id, fingerprint, scraped_at and exact events.

Example receiver decision:

for (const row of datasetItems) {
  if (row.change_type !== 'CHANGED') continue;
  await sendAlert({
    domain: row.domain,
    externalId: row.external_id,
    events: row.events,
    observedAt: row.scraped_at,
  });
}

Use with AI agents through Apify MCP

Expose the Actor as a typed tool in an authenticated MCP-compatible client:

https://mcp.apify.com?tools=automa-flow/domain-rdap-change-monitor

Example prompt:

Run automa-flow/domain-rdap-change-monitor for these domains. Summarize expiry,
registrar, nameserver, DNSSEC, DNS, and TLS certificate facts; separate
NOT_FOUND from FAILED checks and cite each source_url.

After publication the same tool URL becomes discoverable to Store users. The Actor intentionally excludes registrant personal data from its output.

Output contract

One row describes one source observation at scraped_at. Stable identity and history-ready fields are always present:

source, source_id, source_url, scraped_at, schema_version, fingerprint

The fingerprint covers semantic registration and selected DNS/TLS state. It excludes scraped_at, Actor/run IDs, headers, rate-limit metadata and derived days_to_expiry, so an unchanged domain has the same fingerprint tomorrow. Rows are append-only observations; the Actor does not build a historical database. KVS stores only the last full success plus warning/drop-confirmation metadata.

See examples/sample_output.json and the Dataset views for observations, changes, expiry and failures.

Coverage and limits

• RDAP coverage varies by registry/TLD and changes with the IANA bootstrap.
• Live research on 2026-09-01 found supported authoritative RDAP for .com, .net, .org, .ai, .uk, .fr, .pl, .cz, .dev and .app; .io, .co, .de, .sk and .rs had no current IANA bootstrap route.
• Some registries omit expiration, registrar ID or DNSSEC. Missing fields stay null/UNKNOWN; they are never invented.
• Registry rate limits are independent. The Actor uses concurrency 2 per authority, bounded exponential backoff and Retry-After; it never rotates proxies to evade limits.
• A domain with no working DNS/TLS can be a valid registration. Disable the optional surface if you want RDAP-only success/state for that watchlist.
• A 404 is not a guarantee that a domain is available to register. Confirm with a registrar before financial or legal decisions.

Privacy, compliance and affiliation

Registrant name, email, phone, postal address, organization and all registrant/admin/technical contact fields are intentionally excluded even if a registry publishes them. No raw RDAP response or vCard is stored.

This Actor accesses public registry protocol endpoints and does not bypass authentication, CAPTCHA, paywalls or access controls. Users are responsible for registry terms, reasonable request rates and lawful downstream use. This Actor is not affiliated with, endorsed by or operated by ICANN, IANA or any registry.

Cost model

Direct RDAP, DNS and TLS use no browser and no proxy, so proxy cost is normally $0. One unique supported domain uses one authoritative RDAP request; DNS adds at most five RR queries and TLS adds one port-443 connection. Retries, FAILED, PARTIAL, RATE_LIMITED and UNSUPPORTED_TLD rows are not successful billable checks. An authoritative NOT_FOUND is a completed domain check.

Pay per event starts at $0.001 per verified distinct domain check on the Free plan ($1.00 per 1,000) and decreases by Apify subscription tier:

Apify plan	Price per verified domain	Price per 1,000
Free	$0.00100	$1.00
Bronze	$0.00090	$0.90
Silver	$0.00085	$0.85
Gold	$0.00080	$0.80
Platinum	$0.00075	$0.75
Diamond	$0.00070	$0.70

There is no run-start fee. Duplicate aliases, retries, invalid inputs, PARTIAL, FAILED, RATE_LIMITED and UNSUPPORTED_TLD results are not charged. Measured 1,000-domain RDAP platform cost at 512 MB is approximately $0.07 before conservative headroom, including compute, observed transfer, Dataset writes and monitor KVS operations at current Free/Bronze platform unit costs.

The run's maxTotalChargeUsd is enforced before network work. If the remaining budget cannot cover another unique domain, that domain is not contacted and receives an uncharged FAILED row with MAX_TOTAL_CHARGE_REACHED; prior monitor state is preserved.

The production release gate measured a 1,000-domain RDAP batch plus a 100-domain RDAP/DNS/TLS batch; no raw registration payload or contact data was retained in those QA artifacts.