Domain Intelligence

Collect public registration, DNS, mail-policy, TLS, and certificate-transparency observations in one Apify dataset. Supply domain names and receive one row per input entry, including failures. This actor helps with domain inventories, infrastructure research, and initial configuration reviews. It uses public services without API keys and does not log into websites, enumerate accounts, send email, or attempt exploitation.

Input

domains is required: an array containing 1–1,000 domain strings, for example python.org or mozilla.org. Whitespace, trailing dots, case, and internationalized names are normalized. Supply names rather than URLs, paths, ports, or IP literals. Invalid names produce an error row while subsequent entries continue. Duplicate entries remain separate observations and, when successful, separate events.

All five feature switches default to true. includeWhois requests structured registration data through the rdap.org bootstrap redirect service; it does not scrape legacy WHOIS text. includeDns exports A, AAAA, MX, NS, TXT, CNAME, and SOA answers using dnspython and the machine's configured resolver. includeSsl connects to port 443 with certificate verification and hostname-aware SNI. includeEmailSecurity checks SPF, DMARC, and six common DKIM selectors. includeSubdomains queries crt.sh certificate-transparency JSON.

maxSubdomains defaults to 200 and accepts 0–10,000. It limits the returned list after deduplication and sorting, rather than limiting certificate history downloaded. timeoutSecs defaults to 15 and accepts integers from 1–60. It controls individual DNS/HTTP operations and the TLS connection attempt budget, not a whole-domain deadline. The copied network helper's initial public-address resolution uses five seconds per address-family query. Domains run sequentially to reduce bursts against shared public services.

Results

Each row includes the normalized domain, original input, UTC observation timestamp, success flag, error, warnings, and elapsed seconds. Registration fields are registrar, createdAt, expiresAt, updatedAt, nameservers, and registrantCountry. Missing or redacted registration values remain null. Only a country actually exposed in the registrant entity is reported; the registrar's address is not substituted.

dns holds record arrays, including joined TXT string fragments. ipAddresses, ip, asn, and asnName are collected independently of the optional DNS export. The ASN lookup function is copied from the sibling detector, with no runtime import from that package. It uses Team Cymru DNS and an optional ARIN network lookup. An edge/CDN ASN identifies the observed network, not necessarily the website owner's origin hosting.

mxProvider is a list of suffix-based guesses, including Google, Microsoft, Zoho, Proton, Fastmail, Mimecast, Proofpoint, Amazon SES, and Yahoo. Boundary matching avoids misleading suffix lookalikes. Unknown MX hosts yield an empty guess list. Mail gateways can conceal the downstream mailbox provider.

emailSecurity includes SPF records and the observed all mechanism, DMARC records and policy, and DKIM results for default, google, selector1, selector2, k1, and mailjet. These are summaries, not full protocol compliance checks. SPF includes are not recursively evaluated; organizational-domain DMARC fallback is not calculated. Missing DKIM selectors do not prove DKIM is disabled. DNS failures are marked unavailable rather than silently reported as absent.

ssl contains issuer attributes, DNS SANs, UTC validity dates, whole days remaining, and the negotiated TLS protocol. Only verified certificates are returned. TLS failures produce warnings and a null summary. subdomains contains in-scope names from certificate history, excluding the apex. Wildcards are reduced to their named base. subdomainCount counts returned names; subdomainsDiscovered counts deduplicated names before the limit; subdomainsTruncated identifies clipping. Historical names may no longer resolve, and certificate transparency cannot provide an exhaustive inventory.

Local execution

Use Python 3.12 and a dedicated environment from this directory:

python -m venv .venv
.venv/Scripts/python.exe -m pip install -r requirements.txt
$env:APIFY_LOCAL_STORAGE_DIR = "$PWD/storage/manual"
New-Item -ItemType Directory -Force "$env:APIFY_LOCAL_STORAGE_DIR/key_value_stores/default"
Copy-Item INPUT.json "$env:APIFY_LOCAL_STORAGE_DIR/key_value_stores/default/INPUT.json"
.venv/Scripts/python.exe -m src
.venv/Scripts/python.exe -m unittest discover -s tests -v

The included INPUT.json contains six validation domains. validation/run_live.ps1 runs that input in fresh local storage and records rows, elapsed time, and exit status. Read VALIDATION.md for actual observations and limitations. The installation helper documents this machine's restricted temporary-directory workaround; ordinary installations should use pip directly.

Events and reliability

The declared domain-analyzed event costs $0.006 per successful domain. Success means at least one requested source returned usable domain data, including partial results with warnings. Invalid inputs and domains with no usable data are uncharged. The actor stores each row before requesting its event. Persistence and charging are not atomic across interruptions. Local SDK warnings about ignored charges are expected; the declaration must be configured in Apify Console before publication.

Public services can time out, throttle, redact data, or return incomplete histories. JSON responses are limited to 20 MiB. Inspect warnings before interpreting empty fields. Deployment, actual billing, and Store publication are separate from local validation. Reference behavior: RDAP bootstrap, dnspython resolver, and Apify Actor API.

Example output

One real dataset row from validation/results.json, trimmed by omitting fields without changing retained values:

{
  "input": "python.org",
  "domain": "python.org",
  "analyzedAt": "2026-09-26T06:58:02.568736+00:00",
  "success": true,
  "registrar": "Gandi SAS",
  "registrantCountry": null,
  "ip": "151.101.0.223",
  "asn": 54113,
  "mxProvider": [],
  "subdomainCount": 73,
  "subdomainsDiscovered": 73,
  "subdomainsTruncated": false,
  "warnings": [],
  "error": null
}

Raw DNS records, certificate details, and the subdomain list are omitted. The null registrant country reflects withheld registration data; an empty MX-provider guess is not evidence that mail records are absent.

Use cases

• IT operations teams can review registration expiry and TLS observations for a managed domain list.
• Email administration teams can triage domains with missing or unavailable SPF and DMARC observations.
• Security assessment teams can collect certificate-history names as candidates for a separately verified asset inventory.
• Acquisition diligence teams can compare registrar, nameserver, and visible network observations across target domains.

Pricing example: 500 successful domain analyses, including partial successes x $0.006 per domain-analyzed event = $3.00 in event fees, using .actor/pay_per_event.json. Local runs do not bill.

Limitations

A successful row may still contain incomplete source data. Review warnings and observation timestamps before acting on a missing field. Certificate-history names need separate resolution checks, and the six-selector DKIM probe cannot establish whether a domain signs mail with another selector.