Domain WHOIS & DNS Intelligence Scraper

One row per domain: WHOIS/RDAP registration data, the full DNS zone, mail security posture (SPF, DMARC, DNSSEC) and the detected DNS + mail provider — combined, normalised, and ready for your CRM or spreadsheet. No API key, no account, no proxy required.

Paste a list of domains, get back a complete technical and administrative profile of each one. The actor talks to the authoritative RDAP registry (via the official IANA bootstrap at rdap.org) and to Google Public DNS over HTTPS, then merges both worlds into a single flat record.

---

How this differs from my two single-purpose actors

I already publish two focused, cheaper actors. This one is the combined product — use it when you want the whole picture in one row instead of joining two datasets yourself.

	RDAP Domain Scraper	DNS Records Scraper	Domain Intel Scraper (this one)
Registrar, creation & expiry dates	Yes	No	Yes
Domain status codes, DNSSEC	Yes	No	Yes
A / AAAA / MX / NS / TXT / SOA	No	Yes	Yes
SPF record + has_spf flag	No	Partly (raw TXT only)	Yes, parsed
DMARC record + p= policy	No	No	Yes, parsed
Nameserver provider detection	No	No	Yes (60+ providers)
Mail provider detection from MX	No	No	Yes (35+ providers)
Computed age_days / days_until_expiry	No	No	Yes
Output shape	RDAP-only row	DNS-only row	One enriched profile row

If you only need WHOIS data, use the RDAP actor. If you only need raw DNS answers, use the DNS actor. If you need a domain intelligence profile — registration + infrastructure + email security in one line — use this one.

---

What you get per domain

{
  "domain": "stripe.com",
  "registrar": "SafeNames Ltd.",
  "registered_at": "1995-09-12T04:00:00Z",
  "expires_at": "2027-09-11T04:00:00Z",
  "updated_at": "2025-10-01T01:39:51Z",
  "age_days": 11277,
  "days_until_expiry": 409,
  "status": ["client delete prohibited", "client transfer prohibited"],
  "dnssec": false,
  "nameservers": ["ns-1087.awsdns-07.org", "ns-423.awsdns-52.com"],
  "ns_provider": "AWS Route 53",
  "a_records": ["198.202.176.41"],
  "aaaa_records": [],
  "mx_records": ["10 aspmx.l.google.com", "20 alt1.aspmx.l.google.com"],
  "mx_provider": "Google Workspace",
  "txt_records": ["google-site-verification=...", "v=spf1 ip4:198.2.180.60/32 ..."],
  "spf": "v=spf1 ip4:198.2.180.60/32 include:spf1.stripe.com ~all",
  "has_spf": true,
  "dmarc": "v=DMARC1; p=reject; pct=100; rua=mailto:dmarc-reports@stripe.com;",
  "has_dmarc": true,
  "dmarc_policy": "reject",
  "soa": "ns-1882.awsdns-43.co.uk. awsdns-hostmaster.amazon.com. 1 900 900 1209600 3600",
  "ttl_a": 30,
  "source": "rdap+doh",
  "scraped_at": "2026-07-28T14:31:02Z"
}

Every field is nullable — a domain without MX records simply gets an empty list, a ccTLD without RDAP coverage still returns its complete DNS profile.

---

Input

Field	Type	Default	Description
mode	select	domains	Currently one mode: a domain list in, one intelligence row per domain out.
domains	array	["stripe.com", "github.com"]	Domains or full URLs. Scheme, www., path, query and port are stripped automatically.
maxItems	integer	100 (max 1000)	Hard cap on processed domains, so cost stays predictable.

{
  "mode": "domains",
  "domains": ["stripe.com", "github.com", "https://www.shopify.com/pricing"],
  "maxItems": 100
}

---

Use cases

• Domain due diligence — before you buy, partner with, or link to a domain: how old is it, who is the registrar, when does it expire, is DNSSEC on, is the registrar lock set?
• Lead enrichment — append the tech stack to your CRM. Knowing a prospect runs Google Workspace on Cloudflare DNS tells your sales team more than any firmographic field. Perfect for segmenting an outbound list by mail provider or hosting provider.
• Security posture auditing — find every domain in your portfolio (or your customer base) that is missing SPF, missing DMARC, or sitting on p=none. Those are the domains that can be spoofed today.
• M&A and portfolio research — map the DNS and mail infrastructure of an acquisition target's whole domain estate in one run, and spot expiring domains before they lapse.
• Expiry monitoring — schedule the actor weekly and alert on days_until_expiry < 30.
• Competitive intelligence — see which CDN, DNS and email vendors your competitors migrated to, and when the zone last changed.

---

Pricing

Roughly $1 per 1,000 domains in platform compute — the actor runs on 512 MB, uses no proxies, and issues only lightweight JSON requests (1 RDAP call + 7 DNS-over-HTTPS calls per domain). There is nothing to pay to a WHOIS API vendor on top.

Speed & reliability

• Fully async, 10 domains resolved in parallel, pushed in batches of 50.
• curl_cffi with a Chrome 124 TLS fingerprint, with a standard-library urllib fallback.
• A single failing domain never kills the run; it is logged and skipped.
• The run fails loudly (RuntimeError) only if no domain could be resolved at all.

Data source notes

• RDAP is the IETF successor to WHOIS and is served by the registries themselves. Coverage is excellent for gTLDs (.com, .net, .org, .io-style new gTLDs, .dev, .app) and for many ccTLDs. Some ccTLD registries (for example .de) do not publish an RDAP endpoint — those domains still get a complete DNS, SPF and DMARC profile, with the registration fields left null.
• Registrant contact details are redacted by registries under ICANN's GDPR-driven temporary specification. This actor therefore returns the registrar, not personal registrant data.
• DNS answers come from Google Public DNS (dns.google), so they reflect a public resolver's view of the zone.

Related actors

• RDAP Domain Scraper — registration data only, leanest option.
• DNS Records Scraper — raw DNS record types only.
• SSL Certificate Checker — issuer, validity window and expiry of the TLS certificate.
• Security Headers Checker — HSTS, CSP and the rest of the HTTP header posture.

Run all four over the same domain list for a complete external security and infrastructure profile.

Local development

pip install -r requirements.txt
python -m src.main
python tests/test_smoke.py   # offline smoke tests, no network needed

Legal

The actor reads only public registry (RDAP) and public DNS data. It performs no port scanning, no authentication and no intrusive probing.