🔎 Domain Intelligence Scraper: WHOIS, DNS, Email Security, SSL & Subdomains

Domain Intelligence Scraper returns WHOIS/RDAP registration data, DNS records, SPF/DMARC/DKIM email security with an A to F grade, SSL certificate expiry, an HTTP check with all response headers, subdomains from certificate logs, optional reverse DNS and Wayback Machine history, and availability for a list of domains, one row per domain. No login and no API key. Export to CSV, Excel, JSON or XML. Switch on monitor mode and scheduled runs deliver only new or changed domains. Pricing is pay per result and failed lookups are never charged.

Paste domains or full URLs and get one row per domain. Data comes from registry RDAP servers, Google DNS-over-HTTPS, a direct TLS handshake and HTTPS request, crt.sh, Cert Spotter and the Internet Archive. For apify.com a cloud run returned registrar Amazon Registrar, Inc. and a domain age of 17.3 years.

🔎 What does Domain Intelligence Scraper do?

• WHOIS via RDAP through each TLD's registry, plus domain availability (isRegistered: false, cross-checked against DNS).
• DNS records (A, AAAA, MX, NS, TXT, CNAME, SOA, CAA) with DNS and mail provider labels.
• Email security: SPF, DMARC and DKIM into emailScore, emailGrade (A to F) and emailIssues.
• Live SSL certificate and one HTTP check (status, redirects, title, all response headers).
• Opt-in reverse DNS, Wayback Machine history and subdomains from certificate logs.
• Monitor mode: with onlyChanged, scheduled runs return only new or changed domains.

📋 What data can you extract with Domain Intelligence Scraper?

Three record types share one fixed column order of 87 keys, split by recordType: domain, dns-record and subdomain.

Group	Main fields (domain row)
Identity	recordType, domain, url, changeType, scrapedAt
Registration	isRegistered, registrar, registrarIanaId, registrarAbuseEmail, createdAt, expiresAt, domainAgeYears, daysToExpiry, statuses, dnssec, nameServers
DNS	dnsA, dnsAaaa, reverseDns, dnsMx, dnsNs, dnsCname, dnsSoa, dnsCaa, dnsTxt, dnsProvider, mailProvider
Email security	hasSpf, spfRecord, spfPolicy, hasDmarc, dmarcPolicy, dmarcPct, hasDmarcReporting, hasDkim, dkimSelectors, emailScore, emailGrade, emailIssues
SSL	sslIssuer, sslValidFrom, sslValidTo, sslDaysUntilExpiry, sslProtocol, sslSan
HTTP	httpStatus, finalUrl, redirectCount, pageTitle, server, poweredBy, hasHsts, securityHeaders, httpHeaders, httpResponseMs
Subdomains	subdomainCount, subdomains, subdomainSource (crt.sh or certspotter)
Wayback	waybackFirstSeen, waybackLastSeen, waybackSnapshotCount

A dns-record row (only with DNS detail rows) adds dnsType, value and ttl. A subdomain row adds subdomain, wildcard, firstSeen, lastSeen and issuer. DNS, SSL and HTTP fields are left out for unregistered domains, and empty lists are not emitted. reverseDns is filled only when the IPs publish PTR names (Google, GitHub and AWS do; Cloudflare does not). httpHeaders is an object of lowercase header names (no Set-Cookie, 10 KB per row). A domain never captured by Wayback gets waybackSnapshotCount 0. Availability rows carry N/A in the registration fields.

📊 Domain Intelligence Scraper sample output

Real row from an earlier cloud run (trimmed; it predates the email security and HTTP columns):

{
  "recordType": "domain",
  "domain": "apify.com",
  "isRegistered": true,
  "registrar": "Amazon Registrar, Inc.",
  "expiresAt": "2035-06-02T17:14:10Z",
  "domainAgeYears": 17.3,
  "dnssec": true,
  "dnsTxtCount": 11,
  "hasSpf": true,
  "sslIssuer": "Amazon",
  "sslDaysUntilExpiry": 105,
  "url": "https://apify.com",
  "scrapedAt": "2026-10-04T10:00:00.000Z"
}

💰 How much does Domain Intelligence Scraper cost?

Pay per event. You are charged only for rows delivered. Prices are live on this actor (read 2026-10-10).

Event	Price (Free plan)	Price (Gold and above)
Domain row (one per domain)	$0.015	$0.01329
DNS detail row (only if enabled)	$0.01	$0.01
Subdomain row (only if enabled)	$0.002	$0.002

Worked example at Free plan prices: 1,000 domains cost $15.00 (1,000 x $0.015), or $13.29 on the Gold tier. Email security, SSL, headers, reverse DNS, Wayback and the subdomain list are part of the domain row at no charge. A domain with 100 subdomain rows adds $0.20 to its $0.015 domain row. With onlyChanged, a daily run over 1,000 domains where 20 changed costs 20 x $0.015 = $0.30, because unchanged domains are skipped and not charged. Failed lookups and invalid input are free.

🚀 How to use Domain Intelligence Scraper (3 steps)

1. Open the actor and paste your domains (URLs and www. prefixes are cleaned).
2. Choose the modules (WHOIS, DNS, email security, SSL, HTTP, Wayback, subdomains).
3. Click Start, then download CSV, Excel, JSON or XML.

⚙️ Domain Intelligence Scraper input

Field	Default	Meaning
domains	["apify.com","example.org"]	Domains or URLs to analyze (required)
includeWhois, includeDns	true	Registration data; DNS lookups plus DNS and mail provider
dnsRecordTypes	A, MX, NS, TXT	Record types to resolve (A, AAAA, MX, NS, TXT, CNAME, SOA, CAA)
includeEmailSecurity, dkimSelectors	true, empty	SPF, DMARC, DKIM, grade; extra DKIM selectors to try, such as mta
includeSsl, includeHttp	true	Live certificate on port 443; one request to https://domain, up to 5 redirects
includeHttpHeaders	true	All response headers in httpHeaders (needs includeHttp)
includeReverseDns	false	PTR names for up to 4 IPv4 and 2 IPv6 addresses
includeWayback	false	First and last Internet Archive capture and snapshot count
includeSubdomains, includeSubdomainRows	false	Subdomain count and list on the domain row; one row per subdomain
maxSubdomainsPerDomain	200	Cap on the list and rows (1 to 5,000); the count stays true
includeDnsDetailRows	false	Extra row per DNS record with TTL
maxItems	100	Row cap (free users: 10)
onlyChanged, stateKey, resetSeen	false, empty, false	Deliver only new or changed domains; monitor name; forget previous results

Example: monitor changes on a schedule with onlyChanged.

{
  "domains": ["apify.com", "example.org"],
  "onlyChanged": true,
  "stateKey": "client-portfolio",
  "maxItems": 500
}

📦 Domain Intelligence Scraper output

Results go to the default dataset; use recordType to split domain, DNS and subdomain rows. A run with no valid domain stops with a clear error and charges nothing. The onlyChanged fingerprint covers registration (registrar, expiry, statuses, DNSSEC, name servers), DNS (MX, NS, CNAME, CAA), email (SPF, DMARC, DKIM), certificate (issuer, expiry) and HTTP (status, final URL). Reverse DNS, headers, Wayback, A and AAAA addresses and subdomains are excluded to avoid false changes.

⚖️ Domain Intelligence Scraper vs alternatives

Read on 2026-10-10 from the public Apify Store pages. "Not stated" means the page does not mention it.

Capability	This actor	blackfalcondata	shipsatnight
Email auth and grade	SPF, DMARC, DKIM, A to F	Same, plus pass/fail verdicts	Not stated
Change monitoring	onlyChanged	NEW, UPDATED, UNCHANGED, alerts	Not stated
Subdomains, Wayback	Both, Wayback with count	Not stated	Both, Wayback first snapshot only
Headers, reverse DNS	All headers, PTR names	Not stated	Selected headers, no PTR
Blacklist, tech stack	Not built	Blacklist yes	Tech stack yes
Price per 1,000	$15 (Free), $13.29 (Gold)	from $1.00	from $0.50

We are the more expensive option: blackfalcondata charges $0.001 per record and shipsatnight about $0.0056 per domain with five modules. jungle_synthesizer/dns-domain-audit ($5.00 per 1,000 plus a $0.10 start) states reverse DNS and email auth; apivault_labs lists $1.50 and $10 per 1,000 on one page. Not built: blacklist, tech stack, social links and built-in Slack alerts (Apify integrations cover that). Prices drift.

💼 Use cases

• Email deliverability audits: grade SPF, DMARC and DKIM across client domains and list what to fix.
• Domain and certificate monitoring: schedule onlyChanged runs to catch expiring domains and certificates.
• Due diligence and lead enrichment: domain age, registrar, mail provider and Wayback history.
• Attack surface mapping and domain investing: subdomains, bulk availability and expiry screening.

🔌 Run Domain Intelligence Scraper via API and integrations

curl -X POST "https://api.apify.com/v2/acts/recordsdata~domain-intelligence-scraper/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "content-type: application/json" \
  -d '{"domains":["apify.com"],"maxItems":10}'

Use the JavaScript or Python client, schedules, Make, Zapier, n8n, Google Sheets or Airbyte.

🤖 Domain Intelligence Scraper MCP server for AI agents (Claude, Cursor, VS Code)

Connect this actor to any MCP client through the Apify MCP server. This URL exposes only this actor:

https://mcp.apify.com?tools=RecordsData/domain-intelligence-scraper

Config for JSON MCP clients:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=RecordsData/domain-intelligence-scraper"
    }
  }
}

Example agent prompt: "Use the Domain Intelligence Scraper on apify.com, github.com and python.org with WHOIS, DNS, email security, SSL and Wayback on, then list the registrar, domain age, first Wayback capture, email grade and days until SSL expiry for each, and flag any domain with a DMARC policy of none."

Tool description: choose this actor when an agent needs registration, DNS, email security (SPF, DMARC, DKIM grade), SSL, HTTP status and headers, reverse DNS, Wayback history or subdomain facts, or a domain availability check, for a list of domains in one call. Set maxItems to keep runs small.

🛡️ Is it legal to look up domain data?

The actor queries public endpoints made for this purpose: RDAP registries, DNS-over-HTTPS, a TLS handshake and an HTTPS request any browser performs, and public certificate logs. Registrant personal data is often redacted and is not collected; the abuse contact is kept only when it is a role mailbox. You are responsible for using results in line with the law.

❓ Frequently asked questions

What is RDAP and how is it different from WHOIS?

RDAP is the structured, JSON-based replacement for the old WHOIS protocol, served by each TLD's registry. The scraper routes every domain to the correct registry RDAP server through the IANA bootstrap file, so you get registrar and dates as clean fields instead of free text.

Why does a domain show limited registration data in Domain Intelligence Scraper?

Some registries publish only minimal RDAP records, so a few registration fields can be missing for certain TLDs. The scraper still reports whether the domain is registered through its DNS cross-check and fills every field the registry does provide, such as registrar and dates.

Why did I get 0 results or a failed run from the domain scraper?

The run fails with a clear message when every lookup fails because the data sources are unreachable. A list with no valid domain ends the run with one free error row, and with onlyChanged on, a second run with nothing new returns 0 rows. Nothing is charged in any of these cases.

What happens if I send an empty or invalid input to Domain Intelligence Scraper?

The platform rejects a missing or empty domains list with a 400 error, so nothing is created or charged. A list where no entry is a valid domain ends the run as SUCCEEDED with one free error row (errorType: invalid-input plus a hint).

Does the SSL certificate check follow redirects?

No. The scraper does a direct TLS handshake with the domain itself on port 443 and reports that certificate: issuer, valid-to date, days until expiry and SAN list. It does not check www. variants, so submit each hostname you want to monitor. The separate HTTP check follows up to 5 redirects.

How do I check if a domain is available with Domain Intelligence Scraper?

Check the isRegistered field. When it is false, the registry has no record and DNS has no nameservers for the domain, so it is available to register. The scraper cross-checks RDAP against DNS to avoid false positives on TLDs with weak RDAP. Availability rows are billed as normal domain rows.

Am I charged for failed domain lookups?

No. A row whose WHOIS lookup failed carries an error field and is delivered without a charge, and rows for invalid input are free too. You pay only for rows with real results, and the run stops cleanly at your maximum charge limit.

Can I try Domain Intelligence Scraper for free?

Free plan users get a 10-row preview per run, enough to test a small list. At Free plan prices, those 10 domain rows cost $0.15. Paid Apify plans unlock larger runs and lower per-row prices.

Can I get one row per DNS record from Domain Intelligence Scraper?

Yes. Enable includeDnsDetailRows and the scraper adds a separate row for every DNS record, with its type, value and TTL, which suits DNS audits. Each row is billed as a DNS detail row at $0.01, on top of the domain row.

How is emailGrade computed in Domain Intelligence Scraper?

The emailScore runs from 0 to 100: SPF 25 points, DMARC 45, DKIM 20, a DMARC reporting address 5 and DNSSEC 5. The emailGrade is A from 90, B from 75, C from 55, D from 35, E from 15, otherwise F. Domains with no real MX record, including a null MX, are not graded.

How do I monitor domains for changes with Domain Intelligence Scraper?

Switch on onlyChanged, name the monitor with stateKey, and schedule the run. A domain is delivered when it is new or when its registration, DNS, email, certificate or HTTP fingerprint moved, with changeType new or changed. Unchanged domains are skipped and free. resetSeen starts over.

How are subdomains found and what is missed?

The actor reads public certificate transparency logs through crt.sh, with Cert Spotter as fallback, and reports the source in subdomainSource. A subdomain that never had a public certificate is not listed. The list is capped by maxSubdomainsPerDomain, while subdomainCount shows the true total. Each subdomain row costs $0.002.

Why does DKIM show false for a domain that uses DKIM?

DKIM keys sit under a selector name only the sender knows, so no lookup can list them. The actor probes 14 common selectors (google, default, selector1, selector2, k1, s1, mail and more) plus yours in dkimSelectors. A private selector shows hasDkim: false until you add it.

How do I get Wayback Machine history and reverse DNS?

Switch on includeWayback for first and last capture, snapshot count and links to the oldest and newest capture, taken from the Internet Archive. Switch on includeReverseDns for PTR names of the domain IPs. Both are off by default and add no charge. Many shared hosts publish no PTR name, so the column is then absent.

Does Domain Intelligence Scraper return all HTTP headers?

Yes, httpHeaders holds every response header of the home page as an object of lowercase names, with no extra request. Set-Cookie is dropped, values are cut at 1,000 characters and the object stops at 10 KB. Turn it off with includeHttpHeaders: false for slimmer rows.

🧰 Want more domain and web data? Other RecordsData scrapers

• Subdomain Finder Scraper
• Wayback Machine Scraper

💬 Support

Open the Issues tab or write to contact.punkrecordsdata@gmail.com. Custom solutions available on request.

Last updated: 2026-10-11