Domain Intelligence Scraper

What does Domain Intelligence Scraper do?

Domain Intelligence Scraper performs comprehensive passive OSINT on any domain — combining 8 free data sources into one output record: WHOIS, RDAP, DNS records, subdomains (from 3 independent sources), open ports + CVEs, passive DNS history, IP geolocation, and HTTP security headers. No API key required.

Why use Domain Intelligence Scraper?

• Security auditing: Discover exposed subdomains, open ports, known CVEs, and missing security headers
• Competitor research: Identify hosting provider, CDN, email stack, and tech fingerprint (CPEs)
• Lead generation: Structured registrant data, creation date, expiry, and contact info via RDAP
• Subdomain enumeration: Aggregates crt.sh + THC (6B+ domain DB) + Wayback Machine + Robtex passive DNS
• Threat screening: Optional ThreatMiner lookup for malware associations and passive DNS history

How to use Domain Intelligence Scraper

1. Open the actor on Apify and click Try for free
2. Enter one or more domains (e.g. example.com, tesla.com)
3. Toggle which data sources to include
4. Click Start — results appear in the Output tab

Input

{
  "domains": ["example.com"],
  "includeWhois": true,
  "includeRdap": true,
  "includeDns": true,
  "includeSubdomains": true,
  "includeWayback": true,
  "includePassiveDns": true,
  "includeShodanInternetdb": true,
  "includeHttpHeaders": true,
  "includeIpInfo": true,
  "includeThreatIntel": false
}

Full URLs also accepted:

{
  "domains": ["https://example.com/about", "tesla.com"]
}

Output

Each item is one domain's full intelligence profile:

{
  "domain": "example.com",
  "ip": "93.184.216.34",
  "whois": "Domain Name: EXAMPLE.COM\nRegistrar: ICANN\nCreation Date: 1995-08-14...",
  "rdap": {
    "ldhName": "EXAMPLE.COM",
    "status": ["client delete prohibited", "client transfer prohibited"],
    "nameservers": [{"ldhName": "a.iana-servers.net"}, {"ldhName": "b.iana-servers.net"}],
    "events": [
      {"eventAction": "registration", "eventDate": "1995-08-14T04:00:00Z"},
      {"eventAction": "expiration", "eventDate": "2025-08-13T04:00:00Z"}
    ]
  },
  "dns": {
    "A": ["93.184.216.34"],
    "NS": ["a.iana-servers.net.", "b.iana-servers.net."],
    "MX": ["0 ."],
    "TXT": ["\"v=spf1 -all\""],
    "SOA": ["ns1.example.com. admin.example.com. 2024010101 3600 900 604800 300"]
  },
  "subdomains": ["api.example.com", "mail.example.com", "www.example.com"],
  "emails_from_certs": [],
  "certificates_raw": [...],
  "passive_dns": {
    "forward": [
      {"rrname": "example.com", "rrdata": "93.184.216.34", "rrtype": "A", "time_last": "2024-01-01"},
      {"rrname": "example.com", "rrdata": "192.0.2.1", "rrtype": "A", "time_last": "2022-06-15"}
    ]
  },
  "shodan": {
    "ip": "93.184.216.34",
    "ports": [80, 443],
    "vulns": ["CVE-2021-44228"],
    "cpes": ["cpe:/a:apache:httpd:2.4.50"],
    "hostnames": ["example.com"],
    "tags": []
  },
  "http_headers": "HTTP/1.1 200 OK\nServer: nginx\nStrict-Transport-Security: max-age=31536000\nX-Frame-Options: DENY\n...",
  "ip_info": {
    "status": "success",
    "country": "United States",
    "regionName": "California",
    "city": "Los Angeles",
    "isp": "Edgecast Inc.",
    "org": "EDGECAST",
    "as": "AS15133 Edgecast Inc.",
    "query": "93.184.216.34"
  }
}

Data sources

Source	Field	Data	Auth
whois command	whois	Raw WHOIS text — registrar, dates, status	None
RDAP	rdap	Structured JSON WHOIS — registrant, events, nameservers	None
DNS resolver (dnspython)	dns	A, AAAA, MX, NS, TXT, CNAME, SOA	None
crt.sh	subdomains, certificates_raw	SSL Certificate Transparency logs	None
THC	subdomains	Subdomains from 6B+ domain database	None
Wayback Machine CDX	subdomains	Subdomains from historical URL crawl index	None
Robtex	passive_dns	Historical IP resolutions (forward + reverse)	None
Shodan InternetDB	shodan	Open ports, known CVEs, CPEs, service tags	None
HackerTarget	http_headers	Full HTTP response headers	None
ip-api.com	ip_info	Country, city, ISP, ASN	None
ThreatMiner	threat_intel	Malware hashes, passive DNS, SSL certs (optional)	None

Data fields

Field	Description
domain	Normalized input domain
whois	Raw WHOIS output
rdap	Structured WHOIS — use rdap.events for registration/expiry dates, rdap.nameservers for NS
dns.A / dns.AAAA	IPv4 / IPv6 addresses
dns.MX	Mail server records
dns.TXT	SPF, DMARC, Google/Microsoft verification tokens
dns.NS	Nameservers
subdomains	Deduplicated list from all 4 subdomain sources
emails_from_certs	Email addresses embedded in certificate fields
certificates_raw	Raw crt.sh entries (issuer, validity, SAN fields)
passive_dns.forward	Historical A/CNAME records — tracks IP changes over time
shodan.ports	Open TCP ports detected on the domain's IP
shodan.vulns	Known CVE IDs associated with running services
shodan.cpes	CPE identifiers (tech fingerprint, e.g. cpe:/a:nginx:nginx:1.24)
http_headers	Raw HTTP response headers including security headers
ip	Resolved IPv4 address
ip_info	Country, city, ISP, ASN of the resolved IP
threat_intel	ThreatMiner: malware hashes, passive DNS, SSL certs (if enabled)

Pricing

• $0.05 per run start
• $0.05 per domain analyzed

Analyzing 100 domains costs approximately $5.05.

FAQ & Disclaimers

Is this legal? Yes — all sources are public. WHOIS and DNS are public infrastructure, crt.sh logs are mandated by CA/Browser Forum, and all other APIs are free public services.

Why are some fields empty? crt.sh can be slow or temporarily unavailable. New domains with no SSL history return no subdomains from crt.sh. ThreatMiner has limited data for small/new domains. Empty is always better than an error.

Why is IP geolocation showing a CDN location instead of the real server? Domains behind Cloudflare, Akamai, or Fastly resolve to CDN edge IPs — the geo data reflects the CDN's nearest PoP, not the origin server. Check shodan.cpes for cpe:/a:cloudflare:cloudflare to detect this.

What is includeThreatIntel and why is it off by default? ThreatMiner enforces a 10 req/min rate limit, so enabling it adds ~30 seconds per domain. Turn it on when you specifically need malware association data.

Support: Open an issue in the Issues tab.