# Domain Intelligence Scraper (`romy/domain-intelligence-scraper`) Actor

Domain Intelligence Scraper performs comprehensive passive OSINT on any domain — combining 8 free data sources into one output record: WHOIS, RDAP, DNS records, subdomains (from 3 independent sources), open ports + CVEs, passive DNS history, IP geolocation, and HTTP security headers

- **URL**: https://apify.com/romy/domain-intelligence-scraper.md
- **Developed by:** [Romy](https://apify.com/romy) (community)
- **Categories:** Developer tools, Automation, Other
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $50.00 / 1,000 domain analyzeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

### What does Domain Intelligence Scraper do?

**Domain Intelligence Scraper** performs comprehensive passive OSINT on any domain — combining 8 free data sources into one output record: WHOIS, RDAP, DNS records, subdomains (from 3 independent sources), open ports + CVEs, passive DNS history, IP geolocation, and HTTP security headers. No API key required.

### Why use Domain Intelligence Scraper?

- **Security auditing**: Discover exposed subdomains, open ports, known CVEs, and missing security headers
- **Competitor research**: Identify hosting provider, CDN, email stack, and tech fingerprint (CPEs)
- **Lead generation**: Structured registrant data, creation date, expiry, and contact info via RDAP
- **Subdomain enumeration**: Aggregates crt.sh + THC (6B+ domain DB) + Wayback Machine + Robtex passive DNS
- **Threat screening**: Optional ThreatMiner lookup for malware associations and passive DNS history

### How to use Domain Intelligence Scraper

1. Open the actor on Apify and click **Try for free**
2. Enter one or more **domains** (e.g. `example.com`, `tesla.com`)
3. Toggle which data sources to include
4. Click **Start** — results appear in the **Output** tab

### Input

```json
{
  "domains": ["example.com"],
  "includeWhois": true,
  "includeRdap": true,
  "includeDns": true,
  "includeSubdomains": true,
  "includeWayback": true,
  "includePassiveDns": true,
  "includeShodanInternetdb": true,
  "includeHttpHeaders": true,
  "includeIpInfo": true,
  "includeThreatIntel": false
}
```

Full URLs also accepted:

```json
{
  "domains": ["https://example.com/about", "tesla.com"]
}
```

### Output

Each item is one domain's full intelligence profile:

```json
{
  "domain": "example.com",
  "ip": "93.184.216.34",
  "whois": "Domain Name: EXAMPLE.COM\nRegistrar: ICANN\nCreation Date: 1995-08-14...",
  "rdap": {
    "ldhName": "EXAMPLE.COM",
    "status": ["client delete prohibited", "client transfer prohibited"],
    "nameservers": [{"ldhName": "a.iana-servers.net"}, {"ldhName": "b.iana-servers.net"}],
    "events": [
      {"eventAction": "registration", "eventDate": "1995-08-14T04:00:00Z"},
      {"eventAction": "expiration", "eventDate": "2025-08-13T04:00:00Z"}
    ]
  },
  "dns": {
    "A": ["93.184.216.34"],
    "NS": ["a.iana-servers.net.", "b.iana-servers.net."],
    "MX": ["0 ."],
    "TXT": ["\"v=spf1 -all\""],
    "SOA": ["ns1.example.com. admin.example.com. 2024010101 3600 900 604800 300"]
  },
  "subdomains": ["api.example.com", "mail.example.com", "www.example.com"],
  "emails_from_certs": [],
  "certificates_raw": [...],
  "passive_dns": {
    "forward": [
      {"rrname": "example.com", "rrdata": "93.184.216.34", "rrtype": "A", "time_last": "2024-01-01"},
      {"rrname": "example.com", "rrdata": "192.0.2.1", "rrtype": "A", "time_last": "2022-06-15"}
    ]
  },
  "shodan": {
    "ip": "93.184.216.34",
    "ports": [80, 443],
    "vulns": ["CVE-2021-44228"],
    "cpes": ["cpe:/a:apache:httpd:2.4.50"],
    "hostnames": ["example.com"],
    "tags": []
  },
  "http_headers": "HTTP/1.1 200 OK\nServer: nginx\nStrict-Transport-Security: max-age=31536000\nX-Frame-Options: DENY\n...",
  "ip_info": {
    "status": "success",
    "country": "United States",
    "regionName": "California",
    "city": "Los Angeles",
    "isp": "Edgecast Inc.",
    "org": "EDGECAST",
    "as": "AS15133 Edgecast Inc.",
    "query": "93.184.216.34"
  }
}
```

### Data sources

| Source | Field | Data | Auth |
|--------|-------|------|------|
| `whois` command | `whois` | Raw WHOIS text — registrar, dates, status | None |
| [RDAP](https://rdap.org) | `rdap` | Structured JSON WHOIS — registrant, events, nameservers | None |
| DNS resolver (dnspython) | `dns` | A, AAAA, MX, NS, TXT, CNAME, SOA | None |
| [crt.sh](https://crt.sh) | `subdomains`, `certificates_raw` | SSL Certificate Transparency logs | None |
| [THC](https://ip.thc.org) | `subdomains` | Subdomains from 6B+ domain database | None |
| [Wayback Machine CDX](https://web.archive.org/cdx/) | `subdomains` | Subdomains from historical URL crawl index | None |
| [Robtex](https://www.robtex.com) | `passive_dns` | Historical IP resolutions (forward + reverse) | None |
| [Shodan InternetDB](https://internetdb.shodan.io) | `shodan` | Open ports, known CVEs, CPEs, service tags | None |
| [HackerTarget](https://hackertarget.com) | `http_headers` | Full HTTP response headers | None |
| [ip-api.com](https://ip-api.com) | `ip_info` | Country, city, ISP, ASN | None |
| [ThreatMiner](https://www.threatminer.org) | `threat_intel` | Malware hashes, passive DNS, SSL certs (optional) | None |

### Data fields

| Field | Description |
|-------|-------------|
| `domain` | Normalized input domain |
| `whois` | Raw WHOIS output |
| `rdap` | Structured WHOIS — use `rdap.events` for registration/expiry dates, `rdap.nameservers` for NS |
| `dns.A` / `dns.AAAA` | IPv4 / IPv6 addresses |
| `dns.MX` | Mail server records |
| `dns.TXT` | SPF, DMARC, Google/Microsoft verification tokens |
| `dns.NS` | Nameservers |
| `subdomains` | Deduplicated list from all 4 subdomain sources |
| `emails_from_certs` | Email addresses embedded in certificate fields |
| `certificates_raw` | Raw crt.sh entries (issuer, validity, SAN fields) |
| `passive_dns.forward` | Historical A/CNAME records — tracks IP changes over time |
| `shodan.ports` | Open TCP ports detected on the domain's IP |
| `shodan.vulns` | Known CVE IDs associated with running services |
| `shodan.cpes` | CPE identifiers (tech fingerprint, e.g. `cpe:/a:nginx:nginx:1.24`) |
| `http_headers` | Raw HTTP response headers including security headers |
| `ip` | Resolved IPv4 address |
| `ip_info` | Country, city, ISP, ASN of the resolved IP |
| `threat_intel` | ThreatMiner: malware hashes, passive DNS, SSL certs (if enabled) |

### Pricing

- **$0.05** per run start
- **$0.05** per domain analyzed

Analyzing 100 domains costs approximately **$5.05**.

### FAQ & Disclaimers

**Is this legal?** Yes — all sources are public. WHOIS and DNS are public infrastructure, crt.sh logs are mandated by CA/Browser Forum, and all other APIs are free public services.

**Why are some fields empty?** crt.sh can be slow or temporarily unavailable. New domains with no SSL history return no subdomains from crt.sh. ThreatMiner has limited data for small/new domains. Empty is always better than an error.

**Why is IP geolocation showing a CDN location instead of the real server?** Domains behind Cloudflare, Akamai, or Fastly resolve to CDN edge IPs — the geo data reflects the CDN's nearest PoP, not the origin server. Check `shodan.cpes` for `cpe:/a:cloudflare:cloudflare` to detect this.

**What is `includeThreatIntel` and why is it off by default?** ThreatMiner enforces a 10 req/min rate limit, so enabling it adds ~30 seconds per domain. Turn it on when you specifically need malware association data.

**Support:** Open an issue in the Issues tab.

# Actor input Schema

## `domains` (type: `array`):

Domains to investigate. Full URLs also accepted (e.g. https://example.com/page).

## `includeWhois` (type: `boolean`):

Fetch raw WHOIS registration data

## `includeRdap` (type: `boolean`):

Fetch structured JSON WHOIS via RDAP — registrar, dates, nameservers, status flags

## `includeDns` (type: `boolean`):

Resolve A, AAAA, MX, NS, TXT, CNAME, SOA records

## `includeSubdomains` (type: `boolean`):

Enumerate subdomains via SSL Certificate Transparency (crt.sh) and THC's 6B+ domain database

## `includeWayback` (type: `boolean`):

Discover subdomains from Wayback Machine's CDX index of historical URLs

## `includePassiveDns` (type: `boolean`):

Historical IP resolutions for this domain — tracks hosting changes over time

## `includeShodanInternetdb` (type: `boolean`):

Fetch open ports, known CVEs, CPEs, and service tags for the domain's IP via Shodan InternetDB — no API key required

## `includeHttpHeaders` (type: `boolean`):

Fetch HTTP response headers via HackerTarget — reveals Server, X-Powered-By, HSTS, CSP, X-Frame-Options

## `includeIpInfo` (type: `boolean`):

Resolve domain IP and fetch country, city, ISP, and ASN

## `includeThreatIntel` (type: `boolean`):

Fetch malware associations, passive DNS history, and SSL certs from ThreatMiner. Note: adds ~30s per domain due to rate limiting (10 req/min).

## Actor input object example

```json
{
  "domains": [
    "example.com"
  ],
  "includeWhois": true,
  "includeRdap": true,
  "includeDns": true,
  "includeSubdomains": true,
  "includeWayback": true,
  "includePassiveDns": true,
  "includeShodanInternetdb": true,
  "includeHttpHeaders": true,
  "includeIpInfo": true,
  "includeThreatIntel": false
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "domains": [
        "example.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("romy/domain-intelligence-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "domains": ["example.com"] }

# Run the Actor and wait for it to finish
run = client.actor("romy/domain-intelligence-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "domains": [
    "example.com"
  ]
}' |
apify call romy/domain-intelligence-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,romy/domain-intelligence-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/RnOmJn2hGIEfb5Rtx/builds/Uk8bT77vyqd9PNGU7/openapi.json