# Domain Intelligence: WHOIS/RDAP, DNS, SSL, Email & Tech (`alain_diaz/domain-intelligence-scraper`) Actor

Bulk domain lookup from free public sources: RDAP/WHOIS (registrar, dates, nameservers), DNS records, SPF/DMARC email security, TLS certificate, HTTP security headers and technology hints. Toggle each module.

- **URL**: https://apify.com/alain_diaz/domain-intelligence-scraper.md
- **Developed by:** [José Alaín Díaz Carrero](https://apify.com/alain_diaz) (community)
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $5.00 / 1,000 domain analyzeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Domain Intelligence: WHOIS/RDAP, DNS, SSL, Email & Tech

Analyze a list of domains in bulk and get one clean JSON record per domain: **registration data (WHOIS/RDAP)**, **DNS records**, **email security (SPF, DMARC, DKIM, MTA-STS)**, the **TLS/SSL certificate**, **HTTP security headers** and **technology hints**. Every module can be switched on or off, so you only wait for (and pay for) what you need.

Only **free public sources** are used: the official RDAP servers (the modern, JSON-based replacement for WHOIS), public DNS-over-HTTPS resolvers, a TLS handshake with the domain on port 443 and a single HTTP request to its home page. No API keys, no logins, no proxies, no scraping of search engines.

### What you can do with it

- **Security & IT:** audit SPF/DMARC across hundreds of client or portfolio domains, find certificates about to expire, spot domains without HSTS/CSP.
- **Sales & lead enrichment:** see which CMS, e-commerce platform, CDN, analytics, chat widget and mail provider (Google Workspace, Microsoft 365...) a prospect uses.
- **Domain investors & brand protection:** registrar, creation date, expiry date and status codes for a watch list; detect unregistered names.
- **SEO & agencies:** redirect target, status code, title and meta description, server and hosting hints for a batch of sites.
- **Due diligence:** domain age, nameservers, DNSSEC and certificate details.

### Input

| Field | Type | Default | Description |
|---|---|---|---|
| `domains` | array of strings | required | Domains or URLs, e.g. `example.com` or `https://www.apify.com/store`. Protocol, path, port and credentials are removed, IDNs are converted to punycode, duplicates are dropped. |
| `includeRdap` | boolean | `true` | WHOIS/RDAP: registrar, dates, domain age, status, nameservers, DNSSEC |
| `includeDns` | boolean | `true` | A, AAAA, CNAME, NS, MX, TXT, CAA |
| `includeEmailSecurity` | boolean | `true` | SPF, DMARC, MTA-STS, TLS-RPT, mail provider, findings |
| `checkDkim` | boolean | `false` | Probe common DKIM selectors (adds a few DNS queries per domain) |
| `dkimSelectors` | array | built-in list | Your own DKIM selectors to probe |
| `includeTls` | boolean | `true` | TLS certificate: issuer, validity, SANs, protocol, key |
| `includeHttp` | boolean | `true` | Final URL, status, redirects, server, title, security headers |
| `includeTech` | boolean | `true` | Technology hints from headers, HTML, DNS and email provider |
| `timeoutSecs` | integer | `10` | Timeout for each network lookup (3-60) |
| `maxConcurrency` | integer | `5` | Domains analyzed in parallel (1-20) |

#### Example inputs

**1. Everything for a few domains (minimal)**

```json
{ "domains": ["example.com", "apify.com", "wikipedia.org"] }
```

**2. Email-security audit only (fast and cheap)**

```json
{
  "domains": ["client1.com", "client2.io", "client3.org"],
  "includeRdap": false,
  "includeTls": false,
  "includeHttp": false,
  "includeTech": false,
  "includeDns": false,
  "includeEmailSecurity": true,
  "checkDkim": true,
  "dkimSelectors": ["default", "google", "selector1", "selector2", "mailchimp"]
}
```

**3. Certificate and registration expiry watch**

```json
{
  "domains": ["mycompany.com", "https://shop.mycompany.com/cart"],
  "includeRdap": true,
  "includeTls": true,
  "includeDns": false,
  "includeEmailSecurity": false,
  "includeHttp": false,
  "includeTech": false
}
```

Schedule it daily or weekly and filter on `daysToExpiry` and `tls.daysRemaining` to get expiry alerts through Apify integrations (Slack, email, webhooks, Google Sheets).

### Output

One dataset item per domain. Fields appear only when the module is enabled. Example (shortened; the full record is in `sample_output.json`):

```json
{
  "domain": "apify.com",
  "input": "apify.com",
  "status": "ok",
  "registered": true,
  "registrar": "Amazon Registrar, Inc.",
  "createdAt": "2009-06-02T17:14:10Z",
  "expiresAt": "2035-06-02T17:14:10Z",
  "daysToExpiry": 3141,
  "nameservers": ["ns-1225.awsdns-25.org", "..."],
  "dns": { "a": ["..."], "mx": [{ "priority": 1, "exchange": "aspmx.l.google.com" }], "txt": ["..."], "nxdomain": false },
  "emailSecurity": {
    "spf": { "record": "v=spf1 ...", "policy": "softfail" },
    "dmarc": { "policy": "reject", "rua": ["mailto:..."] },
    "mailProvider": "Google Workspace",
    "findings": []
  },
  "tls": { "issuerOrganization": "Amazon", "validTo": "2027-01-16T23:59:59.000Z", "daysRemaining": 82, "sans": ["apify.com", "..."], "protocol": "TLSv1.3" },
  "http": { "finalUrl": "https://apify.com/", "statusCode": 200, "title": "...", "securityHeadersScore": 5 },
  "technologies": [{ "name": "Amazon CloudFront", "category": "CDN" }, { "name": "Next.js", "category": "JavaScript framework" }],
  "notes": [],
  "errors": {}
}
```

#### Field reference

| Group | Fields |
|---|---|
| Core | `domain` (normalized, ASCII), `input` (as given), `unicodeDomain` (IDNs only), `scrapedAt`, `status` (`ok`, `partial`, `failed`) |
| Registration (top level) | `registered`, `registeredDomain`, `registrar`, `createdAt`, `expiresAt`, `daysToExpiry`, `nameservers` |
| `rdap` | `registrarIanaId`, `updatedAt`, `domainAgeDays`, `status` (EPP codes), `dnssecSigned`, `rdapServer` |
| `dns` | `a`, `aaaa`, `cname`, `ns`, `mx` (priority + exchange), `txt`, `caa`, `nxdomain` |
| `emailSecurity` | `spf` (`record`, `policy`, `includes`, `lookupMechanisms`, `multipleRecords`), `dmarc` (`policy`, `subdomainPolicy`, `pct`, `rua`, `ruf`, `adkim`, `aspf`), `dkim` (`selectorsFound`, `wildcardRecord`), `mtaSts`, `tlsRpt`, `mailProvider`, `findings` |
| `tls` | `subjectCommonName`, `issuerOrganization`, `issuerCommonName`, `validFrom`, `validTo`, `daysRemaining`, `sans`, `protocol`, `cipher`, `authorized`, `authorizationError`, `serialNumber`, `fingerprint256`, `keyType`, `keyBits` |
| `http` | `requestedUrl`, `finalUrl`, `statusCode`, `redirects`, `httpsRedirect`, `server`, `poweredBy`, `contentType`, `title`, `metaDescription`, `securityHeaders` (HSTS, CSP, X-Frame-Options, X-Content-Type-Options, Referrer-Policy, Permissions-Policy), `securityHeadersScore` (0-6) |
| `technologies` | list of `{ name, category }` |
| Diagnostics | `notes` (informational, e.g. "domain does not resolve"), `errors` (per-module error messages) |

The dataset has ready-made views: Overview, Registration, Email security, TLS & HTTP and Technologies.

### Error handling

Each module runs independently with its own timeout. If one fails (for example a site that blocks the HTTP request), the others still return data and the failure is listed in `errors`, with `status: "partial"`. A domain that does not exist returns `registered: false`, `dns.nxdomain: true` and a note; TLS and HTTP are skipped for domains without an A/AAAA/CNAME record. If **every** enabled module fails (for example a network problem), the item is stored with `status: "failed"` and is **not charged**.

### Pricing (pay per event)

The Actor uses pay-per-event pricing: a small platform start fee per run plus a charge per domain analyzed (`domain-result`). Failed domains are not charged and the run stops early if your maximum run cost is reached. See the pricing section on this page for the current price. Compute use is very low, 256 MB is enough.

### FAQ

**Is this legal / allowed?** Yes for its intended use: it only reads data that registries, DNS servers and websites make publicly available (RDAP, DNS, a standard TLS handshake and one request to the home page). See the legal note below.

**Why RDAP and not classic WHOIS?** RDAP is the official, structured successor of WHOIS, free and JSON-based. Registrant personal data is usually redacted for privacy (GDPR), so this Actor does not return registrant names, emails or phone numbers.

**A country-code domain returned `errors.rdap`.** Some TLDs (for example `.de` and `.io`) do not publish an RDAP service in the IANA registry yet. Other modules still work and `status` is `partial`.

**Does it work with subdomains?** Yes. `blog.example.com` gets its own DNS, TLS and HTTP data, and RDAP falls back to the parent registered domain.

**How accurate is technology detection?** It is based on response headers, the HTML of the home page and DNS/email providers. It finds the common CMS, frameworks, CDN, analytics and chat tools but is not a complete fingerprint of a site's stack. Sites protected by bot-blocking may return limited data.

**What does DKIM probing do?** DKIM selectors are not listed in DNS, so the Actor queries `<selector>._domainkey.<domain>` for a list of common selectors (or yours). A selector that is not found does not mean DKIM is absent. If the domain has a wildcard DKIM record, probing is skipped and `wildcardRecord` is `true`.

**Do I need proxies?** No. DNS goes through public DNS-over-HTTPS (Cloudflare, then Google) with the system resolver as fallback.

**Why is TLS or HTTP skipped?** When a domain has no A/AAAA/CNAME record there is nothing to connect to, so those checks are skipped and a note explains it.

**Can I export to CSV, Excel or Google Sheets?** Yes. Download JSON, CSV, Excel, XML or HTML, or use Apify integrations (Slack, email, webhooks, Google Sheets).

**How many domains can I run at once?** Thousands per run. Keep `maxConcurrency` at 5-10 for very large lists to stay polite to public RDAP servers; the Actor retries once on HTTP 429.

### Legal note

This Actor only uses publicly available information from RDAP registry servers, public DNS resolvers and the domain's own public web server (a TLS handshake and a request to the home page, like any browser). It does not log in, bypass access controls, collect registrant personal data or scan ports. Use it responsibly and only for lawful purposes, comply with applicable laws (including data protection rules) and with the terms of the services you query. This Actor is independent and not affiliated with or endorsed by any registry, registrar, DNS provider or technology vendor named in its output.

### Notes

- Only public, no-authentication sources are used.
- Home page body reads are capped at 300 KB.
- Results reflect the moment of the run; DNS answers from public resolvers may be cached for their TTL.

# Actor input Schema

## `domains` (type: `array`):

Domains or URLs to analyze, e.g. `example.com` or `https://www.apify.com/store`. Protocol, path, port and credentials are stripped; internationalized domains are converted to punycode; duplicates are removed. Subdomains are accepted (RDAP then reports the parent registered domain).

## `includeRdap` (type: `boolean`):

Registrar, creation / update / expiry dates, domain age, status codes, nameservers and DNSSEC from the public RDAP registry servers.

## `includeDns` (type: `boolean`):

A, AAAA, CNAME, NS, MX, TXT and CAA records (resolved over DNS-over-HTTPS).

## `includeEmailSecurity` (type: `boolean`):

Parses SPF and DMARC, checks MTA-STS and TLS-RPT, detects the mail provider and lists findings such as a missing DMARC record or `p=none`.

## `checkDkim` (type: `boolean`):

Query `<selector>._domainkey.<domain>` for a list of common selectors. DKIM selectors cannot be listed in DNS, so this only finds the selectors you test. Adds a few DNS queries per domain.

## `dkimSelectors` (type: `array`):

Custom list of DKIM selectors. Leave empty to use the built-in list (default, google, selector1, selector2, k1, s1, ...). Only used when "Probe common DKIM selectors" is on.

## `includeTls` (type: `boolean`):

Connects to port 443 and reads the certificate: issuer, validity dates, days remaining, SANs, key type and size, TLS protocol and whether the chain is trusted.

## `includeHttp` (type: `boolean`):

Requests the home page (HTTPS first, then HTTP) and follows up to 6 redirects. Returns final URL, status code, server header, page title, meta description and the presence of HSTS, CSP, X-Frame-Options, X-Content-Type-Options, Referrer-Policy and Permissions-Policy.

## `includeTech` (type: `boolean`):

Detects common CMS, frameworks, CDN / hosting, web servers, analytics and chat widgets from headers and the HTML of the home page, plus DNS and email providers. These are hints, not a guaranteed full technology stack.

## `timeoutSecs` (type: `integer`):

Network timeout applied to each lookup (RDAP, DNS, TLS, HTTP).

## `maxConcurrency` (type: `integer`):

How many domains are analyzed in parallel. Lower it if you hit RDAP rate limits on very large lists.

## Actor input object example

```json
{
  "domains": [
    "example.com",
    "apify.com",
    "wikipedia.org"
  ],
  "includeRdap": true,
  "includeDns": true,
  "includeEmailSecurity": true,
  "checkDkim": false,
  "includeTls": true,
  "includeHttp": true,
  "includeTech": true,
  "timeoutSecs": 10,
  "maxConcurrency": 5
}
```

# Actor output Schema

## `results` (type: `string`):

Dataset items with one record per analyzed domain.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "domains": [
        "example.com",
        "apify.com",
        "wikipedia.org"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("alain_diaz/domain-intelligence-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "domains": [
        "example.com",
        "apify.com",
        "wikipedia.org",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("alain_diaz/domain-intelligence-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "domains": [
    "example.com",
    "apify.com",
    "wikipedia.org"
  ]
}' |
apify call alain_diaz/domain-intelligence-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,alain_diaz/domain-intelligence-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/fuz6BiJNyfdCG0AFR/builds/fz9HgKp41StCRsxpW/openapi.json