Domain Intelligence Scraper

What does Domain Intelligence Scraper do?

Domain Intelligence Scraper audits any list of domains in a single run. It checks DNS records, WHOIS, SSL/TLS certificates, HTTP/HTTPS reachability, email authentication (SPF, DMARC, DKIM, DNSSEC) and domain blacklists. Each result rolls up into a 0-100 security score with actionable flags. Paste bare domains or full URLs — they're auto-cleaned to the apex — and pick which checks to run. Turn on monitoring to track certificate expiry, WHOIS renewal, DNS changes and blacklist appearances over time.

How to use this actor

• 👉 Register for a free Apify account — no credit card required.
• 🎉 Just click Sign up free on Apify → and complete a quick signup.
• 💰 A free Apify account includes $5 in monthly credits — enough to test this actor.
• ⏳ Scrape during the free trial, with no commitment or upfront payment required.

Key features

• 🛡️ 0-100 security score — every domain gets a single at-a-glance posture score plus actionable flags (SSL_EXPIRING_SOON, NO_DMARC, BLACKLISTED, DOMAIN_EXPIRING_SOON) so you can triage a list fast.
• 📧 Email authentication grade (A–F) — SPF, DMARC, DKIM and DNSSEC parsed into pass/warn/fail verdicts and a deliverability grade — the inbox-readiness signal cold-email, RevOps and security teams actually need.
• 🔒 SSL/TLS certificate audit — issuer, validity window, days-to-expiry, chain validity, protocol and SANs, with expired / expiring-soon / invalid flags.
• 🌐 Full DNS + WHOIS — A, AAAA, MX, TXT, CNAME, NS, SOA and CAA records plus DNSSEC, alongside registrar, creation / expiry dates and nameservers.
• 🚫 Domain blacklist check — checks each domain's resolved IP and name against major DNSBLs so you catch reputation problems early.
• ♻️ Change monitoring — incremental mode flags NEW / UPDATED domains and detects certificate renewals, WHOIS expiry, DNS drift and blacklist appearances since the previous run.
• 🔔 Notifications — push new findings to Telegram, Discord, Slack, WhatsApp or a generic webhook (n8n / Make / Zapier).
• 📦 Compact + MCP output — a compact at-a-glance mode and MCP-connector export for AI-agent and automation workflows.

What data can you get for each domain?

Each result includes Core domain fields (domain, inputUrl, securityScore, flags, dns, whois, ssl, and http, and more) and contact information (emailGrade and emailAuth). In standard mode, all fields are always present — unavailable data points are returned as null, never omitted. In compact mode, only core fields are returned.

Input

Configure the actor through the input schema in Apify Console.

Key parameters:

• domains — Domains or full URLs to inspect — one per line, or paste a JSON array. Full URLs are auto-cleaned to the apex domain (https://www.example.com/path → example.com).
• checks — Which inspections to run per domain. Leave all selected for a full audit; deselect to run faster / cheaper on just what you need. (default: ["dns","whois","ssl","http","emailAuth","blacklist"])
• recordTypes — Which DNS record types to resolve when the DNS check is enabled. (default: ["A","AAAA","MX","TXT","CNAME","NS","SOA","CAA"])
• dkimSelectors — DKIM cannot be enumerated, so common selectors (default, google, selector1/2, k1…) are probed automatically. Add your ESP's selector here to catch custom setups. (default: [])
• compact — At-a-glance verdict only (domain, security score, email grade, flags) — drops the raw record blocks. Ideal for AI-agent / MCP workflows. (default: false)
• excludeEmptyFields — Drop null, empty-string, and empty-array fields from each record before push. Smaller payloads for AI agents and dashboards. (default: false)
• incrementalMode — Compare each domain against the previous run and flag changes (NEW / UPDATED / UNCHANGED). Detects cert renewals, WHOIS expiry, DNS changes, email-auth drift and blacklist appearances over time. stateKey is optional — defaults to a stable key derived from the domain list + checks. (default: false)
• stateKey — Optional. Stable identifier for the monitored domain set. Leave empty to auto-generate from the domain list + checks.
• emitUnchanged — When monitoring, also output domains whose signals did not change since the last run. Off = only NEW and UPDATED domains are emitted. (default: false)
• telegramToken — Telegram bot token (from @BotFather). Required for Telegram notifications.
• telegramChatId — Telegram chat or channel ID (e.g. "-100123456789"). Required when telegramToken is set.
• ...and 12 more parameters

Input examples

Basic search — Pick one or more checks from the supported taxonomy.

→ Full payload per result — all standard fields populated where the source provides them.

{
  "checks": [
    "dns"
  ]
}

Incremental tracking — Only emit domains that changed since the previous run with this stateKey.

→ First run builds the baseline state. Subsequent runs emit only records that are new or whose tracked content changed. Set emitUnchanged: true to include unchanged records as well.

{
  "checks": [
    "dns"
  ],
  "incrementalMode": true,
  "stateKey": "dns-tracker"
}

Compact output for AI agents — Return only core fields for AI-agent and MCP workflows.

→ Small payload with the most important fields — ideal for piping into LLMs without token overhead.

{
  "checks": [
    "dns"
  ],
  "compact": true
}

Output

Each run produces a dataset of structured domain records. Results can be downloaded as JSON, CSV, or Excel from the Dataset tab in Apify Console.

Example domain record

{
  "domain": "github.com",
  "inputUrl": "github.com",
  "securityScore": 96,
  "emailGrade": "A",
  "dns": {
    "records": {
      "A": [
        "140.82.121.4"
      ],
      "NS": [
        "dns4.p08.nsone.net",
        "ns-421.awsdns-52.com",
        "ns-520.awsdns-01.net",
        "ns-1283.awsdns-32.org",
        "ns-1707.awsdns-21.co.uk",
        "... 3 more items"
      ],
      "SOA": [
        {
          "nsname": "dns1.p08.nsone.net",
          "hostmaster": "hostmaster.nsone.net",
          "serial": 1656468023,
          "refresh": 43200,
          "retry": 7200,
          "expire": 1209600,
          "minttl": 3600
        }
      ],
      "MX": [
        {
          "priority": 0,
          "exchange": "github-com.mail.protection.outlook.com"
        }
      ],
      "CAA": [
        {
          "critical": 0,
          "issue": "digicert.com"
        },
        {
          "critical": 0,
          "issue": "globalsign.com"
        },
        {
          "critical": 0,
          "issue": "letsencrypt.org"
        },
        {
          "critical": 0,
          "issue": "sectigo.com"
        },
        {
          "critical": 0,
          "issuewild": "digicert.com"
        },
        "... 2 more items"
      ],
      "TXT": [
        "shopify-verification-code=t1YPwcmvnxZyBycaCpk1MPyWoFs72o",
        "krisp-domain-verification=ZlyiK7XLhnaoUQb2hpak1PLY7dFkl1WE",
        "miro-verification=d2e174fdb00c71e0bcf58f8e58c3da2dd80dcfa9",
        "facebook-domain-verification=39xu4jzl7roi7x0n93ldkxjiaarx50",
        "serval-domain-verification-ydryhj=qbkiEakpwEpTvHh5fIiCqtaue",
        "... 17 more items"
      ]
    },
    "dnssec": false,
    "errors": {
      "AAAA": "ENODATA",
      "CNAME": "ENODATA"
    }
  },
  "whois": {
    "raw": "Domain Name: github.com\nRegistry Domain ID: 1264983250_DOMAIN_COM-VRSN\nRegistrar WHOIS Server: whois.markmonitor.com\nRegistrar URL: http://www.markmonitor.com\nUpdated Date: 2024-09-07T09:16:33+0000\nCr...",
    "server": "whois.markmonitor.com",
    "registrar": "MarkMonitor, Inc.",
    "createdDate": "2007-10-09T18:20:50+0000",
    "updatedDate": "2024-09-07T09:16:33+0000",
    "expiryDate": "2026-10-09T00:00:00+0000",
    "nameServers": [
      "dns3.p08.nsone.net",
      "ns-520.awsdns-01.net",
      "ns-1707.awsdns-21.co.uk",
      "ns-1283.awsdns-32.org",
      "dns4.p08.nsone.net",
      "... 3 more items"
    ],
    "status": [
      "clientUpdateProhibited (https://www.icann.org/epp#clientUpdateProhibited)",
      "clientTransferProhibited (https://www.icann.org/epp#clientTransferProhibited)",
      "clientDeleteProhibited (https://www.icann.org/epp#clientDeleteProhibited)"
    ],
    "registrantEmails": [
      "whoisrequest@markmonitor.com"
    ],
    "daysToExpiry": 83
  },
  "ssl": {
    "ok": true,
    "subject": "github.com",
    "issuer": "Sectigo Limited",
    "altNames": [
      "github.com",
      "www.github.com"
    ],
    "validFrom": "Jul  3 00:00:00 2026 GMT",
    "validTo": "Sep 30 23:59:59 2026 GMT",
    "daysToExpiry": 75,
    "expired": false,
    "certSha256": "17:F8:FD:2E:3F:D2:C1:13:FC:B9:77:2D:8A:4B:AB:B8:52:2D:D0:6D:D0:79:49:15:A4:FF:98:B1:B6:86:3A:00",
    "protocol": "TLSv1.3"
  },
  "http": {
    "http": {
      "status": 200,
      "finalUrl": "https://github.com/",
      "redirected": true,
      "server": "github.com"
    },
    "https": {
      "status": 200,
      "finalUrl": "https://github.com/",
      "redirected": false,
      "server": "github.com"
    },
    "upgradesToHttps": true
  },
  "emailAuth": {
    "hasMx": true,
    "spf": {
      "found": true,
      "record": "v=spf1 ip4:192.30.252.0/22 include:spf.protection.outlook.com include:_netblocks.google.com include:_netblocks2.google.com include:mail.zendesk.com include:_spf.salesforce.com include:servers.mcsv.net...",
      "all": "~all",
      "includes": [
        "spf.protection.outlook.com",
        "_netblocks.google.com",
        "_netblocks2.google.com",
        "mail.zendesk.com",
        "_spf.salesforce.com",
        "... 3 more items"
      ],
      "verdict": "pass"
    },
    "dmarc": {
      "found": true,
      "record": "v=DMARC1; p=quarantine; sp=reject; pct=100; rua=mailto:dmarc@github.com; ruf=mailto:dmarc@github.com; fo=1",
      "policy": "quarantine",
      "pct": 100,
      "rua": [
        "mailto:dmarc@github.com"
      ],
      "verdict": "pass"
    },
    "dkim": {
      "selectorsChecked": [
        "default",
        "google",
        "selector1",
        "selector2",
        "k1",
        "... 8 more items"
      ],
      "found": [
        "google",
        "selector1",
        "k2"
      ],
      "verdict": "pass"
    },
    "dnssec": false,
    "score": 90,
    "grade": "A"
  },
  "blacklist": {
    "ip": "140.82.121.4",
    "checked": [
      "zen.spamhaus.org",
      "b.barracudacentral.org",
      "dnsbl.sorbs.net",
      "dbl.spamhaus.org"
    ],
    "note": "best-effort; cloud resolvers may be refused by some zones"
  },
  "checkedAt": "2026-07-18T10:38:24.412Z",
  "source": "domain-intelligence"
}

Incremental fields

When incremental mode is on, each record also carries:

• changeType — one of NEW, UPDATED, UNCHANGED, REAPPEARED, EXPIRED. Default output covers NEW / UPDATED / REAPPEARED; set emitUnchanged: true to opt into the others.

How to audit domains

1. Go to Domain Intelligence Scraper in Apify Console.
2. Configure the input.
3. Set maxResults to control how many results you need.
4. Click Start and wait for the run to finish.
5. Export the dataset as JSON, CSV, or Excel.

Use cases

• Audit a portfolio of domains for SSL certificates and domain registrations that are expiring soon, before they lapse.
• Score sending domains for email deliverability (SPF / DMARC / DKIM / DNSSEC) before a cold-email or newsletter campaign.
• Monitor DNS, WHOIS and certificate changes across your domains and get alerted the moment something drifts.
• Run a security-posture review across many domains — email authentication, certificate health and blacklist status in one pass.
• Enrich a list of company domains with DNS, WHOIS and infrastructure signals for lead qualification and due diligence.

How much does it cost to audit domains?

Domain Intelligence Scraper uses pay-per-event pricing. You pay a small fee when the run starts and then for each result that is actually produced.

• Run start: $0.00005 per run
• Per result: $0.001 per domain record

Example costs:

• 10 results: $0.01
• 25 results: $0.025
• 100 results: $0.1
• 200 results: $0.2
• 500 results: $0.5

Example: recurring monitoring savings

These examples compare full re-scrapes with incremental runs at different churn rates. Churn is the share of domains that are new or whose tracked content changed since the previous run. Actual churn depends on your query breadth, source activity, and polling frequency — the scenarios below are examples, not predictions.

Example setup: 250 results per run, daily polling (30 runs/month). Event-pricing examples scale linearly with result count.

Churn rate	Full re-scrape run cost	Incremental run cost	Savings vs full re-scrape	Monthly cost after baseline
5% — stable niche query	$0.25	$0.01	$0.24 (95%)	$0.38
15% — moderate broad query	$0.25	$0.04	$0.21 (85%)	$1.13
30% — high-volume aggregator	$0.25	$0.08	$0.17 (70%)	$2.25

Full re-scrape monthly cost at daily polling: $7.50. First month with incremental costs $0.61 / $1.34 / $2.43 for the 5% / 15% / 30% scenarios because the first run builds baseline state at full cost before incremental savings apply.

Platform usage is included in the per-result fee shown above.

FAQ

How many domains can I check per run?

Every domain in your input list is checked — there is no fixed cap, so the number of domains per run is limited only by your list size and the run's time budget.

Does Domain Intelligence Scraper support recurring monitoring?

Yes. Enable incremental mode to only receive new or changed domains on subsequent runs. This is ideal for scheduled monitoring where you want to track changes over time without re-processing the full dataset.

Can I integrate Domain Intelligence Scraper with other apps?

Yes. Domain Intelligence Scraper works with Apify's integrations to connect with tools like Zapier, Make, Google Sheets, Slack, and more. You can also use webhooks to trigger actions when a run completes.

Can I use Domain Intelligence Scraper with the Apify API?

Yes. You can start runs, manage inputs, and retrieve results programmatically through the Apify API. Client libraries are available for JavaScript, Python, and other languages.

Can I use Domain Intelligence Scraper through an MCP Server?

Yes. Apify provides an MCP Server that lets AI assistants and agents call this actor directly. Use compact mode and excludeEmptyFields to keep payloads manageable for LLM context windows.

Is it legal to audit domains?

This actor queries publicly available DNS, WHOIS, certificate and reputation data for the domains you provide. Looking up public domain data is generally legal, but you should ensure your use complies with applicable laws and the queried services' terms, including GDPR where relevant.

Your feedback

If you have questions, need a feature, or found a bug, please open an issue on the actor's page in Apify Console. Your feedback helps us improve.

You might also like

• Email Deliverability Checker — SPF, DMARC, DKIM and DNSSEC audit with an A-F inbox-readiness grade for sending domains.
• SSL Certificate Monitor — Bulk SSL/TLS certificate check: issuer, expiry, chain validity and SANs with expiring-soon alerts.
• DNS Lookup — Bulk DNS record lookup: A, AAAA, MX, TXT, CNAME, NS, SOA, CAA and DNSSEC per domain.

Getting started with Apify

New to Apify? Create a free account with $5 credit — no credit card required.

1. Sign up — $5 platform credit included
2. Open this actor and configure your input
3. Click Start — export results as JSON, CSV, or Excel

Need more later? See Apify pricing.