DNS Records Scraper: Bulk Lookups over DoH

DNS Records Scraper resolves A, AAAA, MX, TXT, NS, CNAME, SOA and CAA records for a list of domains over DNS-over-HTTPS, and decodes SPF and DMARC into readable email-security fields. Numeric response codes and record types come back as names, NXDOMAIN, not 3; MX, not 15.

Google and Cloudflare both publish DNS-over-HTTPS for programmatic use, so there is nothing to configure at your end, and no local resolver to set up either.

Agent skill: SKILL.md

https://api.apify.com/v2/key-value-stores/t7YoTxpZEJOWvw4Ug/records/dns-records-scraper.md

What you get

One dataset item per domain × record type.

Output field	Meaning
domain	The normalised hostname that was queried
recordType	The record type for this lookup, A, MX, TXT …
status, statusCode	Decoded DNS response code (NOERROR, NXDOMAIN, SERVFAIL …) and the raw integer
value	The records of the type you asked for, as a flat array of strings, the quick field to eyeball
records	The whole answer set structured: { name, type, ttl, data }, including the CNAME chain that led to the record
ttl	Shortest TTL among the records of the requested type, in seconds; null when there are none
resolver, failedOver	Which resolver actually answered, and whether it was the fallback
responseTimeMs	Round-trip time for this lookup
hasSpf, spfRecord, spfPolicy	Whether SPF exists, its full record, and its enforcement qualifier (-all hard fail, ~all soft fail). hasSpf is null when the lookup could not be made
hasDmarc, dmarcRecord, dmarcPolicy	Whether DMARC exists, its full record, and the policy (none / quarantine / reject). hasDmarc is null when the lookup could not be made
mxHosts	Mail exchangers as { preference, host }, sorted by preference
nullMx	true when the domain publishes RFC 7505's null MX (0 .), meaning it accepts no mail at all. null when MX was not requested
nameservers	Authoritative nameservers for the domain
scrapedAt	Run timestamp

The email-policy and mxHosts / nullMx / nameservers fields are domain-level, so they are repeated on every record-type item for that domain, one row is enough to answer "is this domain's email locked down?".

value is what you asked for; records is everything that came back. A DNS answer carries the CNAME chain that led to the record, so a query for AAAA on a CNAMEd host answers with the CNAME even when no AAAA record exists. value lists only the records of the requested type, which is what makes an empty value mean "no record of this type". Filter records yourself if you want the chain.

A RUN_SUMMARY record in the key-value store holds per-run counts, rejected inputs, per-lookup failures, email-policy lookups that could not be made, domains the cap left unresolved, and the applied filters.

Common use cases

Audit email security posture at scale, SPF and DMARC for a whole customer list in one run. Sort by dmarcPolicy: everything at none, and everything with hasDmarc: false, is a spoofing risk. hasDmarc: null means that domain was not assessed, not that it is exposed.

{
 "domains": ["apify.com", "stripe.com", "github.com"],
 "recordTypes": ["MX", "TXT"],
 "parseEmailPolicy": true,
 "maxLookups": 100
}

Verify domain configuration before migration, capture the full picture before you cut over, then run it again after and diff.

{
 "domains": ["example.com", "www.example.com", "api.example.com"],
 "recordTypes": ["A", "AAAA", "CNAME", "NS", "SOA", "CAA"],
 "resolver": "cloudflare"
}

Enrich leads with mail-provider detection, mxHosts tells you instantly who runs a prospect's mail: aspmx.l.google.com is Google Workspace, *.mail.protection.outlook.com is Microsoft 365.

{
 "domains": ["stripe.com", "shopify.com", "notion.so"],
 "recordTypes": ["MX"],
 "parseEmailPolicy": false
}

Quick start

Everything about two domains:

{
 "domains": ["apify.com", "github.com"]
}

Email audit, Cloudflare as primary resolver:

{
 "domains": ["apify.com", "stripe.com"],
 "recordTypes": ["MX", "TXT"],
 "resolver": "cloudflare",
 "parseEmailPolicy": true
}

Input

Field	Type	Default	Notes
domains	array	-	Required. Hostnames. Schemes, credentials, ports, paths, trailing dots and email-style inputs are all stripped, and the result is lower-cased and de-duplicated. A www. prefix is kept: in DNS www.example.com is a different name from example.com, with its own records, so it is resolved and charged as its own domain.
recordTypes	array	["A","AAAA","MX","TXT","NS"]	Any of A, AAAA, MX, TXT, NS, CNAME, SOA, CAA. One lookup and one dataset item each.
resolver	string	google	google or cloudflare. The other is used automatically as a fallback.
maxLookups	integer	1000	Ceiling on the dataset items the whole run may produce. 0 means no ceiling.
parseEmailPolicy	boolean	true	Extract SPF from TXT and fetch DMARC from _dmarc.<domain>.

Which combinations make sense. Every domain is queried for every record type, so cost is domains × recordTypes, trimming recordTypes is how you keep a large sweep cheap. parseEmailPolicy adds one DMARC lookup per domain, plus one SPF lookup only when TXT is not already in your list, so pairing it with recordTypes: ["MX","TXT"] is the efficient shape for an email audit. Those extra policy lookups enrich the rows you already receive and are never charged as rows of their own.

maxLookups is a run total, not a per-domain allowance. 40 domains × 5 record types is 200 items; with maxLookups: 50 the run resolves domains in the order you listed them, stops at 50, and names every domain it never reached in RUN_SUMMARY.domainsSkippedByCap. If the ceiling falls part-way through a domain, only the first few of your recordTypes are resolved for it, so mxHosts and nameservers can be empty on that domain's rows; the types it did not reach are named in RUN_SUMMARY.recordTypesSkippedByCap. A value below 0, or one that is not a whole number, ends the run with an error rather than being read as "no ceiling".

CNAME on an apex domain almost always returns NOERROR with zero records, apex CNAMEs are not valid DNS. Query it on sub-domains, where it tells you what a host actually points at.

Output example

{
 "domain": "apify.com",
 "recordType": "MX",
 "status": "NOERROR",
 "statusCode": 0,
 "value": [
 "1 aspmx.l.google.com.",
 "5 alt1.aspmx.l.google.com.",
 "10 aspmx2.googlemail.com."
 ],
 "records": [
 { "name": "apify.com", "type": "MX", "ttl": 86400, "data": "1 aspmx.l.google.com." }
 ],
 "ttl": 86400,
 "resolver": "google",
 "failedOver": false,
 "responseTimeMs": 33,
 "hasSpf": true,
 "spfRecord": "v=spf1 a mx include:_spf.google.com include:mailgun.org -all",
 "spfPolicy": "-all",
 "hasDmarc": true,
 "dmarcRecord": "v=DMARC1; p=reject; sp=reject; pct=100; rua=mailto:dmarc-reports@apify.com; ri=604800",
 "dmarcPolicy": "reject",
 "mxHosts": [
 { "preference": 1, "host": "aspmx.l.google.com" },
 { "preference": 5, "host": "alt1.aspmx.l.google.com" },
 { "preference": 10, "host": "aspmx2.googlemail.com" }
 ],
 "nullMx": false,
 "nameservers": [
 "ns-449.awsdns-56.com",
 "ns-839.awsdns-40.net",
 "ns-1225.awsdns-25.org",
 "ns-1928.awsdns-49.co.uk"
 ],
 "scrapedAt": "2026-08-06T11:43:11.221Z"
}

RUN_SUMMARY looks like this:

{
 "domainsRequested": 2,
 "domainsRejected": ["not a domain!!"],
 "domainsSkippedByCap": [],
 "recordTypesSkippedByCap": [],
 "lookupsRequested": 2,
 "lookupsSkippedByCap": 0,
 "lookupsFailed": 0,
 "failures": [],
 "emailPolicyLookupsFailed": 1,
 "emailPolicyFailures": [
 { "domain": "stripe.com", "lookup": "DMARC", "error": "retryable HTTP 503" }
 ],
 "lookupsSaved": 2,
 "capReached": false,
 "filters": {
 "domains": ["apify.com", "stripe.com"],
 "recordTypes": ["MX"],
 "resolver": "google",
 "parseEmailPolicy": true,
 "maxLookups": 1000
 },
 "finishedAt": "2026-08-06T11:43:48.478Z"
}

Every item lookupsRequested promised and lookupsSaved did not deliver is accounted for by exactly one of three things: failures (the lookup was attempted and did not answer), domainsSkippedByCap (the cap ran out before that domain), or recordTypesSkippedByCap (the cap ran out part-way through it). lookupsSkippedByCap is the total the cap cut. emailPolicyFailures explains every hasSpf or hasDmarc that came back null: those rows are real DNS answers, but the email verdict on them is unknown rather than negative.

Reading the status field

status is the DNS response code, decoded. It is the difference between "this domain has no MX records" and "this domain does not exist".

Status	Means
NOERROR with records	The domain has records of this type
NOERROR with an empty value	The domain exists but has no record of this type, normal for CNAME on an apex, or AAAA on IPv4-only hosts. records may still show the CNAME chain that was followed
NXDOMAIN	The domain does not exist at all
SERVFAIL	The authoritative server broke or DNSSEC validation failed, often a real misconfiguration worth flagging
REFUSED	The authoritative server declined to answer

Limits and behaviour

• maxLookups bounds the whole run. It is the ceiling on delivered items, and therefore on cost. Domains are resolved in the order you listed them and anything the ceiling leaves out is named in RUN_SUMMARY.domainsSkippedByCap and RUN_SUMMARY.recordTypesSkippedByCap, so a missing row never reads as a domain with no records.
• Automatic failover. If your chosen resolver fails after its retries, the other one is tried before the lookup is recorded as failed. Every item names the resolver that actually answered and sets failedOver when it was the backup. Occasional disagreement between the two is normal, they hit different authoritative caches, which is exactly why the fallback is useful.
• Response codes and record types are decoded. DNS-over-HTTPS returns the response code and each answer's type as bare integers. Both are mapped to their IANA names, so answers that contain a CNAME chain on the way to an A record show up as CNAME and A rather than 5 and 1.
• TTL follows the records you asked for. When a response mixes records with different TTLs, the shortest of the requested type is the one that governs when that answer goes stale. A CNAME picked up on the way does not set the TTL of a record that does not exist.
• A null MX is not a mail host. RFC 7505's 0 . means the domain accepts no mail. It is reported as nullMx: true with an empty mxHosts, never as a mail exchanger with a blank hostname.
• Trailing dots are stripped. One resolver echoes fully-qualified names with a root dot and the other does not; mxHosts and nameservers are normalised so results are comparable across resolvers.
• Unparseable input is reported, not silently dropped. Anything that is not a hostname lands in RUN_SUMMARY.domainsRejected; unsupported record types are logged and skipped.
• One failed lookup never kills the run. Failures land in RUN_SUMMARY.failures with the domain and record type; the Actor only throws when every lookup failed.
• A failed SPF or DMARC lookup is a null, never a false. "This domain publishes no DMARC" and "we could not find out" are opposite findings and an audit acts on them differently, so the second one is reported as null and named in RUN_SUMMARY.emailPolicyFailures.
• Passive, public data only. DNS is public infrastructure metadata. Nothing here touches the domains themselves, no port scanning, no connection to the hosts, no probing.

API example

curl -X POST "https://api.apify.com/v2/acts/arman-bd~dns-records-scraper/run-sync-get-dataset-items?token=YOUR_TOKEN" \
 -H "Content-Type: application/json" \
 -d '{
 "domains": ["apify.com", "stripe.com", "github.com"],
 "recordTypes": ["MX", "TXT"],
 "resolver": "google",
 "parseEmailPolicy": true
 }'

JavaScript example

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_TOKEN' });
const run = await client.actor('arman-bd/dns-records-scraper').call({
 domains: ['apify.com', 'stripe.com', 'github.com'],
 recordTypes: ['MX'],
 parseEmailPolicy: true,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const r of items) {
 // hasDmarc is null when the lookup could not be made. Judging that as "no DMARC"
 // would report a protected domain as spoofable, so it gets its own verdict.
 const risk = r.hasDmarc === null ? 'UNKNOWN' : r.dmarcPolicy === 'reject' ? 'ok' : 'SPOOFABLE';
 console.log(`${r.domain}\tSPF ${r.spfPolicy ?? 'none'}\tDMARC ${r.dmarcPolicy ?? 'none'}\t${risk}`);
}

Defaults: 1 GB memory, 15 minute timeout.

FAQ

Do I need a proxy? No. Proxy configuration is not required to run this Actor.

Do I need credentials of my own? No. There is nothing for you to supply beyond the input.

What happens if a source is unavailable? The other resolver is tried automatically. If both fail, that single lookup is recorded in RUN_SUMMARY.failures and the run continues. The Actor only errors out when every lookup fails.

How do I stop a big list from costing more than I expect? Set maxLookups. It caps the items the whole run can deliver, not the items per domain, and whatever it leaves unresolved is listed in RUN_SUMMARY.domainsSkippedByCap and RUN_SUMMARY.recordTypesSkippedByCap.

Can I schedule it? Yes, it is built for it. Run daily over your domain list and diff on value to catch DNS changes, expired records or hijacked sub-domains.

Why do Google and Cloudflare return different IP addresses? Because many domains use geo-aware or load-balanced DNS, so the answer depends on which resolver asked. That is the real state of the world, not an error. Pin resolver if you need runs to be comparable.

Why is hasDmarc false for a domain that clearly sends email? DMARC lives at _dmarc.<domain>, not on the domain itself, and plenty of senders never publish one. That absence is the finding. false always means the lookup ran and found nothing; if the lookup itself could not be made you get null.

What is the difference between -all and ~all in spfPolicy? -all tells receivers to reject mail from unlisted senders; ~all only asks them to mark it suspicious. -all is the stronger posture.

Does it do reverse DNS or zone transfers? No. Forward lookups of specific record types only, no PTR sweeps, no AXFR, nothing intrusive.

Can I integrate it with something else? Yes, Apify API, client libraries, webhooks, scheduled runs, dataset exports (JSON/CSV/Excel) or MCP. Output is structured JSON.