# DNS Records Scraper: Bulk Lookups over DoH (`arman-bd/dns-records-scraper`) Actor Resolve DNS records in bulk over DNS-over-HTTPS: A, AAAA, MX, TXT, NS, CNAME, SOA, CAA and SPF/DMARC. No local resolver to configure, and a run-wide lookup cap you set. - **URL**: https://apify.com/arman-bd/dns-records-scraper.md - **Developed by:** [Arman Hossain](https://apify.com/arman-bd) (community) - **Categories:** Developer tools, SEO tools, MCP servers - **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.56 / 1,000 lookup scrapeds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. - **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use). - **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`. - **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`. - **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`. - **Any other language** — the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## DNS Records Scraper: Bulk Lookups over DoH ![DNS Records Scraper: A, AAAA, MX, TXT, NS, CNAME, SOA and CAA records over DNS-over-HTTPS, with SPF and DMARC decoded into readable fields](https://api.apify.com/v2/key-value-stores/ZQOcNAOHrIgTacAmy/records/dns-records-scraper.jpg) **DNS Records Scraper** resolves A, AAAA, MX, TXT, NS, CNAME, SOA and CAA records for a list of domains over **DNS-over-HTTPS**, and decodes SPF and DMARC into readable email-security fields. Numeric response codes and record types come back as names, `NXDOMAIN`, not `3`; `MX`, not `15`. Google and Cloudflare both publish DNS-over-HTTPS for programmatic use, so there is nothing to configure at your end, and no local resolver to set up either. **Agent skill: [SKILL.md](https://api.apify.com/v2/key-value-stores/t7YoTxpZEJOWvw4Ug/records/dns-records-scraper.md)** ``` https://api.apify.com/v2/key-value-stores/t7YoTxpZEJOWvw4Ug/records/dns-records-scraper.md ``` ### What you get One dataset item per domain × record type. | Output field | Meaning | |---|---| | `domain` | The normalised hostname that was queried | | `recordType` | The record type for this lookup, `A`, `MX`, `TXT` … | | `status`, `statusCode` | Decoded DNS response code (`NOERROR`, `NXDOMAIN`, `SERVFAIL` …) and the raw integer | | `value` | The records of the type you asked for, as a flat array of strings, the quick field to eyeball | | `records` | The **whole** answer set structured: `{ name, type, ttl, data }`, including the CNAME chain that led to the record | | `ttl` | Shortest TTL among the records of the requested type, in seconds; `null` when there are none | | `resolver`, `failedOver` | Which resolver actually answered, and whether it was the fallback | | `responseTimeMs` | Round-trip time for this lookup | | `hasSpf`, `spfRecord`, `spfPolicy` | Whether SPF exists, its full record, and its enforcement qualifier (`-all` hard fail, `~all` soft fail). `hasSpf` is `null` when the lookup could not be made | | `hasDmarc`, `dmarcRecord`, `dmarcPolicy` | Whether DMARC exists, its full record, and the policy (`none` / `quarantine` / `reject`). `hasDmarc` is `null` when the lookup could not be made | | `mxHosts` | Mail exchangers as `{ preference, host }`, sorted by preference | | `nullMx` | `true` when the domain publishes RFC 7505's null MX (`0 .`), meaning it accepts no mail at all. `null` when `MX` was not requested | | `nameservers` | Authoritative nameservers for the domain | | `scrapedAt` | Run timestamp | The email-policy and `mxHosts` / `nullMx` / `nameservers` fields are domain-level, so they are repeated on every record-type item for that domain, one row is enough to answer "is this domain's email locked down?". **`value` is what you asked for; `records` is everything that came back.** A DNS answer carries the CNAME chain that led to the record, so a query for `AAAA` on a CNAMEd host answers with the CNAME even when no `AAAA` record exists. `value` lists only the records of the requested type, which is what makes an empty `value` mean "no record of this type". Filter `records` yourself if you want the chain. A `RUN_SUMMARY` record in the key-value store holds per-run counts, rejected inputs, per-lookup failures, email-policy lookups that could not be made, domains the cap left unresolved, and the applied filters. ### Common use cases **Audit email security posture at scale**, SPF and DMARC for a whole customer list in one run. Sort by `dmarcPolicy`: everything at `none`, and everything with `hasDmarc: false`, is a spoofing risk. `hasDmarc: null` means that domain was not assessed, not that it is exposed. ```json { "domains": ["apify.com", "stripe.com", "github.com"], "recordTypes": ["MX", "TXT"], "parseEmailPolicy": true, "maxLookups": 100 } ``` **Verify domain configuration before migration**, capture the full picture before you cut over, then run it again after and diff. ```json { "domains": ["example.com", "www.example.com", "api.example.com"], "recordTypes": ["A", "AAAA", "CNAME", "NS", "SOA", "CAA"], "resolver": "cloudflare" } ``` **Enrich leads with mail-provider detection**, `mxHosts` tells you instantly who runs a prospect's mail: `aspmx.l.google.com` is Google Workspace, `*.mail.protection.outlook.com` is Microsoft 365. ```json { "domains": ["stripe.com", "shopify.com", "notion.so"], "recordTypes": ["MX"], "parseEmailPolicy": false } ``` ### Quick start Everything about two domains: ```json { "domains": ["apify.com", "github.com"] } ``` Email audit, Cloudflare as primary resolver: ```json { "domains": ["apify.com", "stripe.com"], "recordTypes": ["MX", "TXT"], "resolver": "cloudflare", "parseEmailPolicy": true } ``` ### Input | Field | Type | Default | Notes | |---|---|---|---| | `domains` | array | - | **Required.** Hostnames. Schemes, credentials, ports, paths, trailing dots and email-style inputs are all stripped, and the result is lower-cased and de-duplicated. A `www.` prefix is **kept**: in DNS `www.example.com` is a different name from `example.com`, with its own records, so it is resolved and charged as its own domain. | | `recordTypes` | array | `["A","AAAA","MX","TXT","NS"]` | Any of `A`, `AAAA`, `MX`, `TXT`, `NS`, `CNAME`, `SOA`, `CAA`. One lookup and one dataset item each. | | `resolver` | string | `google` | `google` or `cloudflare`. The other is used automatically as a fallback. | | `maxLookups` | integer | `1000` | Ceiling on the dataset items the **whole run** may produce. `0` means no ceiling. | | `parseEmailPolicy` | boolean | `true` | Extract SPF from TXT and fetch DMARC from `_dmarc.`. | **Which combinations make sense.** Every domain is queried for every record type, so cost is `domains × recordTypes`, trimming `recordTypes` is how you keep a large sweep cheap. `parseEmailPolicy` adds one DMARC lookup per domain, plus one SPF lookup only when `TXT` is not already in your list, so pairing it with `recordTypes: ["MX","TXT"]` is the efficient shape for an email audit. Those extra policy lookups enrich the rows you already receive and are never charged as rows of their own. **`maxLookups` is a run total, not a per-domain allowance.** 40 domains × 5 record types is 200 items; with `maxLookups: 50` the run resolves domains in the order you listed them, stops at 50, and names every domain it never reached in `RUN_SUMMARY.domainsSkippedByCap`. If the ceiling falls part-way through a domain, only the first few of your `recordTypes` are resolved for it, so `mxHosts` and `nameservers` can be empty on that domain's rows; the types it did not reach are named in `RUN_SUMMARY.recordTypesSkippedByCap`. A value below `0`, or one that is not a whole number, ends the run with an error rather than being read as "no ceiling". `CNAME` on an apex domain almost always returns `NOERROR` with zero records, apex CNAMEs are not valid DNS. Query it on sub-domains, where it tells you what a host actually points at. ### Output example ```json { "domain": "apify.com", "recordType": "MX", "status": "NOERROR", "statusCode": 0, "value": [ "1 aspmx.l.google.com.", "5 alt1.aspmx.l.google.com.", "10 aspmx2.googlemail.com." ], "records": [ { "name": "apify.com", "type": "MX", "ttl": 86400, "data": "1 aspmx.l.google.com." } ], "ttl": 86400, "resolver": "google", "failedOver": false, "responseTimeMs": 33, "hasSpf": true, "spfRecord": "v=spf1 a mx include:_spf.google.com include:mailgun.org -all", "spfPolicy": "-all", "hasDmarc": true, "dmarcRecord": "v=DMARC1; p=reject; sp=reject; pct=100; rua=mailto:dmarc-reports@apify.com; ri=604800", "dmarcPolicy": "reject", "mxHosts": [ { "preference": 1, "host": "aspmx.l.google.com" }, { "preference": 5, "host": "alt1.aspmx.l.google.com" }, { "preference": 10, "host": "aspmx2.googlemail.com" } ], "nullMx": false, "nameservers": [ "ns-449.awsdns-56.com", "ns-839.awsdns-40.net", "ns-1225.awsdns-25.org", "ns-1928.awsdns-49.co.uk" ], "scrapedAt": "2026-08-06T11:43:11.221Z" } ``` `RUN_SUMMARY` looks like this: ```json { "domainsRequested": 2, "domainsRejected": ["not a domain!!"], "domainsSkippedByCap": [], "recordTypesSkippedByCap": [], "lookupsRequested": 2, "lookupsSkippedByCap": 0, "lookupsFailed": 0, "failures": [], "emailPolicyLookupsFailed": 1, "emailPolicyFailures": [ { "domain": "stripe.com", "lookup": "DMARC", "error": "retryable HTTP 503" } ], "lookupsSaved": 2, "capReached": false, "filters": { "domains": ["apify.com", "stripe.com"], "recordTypes": ["MX"], "resolver": "google", "parseEmailPolicy": true, "maxLookups": 1000 }, "finishedAt": "2026-08-06T11:43:48.478Z" } ``` Every item `lookupsRequested` promised and `lookupsSaved` did not deliver is accounted for by exactly one of three things: `failures` (the lookup was attempted and did not answer), `domainsSkippedByCap` (the cap ran out before that domain), or `recordTypesSkippedByCap` (the cap ran out part-way through it). `lookupsSkippedByCap` is the total the cap cut. `emailPolicyFailures` explains every `hasSpf` or `hasDmarc` that came back `null`: those rows are real DNS answers, but the email verdict on them is unknown rather than negative. ### Reading the status field `status` is the DNS response code, decoded. It is the difference between "this domain has no MX records" and "this domain does not exist". | Status | Means | |---|---| | `NOERROR` with records | The domain has records of this type | | `NOERROR` with an empty `value` | The domain exists but has no record of this type, normal for `CNAME` on an apex, or `AAAA` on IPv4-only hosts. `records` may still show the CNAME chain that was followed | | `NXDOMAIN` | The domain does not exist at all | | `SERVFAIL` | The authoritative server broke or DNSSEC validation failed, often a real misconfiguration worth flagging | | `REFUSED` | The authoritative server declined to answer | ### Limits and behaviour - **`maxLookups` bounds the whole run.** It is the ceiling on delivered items, and therefore on cost. Domains are resolved in the order you listed them and anything the ceiling leaves out is named in `RUN_SUMMARY.domainsSkippedByCap` and `RUN_SUMMARY.recordTypesSkippedByCap`, so a missing row never reads as a domain with no records. - **Automatic failover.** If your chosen resolver fails after its retries, the other one is tried before the lookup is recorded as failed. Every item names the resolver that actually answered and sets `failedOver` when it was the backup. Occasional disagreement between the two is normal, they hit different authoritative caches, which is exactly why the fallback is useful. - **Response codes and record types are decoded.** DNS-over-HTTPS returns the response code and each answer's type as bare integers. Both are mapped to their IANA names, so answers that contain a CNAME chain on the way to an A record show up as `CNAME` and `A` rather than `5` and `1`. - **TTL follows the records you asked for.** When a response mixes records with different TTLs, the shortest of the requested type is the one that governs when that answer goes stale. A CNAME picked up on the way does not set the TTL of a record that does not exist. - **A null MX is not a mail host.** RFC 7505's `0 .` means the domain accepts no mail. It is reported as `nullMx: true` with an empty `mxHosts`, never as a mail exchanger with a blank hostname. - **Trailing dots are stripped.** One resolver echoes fully-qualified names with a root dot and the other does not; `mxHosts` and `nameservers` are normalised so results are comparable across resolvers. - **Unparseable input is reported, not silently dropped.** Anything that is not a hostname lands in `RUN_SUMMARY.domainsRejected`; unsupported record types are logged and skipped. - **One failed lookup never kills the run.** Failures land in `RUN_SUMMARY.failures` with the domain and record type; the Actor only throws when every lookup failed. - **A failed SPF or DMARC lookup is a `null`, never a `false`.** "This domain publishes no DMARC" and "we could not find out" are opposite findings and an audit acts on them differently, so the second one is reported as `null` and named in `RUN_SUMMARY.emailPolicyFailures`. - **Passive, public data only.** DNS is public infrastructure metadata. Nothing here touches the domains themselves, no port scanning, no connection to the hosts, no probing. ### API example ```bash curl -X POST "https://api.apify.com/v2/acts/arman-bd~dns-records-scraper/run-sync-get-dataset-items?token=YOUR_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "domains": ["apify.com", "stripe.com", "github.com"], "recordTypes": ["MX", "TXT"], "resolver": "google", "parseEmailPolicy": true }' ``` ### JavaScript example ```js import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: 'YOUR_TOKEN' }); const run = await client.actor('arman-bd/dns-records-scraper').call({ domains: ['apify.com', 'stripe.com', 'github.com'], recordTypes: ['MX'], parseEmailPolicy: true, }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); for (const r of items) { // hasDmarc is null when the lookup could not be made. Judging that as "no DMARC" // would report a protected domain as spoofable, so it gets its own verdict. const risk = r.hasDmarc === null ? 'UNKNOWN' : r.dmarcPolicy === 'reject' ? 'ok' : 'SPOOFABLE'; console.log(`${r.domain}\tSPF ${r.spfPolicy ?? 'none'}\tDMARC ${r.dmarcPolicy ?? 'none'}\t${risk}`); } ``` **Defaults:** 1 GB memory, 15 minute timeout. ### FAQ **Do I need a proxy?** No. Proxy configuration is not required to run this Actor. **Do I need credentials of my own?** No. There is nothing for you to supply beyond the input. **What happens if a source is unavailable?** The other resolver is tried automatically. If both fail, that single lookup is recorded in `RUN_SUMMARY.failures` and the run continues. The Actor only errors out when every lookup fails. **How do I stop a big list from costing more than I expect?** Set `maxLookups`. It caps the items the whole run can deliver, not the items per domain, and whatever it leaves unresolved is listed in `RUN_SUMMARY.domainsSkippedByCap` and `RUN_SUMMARY.recordTypesSkippedByCap`. **Can I schedule it?** Yes, it is built for it. Run daily over your domain list and diff on `value` to catch DNS changes, expired records or hijacked sub-domains. **Why do Google and Cloudflare return different IP addresses?** Because many domains use geo-aware or load-balanced DNS, so the answer depends on which resolver asked. That is the real state of the world, not an error. Pin `resolver` if you need runs to be comparable. **Why is `hasDmarc` false for a domain that clearly sends email?** DMARC lives at `_dmarc.`, not on the domain itself, and plenty of senders never publish one. That absence is the finding. `false` always means the lookup ran and found nothing; if the lookup itself could not be made you get `null`. **What is the difference between `-all` and `~all` in `spfPolicy`?** `-all` tells receivers to reject mail from unlisted senders; `~all` only asks them to mark it suspicious. `-all` is the stronger posture. **Does it do reverse DNS or zone transfers?** No. Forward lookups of specific record types only, no PTR sweeps, no AXFR, nothing intrusive. **Can I integrate it with something else?** Yes, Apify API, client libraries, webhooks, scheduled runs, dataset exports (JSON/CSV/Excel) or MCP. Output is structured JSON. # Actor input Schema ## `domains` (type: `array`): Hostnames to resolve. Paste them however you have them. 'apify.com', 'https://apify.com/store' and 'user@apify.com' all normalise to 'apify.com'. Sub-domains work as written, so 'mail.apify.com' resolves the sub-domain, not the apex. ## `recordTypes` (type: `array`): Which record types to look up for every domain. Each one is a separate lookup and a separate dataset item. A, MX and TXT cover most audits; add CAA to see which certificate authorities are permitted, and SOA to identify the authoritative zone. ## `resolver` (type: `string`): Which DNS-over-HTTPS provider answers first. The two occasionally disagree because they hit different authoritative caches. If the chosen one fails, the other is tried automatically and every record says which actually answered. ## `maxLookups` (type: `integer`): Hard ceiling on the number of dataset items the whole run may produce, and therefore on what it costs. One item is one domain and one record type, so 10 domains × 5 record types is 50. Domains are resolved in the order you listed them and the run stops the moment the ceiling is reached; anything left is named in RUN\_SUMMARY.domainsSkippedByCap. Set 0 for no ceiling. ## `parseEmailPolicy` (type: `boolean`): Extract the SPF record from the domain's TXT records and fetch the DMARC record from \_dmarc., then decode both policies into readable fields. Adds one lookup per domain and is what makes an email-security audit possible in a single run. ## Actor input object example ```json { "domains": [ "stripe.com", "https://www.cloudflare.com/" ], "recordTypes": [ "A", "AAAA", "MX", "TXT", "NS" ], "resolver": "google", "maxLookups": 1000, "parseEmailPolicy": true } ``` # Actor output Schema ## `items` (type: `string`): Every record the run produced. ## `runsummary` (type: `string`): The RUN\_SUMMARY record from the run's key-value store. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "domains": [ "apify.com", "github.com" ], "recordTypes": [ "A", "AAAA", "MX", "TXT", "NS" ], "maxLookups": 1000 }; // Run the Actor and wait for it to finish const run = await client.actor("arman-bd/dns-records-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "domains": [ "apify.com", "github.com", ], "recordTypes": [ "A", "AAAA", "MX", "TXT", "NS", ], "maxLookups": 1000, } # Run the Actor and wait for it to finish run = client.actor("arman-bd/dns-records-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "domains": [ "apify.com", "github.com" ], "recordTypes": [ "A", "AAAA", "MX", "TXT", "NS" ], "maxLookups": 1000 }' | apify call arman-bd/dns-records-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,arman-bd/dns-records-scraper" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/soRVjNcojHaza2zOO/builds/jiK7PAuxE6UptNy8o/openapi.json