Certificate Transparency Scraper: Subdomain Discovery

Query certificate transparency logs via crt.sh to enumerate subdomains, issuers, validity windows and certificate history for any domain. Passive attack-surface discovery. Give it domains and it returns one structured record per certificate.

Agent skill: SKILL.md

https://api.apify.com/v2/key-value-stores/t7YoTxpZEJOWvw4Ug/records/certificate-transparency-scraper.md

What it does

Every publicly trusted TLS certificate issued since 2018 must be logged to a public, append-only certificate transparency (CT) log. That makes CT logs an unusually complete record of an organisation's host names, including hosts that were never linked to, never indexed, and long since decommissioned.

This Actor queries crt.sh, the public CT log search service, for each domain you give it and returns:

• every subject alternative name in every certificate ever issued for the domain or its subdomains,
• the issuing CA (name and crt.sh CA id),
• the validity window (notBefore / notAfter) and whether the certificate has already expired,
• the serial number, the crt.sh certificate id and a direct link to the certificate,
• the CT log entry timestamp.

Two modes:

• Certificate history (default), one record per certificate, newest first. Shows renewal cadence, CA changes and issuance patterns.
• Unique subdomains (deduplicateSubdomains: true), one record per distinct host name, expanded out of every certificate's SAN list. This is the clean inventory you want for attack-surface mapping.

In both modes a certificate is only ever returned — and only ever charged — once. CT logging records every certificate at least twice, as the precertificate the CA committed to and as the certificate it then issued, and two domains in one run overlap whenever one sits beneath the other. The default mode collapses those repeats on the certificate's own identity, its issuer and serial number; the unique-subdomain mode collapses them on the host name. Either way the suppressed rows are counted in RUN_SUMMARY.duplicatesSkipped instead of charged, and a certificate two of your domains share is delivered once, under whichever of them the run reached first.

Input

Field	Type	Required	Default	Description
domains	array	✅	-	Domains to enumerate. The search matches the domain and every subdomain beneath it, so apify.com also returns docs.apify.com. Full URLs, *. wildcard prefixes and mixed case are normalized to one query, so the same domain written several ways is queried, and charged, once. An entry that is not a host name is skipped and named in RUN_SUMMARY.inputsRejected.
includeExpired	boolean		true	Include certificates whose validity window has ended. Expired certificates are where the forgotten hosts hide. Turning this off returns only the live estate, and a smaller result set.
deduplicateSubdomains	boolean		false	Emit one record per unique host name instead of one per certificate. Every SAN is expanded and the most recent certificate covering each name is kept.
maxResults	integer		1000	Cap the total records saved across the whole run, shared evenly across the domains, with whatever a quiet domain leaves unused going to the ones with more history. Records are newest first, so a low cap keeps the most recent issuance. 0 means no limit.

maxResultsPerDomain was the earlier name for the cap. It is still accepted and now means the same run-wide total; if both are given, the tighter one applies. A cap that is negative, fractional or not a number is refused and the run ends before anything is fetched or charged.

Example input

{
 "domains": ["apify.com", "https://example.com/path", "*.github.com"],
 "includeExpired": true,
 "deduplicateSubdomains": true,
 "maxResults": 40
}

Which combinations make sense

• Attack-surface mapping: includeExpired: true + deduplicateSubdomains: true. Widest possible host list, one row each.
• Live estate audit: includeExpired: false + deduplicateSubdomains: true. Only names with a currently valid certificate, and a smaller result set.
• Certificate history / CA audit: deduplicateSubdomains: false. You want the renewals, they are the signal. What you do not get is the same certificate twice.
• maxResults interacts with dedup: the cap counts saved records, so with dedup on it caps unique host names, and with it off it caps certificates.

Output

One dataset item per certificate (or per unique subdomain in dedup mode). A real record from apify.com:

{
 "domain": "apify.com", // the input domain this came from
 "subdomain": "affiliate.apify.com", // the host name for this record
 "commonName": "affiliate.apify.com", // certificate CN
 "nameValue": ["affiliate.apify.com"], // every SAN in the certificate, de-duplicated
 "issuerName": "C=US, O=Let's Encrypt, CN=YR2",
 "issuerCaId": 432477, // the log search's internal CA identifier
 "serialNumber": "053e26ba7c9f4baa4c9bc1936a9fc4f47f33",
 "notBefore": "2026-08-09T03:25:05", // UTC, naive, as the log reports it
 "notAfter": "2026-11-07T03:25:04",
 "isExpired": false, // computed against run time
 "entryTimestamp": "2026-08-09T04:23:36.294", // when the CT log accepted the entry
 "certificateId": 28666584809,
 "certificateUrl": "https://crt.sh/?id=28666584809",
 "scrapedAt": "2026-08-11T21:06:37.751Z"
}

A multi-name certificate populates nameValue fully, for example ["*.apify.com", "apify.com"] on an Amazon RSA 2048 M04 wildcard.

RUN_SUMMARY

Written to the key-value store as RUN_SUMMARY:

{
 "domainsRequested": 2,
 "domainsQueried": 1,
 "domainsFailed": 0,
 "domainsSkipped": [],
 "inputsRejected": [
 { "input": "zzz-nonexistent-target-99999", "error": "not a host name — expected something like \"example.com\"" }
 ],
 "logEntriesFound": 970,
 "certificatesFound": 509,
 "uniqueSubdomainsFound": 39,
 "duplicatesSkipped": 1003,
 "recordsSaved": 39,
 "capReached": false,
 "failures": [],
 "filters": {
 "domains": ["apify.com"],
 "includeExpired": true,
 "deduplicateSubdomains": true,
 "maxResults": 40,
 "capField": "maxResults"
 },
 "finishedAt": "2026-08-11T21:06:37.778Z"
}

That run is real. apify.com produced 970 CT log entries, which are 509 actual certificates carrying 39 distinct host names; in this mode those 39 names are the 39 saved rows, and everything else was a repeat that was not charged. The second entry is not a host name, so it was skipped and named rather than dropped.

Reading it: recordsSaved is what you were charged for. duplicatesSkipped is what you were not — repeats of a certificate or of a host name that the run had already delivered. capReached: true means the cap truncated the run rather than the data running out, and domainsSkipped names domains that were never queried because the cap was already spent. failures lists any domain the search could not serve, with the reason.

Use cases

1. Map an organisation's external attack surface. Every host name the organisation ever certified, including ones that no longer resolve.

{ "domains": ["example.com"], "includeExpired": true, "deduplicateSubdomains": true, "maxResults": 0 }

2. Discover forgotten staging subdomains. Expired certificates for staging., dev., uat. hosts are the classic finding, they often outlive the team that made them.

{ "domains": ["example.com"], "includeExpired": true, "deduplicateSubdomains": true }

Then filter the dataset on subdomain for staging, dev, test, internal.

3. Monitor for typosquatting certificates. Watch look-alike domains for newly issued certificates, a certificate on a typosquat is usually the last step before a phishing campaign goes live.

{ "domains": ["exarnple.com", "example-support.com", "examp1e.com"], "includeExpired": false, "maxResults": 25 }

Schedule daily and alert on any new certificateId.

Limits and behaviour

• A busy domain takes a while. A domain with a long certificate history comes back as one large response, and the run waits for it. Leave the default 900 s run timeout alone if you are querying several domains.
• A domain that does not come back cleanly is retried, up to 6 attempts with linear backoff, and anything that is not the expected structured answer counts as a failure to retry rather than as an empty result. The ladder is bounded: a domain that is still not answering after a few minutes is given up on and recorded in RUN_SUMMARY.failures, so one unreachable domain cannot spend the whole run and leave you without a summary.
• A domain with no certificates is not an error. It records zero results and moves on.
• You are charged once per delivered record, and never twice for the same certificate. CT logging stores every certificate at least twice — precertificate and certificate — and overlapping domains in one run return the same certificate again. Those repeats are collapsed on issuer plus serial number, counted in RUN_SUMMARY.duplicatesSkipped, and not charged.
• The cap is a run-wide total. maxResults bounds the whole run, not each domain, so two domains at maxResults: 40 deliver at most 40 rows between them. The allowance starts as an even split, and anything a domain leaves unspent — because it was empty, or because everything it returned was already delivered — is filled at the end from the certificates the other domains held back, so a run stops short of the cap only when the certificates genuinely run out. A cap that is negative, fractional or not a number ends the run with an error before anything is fetched, rather than being read as "no limit".
• Failures never abort the run. A domain that fails after all retries is logged, recorded in RUN_SUMMARY.failures, and the run continues. The run only fails outright if every domain failed.
• Newest first. Within each domain, records are ordered by CT log entry timestamp, descending, and the sort happens before the cap is applied, so a low maxResults keeps the most recent issuance rather than whatever the search happened to list first.
• CT covers publicly trusted certificates only. Internal PKI, self-signed certificates and hosts that never had a certificate will not appear. CT is a floor on the attack surface, not a ceiling.
• No pagination. The complete result set arrives in one response; there is nothing to page through.

FAQ

Do I need a proxy? No. Proxy configuration is not required to run this Actor.

Do I need an account on the source? No. You supply no credentials.

What happens if a source is unavailable? It gets up to 6 attempts, or a few minutes, whichever ends first; then it is reported in RUN_SUMMARY.failures and the run continues with the next domain. Only a total failure of every domain aborts the run.

Can I schedule it? Yes, it is designed for scheduled runs. Daily monitoring for newly issued certificates is a primary use case; compare certificateId values between runs to spot new issuance.

Why did my run take several minutes? A long certificate history is a large response, and domains are queried one at a time with a pause between them, so a run over several domains is minutes rather than seconds.

Am I charged for the duplicate certificates in the logs? No. One certificate is one charged row, whichever mode you run and however many log entries or overlapping domains carry it. Rows that were suppressed as duplicates are reported in RUN_SUMMARY.duplicatesSkipped.

Is this intrusive? Will the target notice? No. The Actor never contacts the target domain. It only reads a public append-only log. This is entirely passive OSINT.

Why does one host name still appear on many rows? Because the default mode is one record per certificate, and a host renewed every 90 days accumulates dozens of genuinely different certificates over the years. Set deduplicateSubdomains: true for a clean host inventory.