Subdomain Enumeration via Certificate Transparency

Subdomain Enumeration via Certificate Transparency

Discover every subdomain for any domain by querying Certificate Transparency (CT) logs. Aggregate results from crt.sh and CertSpotter, with automatic deduplication, sorting, change tracking, and continuous monitoring — all in a single Apify actor.

---

What does it do?

Discover every subdomain for any domain by querying Certificate Transparency (CT) logs. Aggregate results from crt.sh and CertSpotter, with automatic deduplication, sorting, change tracking, and continuous monitoring — all in a single Apify actor.

---

Features

1. Multi-Source Aggregation — Queries crt.sh (primary) and CertSpotter (secondary) for maximum CT log coverage.
2. Automatic Deduplication — Merges results from all sources, removing exact duplicates while preserving source and certificate metadata.
3. Wildcard Detection — Identifies and counts wildcard certificate entries (*.example.com) separately.
4. Diff Reporting — Compares the current scan against the previous scan of the same domain. Produces lists of new, removed, and unchanged subdomains with counts.
5. Continuous Monitoring — Persistent state file tracks all subdomains ever discovered for a domain. Each run records a change event showing what's new since the last check.
6. Batch Mode — Scan multiple domains in a single run. Provide an array of domain objects via the batchData field.
7. Exponential-Backoff Retry — Transient HTTP errors (timeouts, 429, 503) are retried with exponential backoff for resilience.
8. Concurrency Control — Source-level and domain-level semaphores prevent overwhelming upstream APIs.
9. Sorted Output — Subdomains are sorted alphabetically with wildcard entries grouped at the end.
10. Detailed Metadata — Each entry includes the source, certificate ID, validity dates, and common name where available.

---

Why use this?

• Two CT sources in one tool — crt.sh and CertSpotter together cover more certificate data than either alone. No need to run separate tools and merge results manually.
• Stateful change tracking — The diff and monitor modes remember what you found last time, so you can run this on a schedule (e.g., daily cron) and get a clean before-and-after report every time.
• Enterprise-ready error handling — Retry logic, rate limiting, graceful degradation, and informative error messages mean this actor works reliably in automated pipelines.
• Wildcard awareness — Wildcard certificates (*.example.com) are counted and flagged separately, giving you a clearer picture of the domain's certificate posture.

---

Who is it for?

Persona	What they use it for
Penetration Tester	Mapping the full attack surface of a target domain before an engagement
Security Team	Monitoring for unauthorised subdomain creation or certificate issuance
DevOps / SRE	Tracking infrastructure drift — new staging environments, removed services
Bug Bounty Hunter	Discovering hidden or forgotten subdomains for vulnerability research
Incident Responder	Quickly enumerating all publicly exposed subdomains of a compromised domain
IT Asset Manager	Maintaining an inventory of all domain-linked services and subdomains

---

Input Parameters

Field	Type	Default	Description
domain	string	example.com	Target domain to enumerate. Protocol prefixes and URL paths are automatically stripped.
fullScan	boolean	false	When true, queries both crt.sh and CertSpotter for maximum coverage. When false, queries crt.sh only (faster, cheaper).
diffReport	boolean	false	When true, compares results against the previous scan snapshot and emits new/removed subdomain lists. Requires a prior scan to exist.
monitorMode	boolean	false	When true, maintains a persistent history of all ever-discovered subdomains and records change deltas. Ideal for scheduled/periodic runs.
batchMode	boolean	false	When true, scans multiple domains from the batchData array. The single domain field is ignored.
batchData	array	[]	Array of domain objects for batch mode. Each object must have a domain field: [{"domain": "example.com"}, {"domain": "google.com"}]

Example Input JSON

{
  "domain": "example.com",
  "fullScan": true,
  "diffReport": false,
  "monitorMode": false,
  "batchMode": false
}

Batch mode example:

{
  "batchMode": true,
  "fullScan": true,
  "batchData": [
    { "domain": "example.com" },
    { "domain": "google.com" },
    { "domain": "github.com" }
  ]
}

---

Output Format

The actor pushes one dataset item per domain scanned. Each item contains the full enumeration report.

Output Fields

Field	Type	Description
domain	string	The target domain that was scanned
subdomains	array	Deduplicated, sorted list of discovered subdomains with metadata
total_found	number	Raw count of entries returned before deduplication
unique_domains	number	Count of unique subdomains after deduplication
wildcard_count	number	Number of wildcard certificate entries (starting with *.)
sources_used	array	List of CT log sources that were successfully queried
scan_mode	string	Operation mode used: single-scan, full-scan, diff-report, or monitor
scan_date	string	ISO-8601 timestamp of the scan
elapsed_ms	number	Wall-clock duration of the enumeration in milliseconds
diff	object	(present only when diffReport=true) Diff report with new/removed/unchanged counts
monitor_delta	object	(present only when monitorMode=true) Changes since the last monitor run
error	string	Error message if the scan failed for this domain

Subdomain Entry Fields

Field	Type	Description
domain	string	Fully-qualified subdomain (e.g., api.example.com). May start with *. for wildcards.
source	string	Source identifier: crt.sh or certspotter
certificate_id	number	Certificate identifier from the source
not_before	string	Certificate validity start date
not_after	string	Certificate validity end date
common_name	string	Certificate Common Name (available from crt.sh)

Example Output JSON

{
  "domain": "example.com",
  "subdomains": [
    {
      "domain": "admin.example.com",
      "source": "crt.sh",
      "certificate_id": 12345678,
      "not_before": "2026-01-01T00:00:00",
      "not_after": "2027-01-01T00:00:00",
      "common_name": "admin.example.com"
    },
    {
      "domain": "api.example.com",
      "source": "certspotter",
      "certificate_id": 87654321,
      "not_before": "2026-03-15T00:00:00",
      "not_after": "2027-03-15T00:00:00"
    },
    {
      "domain": "*.example.com",
      "source": "crt.sh",
      "certificate_id": 11223344,
      "not_before": "2026-06-01T00:00:00",
      "not_after": "2027-06-01T00:00:00"
    }
  ],
  "total_found": 184,
  "unique_domains": 147,
  "wildcard_count": 3,
  "sources_used": ["crt.sh", "certspotter"],
  "scan_mode": "full-scan",
  "scan_date": "2026-07-20T10:00:00.000000+00:00",
  "elapsed_ms": 3842.1,
  "diff": {
    "new_domains": ["new-admin.example.com"],
    "removed_domains": ["old-api.example.com"],
    "new_count": 1,
    "removed_count": 1,
    "unchanged_count": 145,
    "baseline_scan_date": "2026-07-19T10:00:00.000000+00:00",
    "current_scan_date": "2026-07-20T10:00:00.000000+00:00"
  }
}

---

API Usage

cURL

# Single domain, full scan
curl -X POST "https://api.apify.com/v2/acts/perryay~subdomain-enumeration-ct-logs/runs?token=YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "domain": "example.com",
    "fullScan": true
  }'

# Diff report
curl -X POST "https://api.apify.com/v2/acts/perryay~subdomain-enumeration-ct-logs/runs?token=YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "domain": "example.com",
    "diffReport": true
  }'

# Batch mode
curl -X POST "https://api.apify.com/v2/acts/perryay~subdomain-enumeration-ct-logs/runs?token=YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "batchMode": true,
    "fullScan": true,
    "batchData": [
      { "domain": "example.com" },
      { "domain": "google.com" }
    ]
  }'

# Fetch results
curl "https://api.apify.com/v2/datasets/YOUR_DATASET_ID/items?token=YOUR_API_TOKEN&format=json"

Python (Apify SDK)

import asyncio
from apify_client import ApifyClient

# Synchronous client
client = ApifyClient("YOUR_API_TOKEN")

# Run the actor and wait for results
run = client.actor("perryay/subdomain-enumeration-ct-logs").call(
    run_input={
        "domain": "example.com",
        "fullScan": True,
        "diffReport": False,
    },
)

# Fetch dataset
dataset_id = run["defaultDatasetId"]
items = client.dataset(dataset_id).list_items().items

for item in items:
    print(f"{item['domain']}: {item['unique_domains']} unique subdomains")
    if item.get("diff"):
        d = item["diff"]
        print(f"  +{d['new_count']} new, -{d['removed_count']} removed")

---

Use Cases

1. Penetration Testing Reconnaissance — Before any security assessment, enumerate all subdomains to map the full attack surface. Wildcard entries highlight potential domain takeover vectors.
2. Continuous Security Monitoring — Schedule the actor to run daily with monitorMode=true. Each run produces a delta of new subdomains, alerting you to unapproved infrastructure.
3. M&A Due Diligence — When evaluating a target company, enumerate their primary domains to discover shadow IT, forgotten services, and exposed management interfaces.
4. Bug Bounty Automation — Integrate into your recon pipeline. Run against a target, pipe the subdomain list to a port scanner or web crawler for deeper analysis.
5. DevOps Infrastructure Audit — Track which subdomains have valid certificates and which have expired (not_after dates). Identify certs that need renewal before they break TLS.
6. Cloud Migration Tracking — After moving services to a new cloud provider, monitor for old subdomains that should have been decommissioned (detected as removed_domains in diff mode).
7. Subdomain Takeover Prevention — Set up weekly scans for your own domains. If a subdomain disappears from CT logs but still resolves in DNS, it might be vulnerable to takeover.
8. CDN / WAF Onboarding — Before migrating traffic through a CDN, enumerate all subdomains so the onboarding team doesn't miss any origin.

---

FAQ

Q1: How does this actor find subdomains? A: It queries Certificate Transparency (CT) logs — public, append-only ledgers of every X.509 certificate issued by trusted Certificate Authorities. Every time a CA issues a certificate for your domain or one of its subdomains, that certificate (with its Subject Alternative Names) is recorded in a CT log. The actor queries two CT log aggregators: crt.sh (run by Sectigo / the community) and CertSpotter (run by SSLMate). Between them, they cover the vast majority of publicly trusted CAs.

Q2: Is this a live DNS scanner? Does it perform DNS lookups? A: No. This actor does not perform any DNS queries, port scans, or active network probes. It is purely passive — it inspects certificate log data. This makes it safe to run against any domain without alerting the target's monitoring systems.

Q3: Why scan two sources? Isn't one enough? A: Each CT log aggregator has a different coverage window, update latency, and backend. crt.sh has the broadest historical data but may lag a few hours on recent issuances. CertSpotter updates faster but has a smaller historical window. Using both maximises both coverage and freshness.

Q4: How many subdomains can I expect to find? A: It varies enormously by domain. A small blog might have 3–5 subdomains; a large tech company like Google or Microsoft can have tens of thousands. The actor includes a 10,000-entry safety limit per source to prevent runaway memory usage.

Q5: What's the difference between "diff report" and "monitor mode"? A: Diff report compares the current scan against exactly one previous snapshot (the last scan). Monitor mode maintains a full history of all subdomains ever seen across all scans of that domain, accumulating change events. Use diff for simple before/after comparisons; use monitor for long-running scheduled tracking.

Q6: Does this actor support IDN / internationalised domain names? A: Yes. The ct log sources return Punycode-encoded domains (xn--...) which the actor passes through as-is. You can provide both plain ASCII and Punycode input.

Q7: Will I get rate-limited by crt.sh or CertSpotter? A: The actor includes rate-limiting safeguards (concurrent request semaphores, exponential-backoff retries, per-source timeouts). However, for very large runs or high-frequency schedules, the upstream APIs may still rate-limit you. The actor logs HTTP 429 responses and retries with backoff.

Q8: What happens to the state files when I stop using the actor? A: State files are stored in the actor's persistent storage on the Apify platform. They persist between runs for the same actor instance so that diff and monitor modes can track changes over time. If you stop monitoring a domain, the state files remain but do not affect new runs with a different domain.

Q9: Can I use this actor for 10,000 domains in one run? A: The actor was designed for up to a few hundred domains per run. For massive lists, split into batches of 50–100 and run them sequentially or use the actor's batch mode with a reasonable array size. The 3-domain concurrency cap prevents overwhelming upstream APIs.

Q10: Does this actor support wildcard subdomain enumeration (like *.example.com)? A: Yes — it detects wildcard entries present in certificates (e.g., *.example.com) and reports them separately in the wildcard_count field. However, it does not generate every possible subdomain from a wildcard; it only reports what's actually in the certificate logs.

Q11: How fresh is the data from crt.sh and CertSpotter? A: crt.sh typically reflects certificate issuances within a few hours. CertSpotter is generally faster, often reflecting issuances within 30–60 minutes. Neither is real-time (CT log submission + processing delay).

Q12: Is there a limit on the number of subdomains returned? A: Each source has a 10,000-entry safety limit per domain. Combined, you could receive up to 20,000 raw entries, which are then deduplicated. This is sufficient for the vast majority of domains.

---

Usage & Billing

This actor uses Apify's PAY_PER_EVENT pricing model. You are charged only for successful runs based on the events your usage triggers.

Charge Events

Event	Trigger
apify-actor-start	Every run (base)
full-scan	fullScan: true (queries both CT sources)
diff-report	diffReport: true
monitor	monitorMode: true

Platform costs (Apify's infrastructure fee) are passed through to the customer.

---

MCP Integration

This actor can be used through the Apify MCP server. Once connected, your MCP client (Claude Desktop, Cursor, etc.) can discover and run this actor from the Apify Store.

Quick Start

1. Install the Apify connector in your MCP client:

   • Claude Desktop: Search for "Apify" in the connector directory, or use the remote server at https://mcp.apify.com
   • Other clients: See the Apify MCP server docs for setup instructions

2. Ask your AI assistant to enumerate subdomains using natural language. For example:

   "Enumerate all subdomains for example.com using CT logs" "Run a full scan on github.com and show me the wildcard subdomains"

Claude Desktop Configuration

Add the following to your claude_desktop_config.json:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com"
    }
  }
}

On first connection, your browser opens to sign in to Apify and authorize access.

Bearer token alternative: For headless environments, CI/CD pipelines, or clients without browser-based OAuth, you can authenticate directly with your Apify API token:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com",
      "headers": {
        "Authorization": "Bearer YOUR_APIFY_TOKEN"
      }
    }
  }
}

Get your API token from Apify Console → API & Integrations section.

---

Related Tools

• SSL Certificate Checker — Check SSL/TLS certificate expiry dates and issuer details for a domain. Complements subdomain enumeration by verifying certificate health.
• Domain Intel — Domain registration and WHOIS intelligence gathering.
• JSON Studio — Format, validate, and transform JSON data for post-processing subdomain lists.

---