Certificate Transparency Subdomain Scraper

---

🎯 What this scrapes

Every TLS certificate ever issued for a domain is logged forever in public Certificate Transparency logs — a firehose that crt.sh exposes as a free, keyless JSON endpoint. The catch: the raw payload is large, un-deduplicated, and carries no computed security signal. This Actor queries crt.sh per root domain and turns that firehose into a clean, capped dataset: one row per certificate (or per unique subdomain, your choice) with expiry countdown, expired/wildcard flags, and issuer metadata already computed — ready for an attack-surface inventory, a cert-hygiene sweep, or a daily subdomain-discovery job.

🔥 What we handle for you

• 🔎 One query per domain, whole log in one shot — crt.sh has no pagination; we fetch the complete result and process it in memory, so results land in one pass.
• 🧮 Computed security signals — days_until_expiry, is_expired, and is_wildcard come pre-computed, not left for you to derive from raw timestamps.
• 🧊 Deduplication, your way — one row per unique subdomain (keeping the newest cert), or one row per raw certificate/SAN entry if you want the closer-to-raw shape.
• 🚧 Egress guard built in — maxCertsPerDomain caps the response before a single row is built, so a domain with 10,000+ historical certificates can't blow your bill.
• 🔁 Retries with exponential backoff on 408 / 429 / 503 and network errors — a slow moment on crt.sh's shared database doesn't fail your run.
• 💰 Pay-Per-Event pricing — you only pay for certificates that land in your dataset. No data, no charge beyond the warm-up fee.

💡 Use cases

• Attack-surface enumeration — pentest and bug bounty teams discovering subdomains from CT logs instead of brute-force DNS.
• Certificate hygiene monitoring — schedule daily runs across your own fleet and catch expiring or duplicate certificates.
• Brand protection — spot lookalike-domain certificate issuance for enforcement leads.
• Security research — track issuer patterns and wildcard usage across a domain's certificate history.

⚙️ How to use it

1. Click Try for free at the top of the page.
2. Fill in the input form — most fields have sensible defaults.
3. Click Start. Output streams into the run's dataset.
4. Export from Storage → Dataset as JSON, CSV, or Excel — or fetch via the API.

📥 Input

Field	Type	Required	Default	Notes
domains	array	yes	['github.com', 'gitlab.com']	Root domains to query against crt.sh, e.g. example.com. One crt.sh query per domain.
maxCertsPerDomain	integer	no	200	Hard cap on rows emitted per domain — the egress guard. crt.sh can return tens of thousands of records for a large…
deduplicateSubdomains	boolean	no	True	When true, emit one row per unique subdomain (keeping the newest certificate's metadata). When false, emit one row per…
includeExpired	boolean	no	True	When false, drop certificates whose validity end date is in the past.
proxyConfiguration	object	no	{'useApifyProxy': True}	Apify Proxy configuration. crt.sh is a public keyless JSON API with no anti-bot behavior, so no RESIDENTIAL group is…

Example input

{
  "domains": [
    "github.com",
    "gitlab.com"
  ],
  "maxCertsPerDomain": 10,
  "deduplicateSubdomains": true,
  "includeExpired": true,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}

📤 Output

Every row is one dataset item.

Field	Type	Notes
domain	string	The queried root domain.
subdomain	string	Parsed from name_value (or common_name fallback).
certificate_id	integer	crt.sh certificate id.
issuer_name	string	crt.sh issuer_name.
common_name	string	crt.sh common_name.
serial_number	string	crt.sh serial_number, when published.
not_before	string	Certificate validity start, ISO-8601.
not_after	string	Certificate validity end, ISO-8601.
entry_timestamp	string	crt.sh CT-log entry timestamp, ISO-8601, when published.
days_until_expiry	integer	(not_after - now).days — negative when already expired.
is_expired	boolean	True when not_after is in the past.
is_wildcard	boolean	True when the subdomain starts with '*.'.

Example output

{
  "domain": "github.com",
  "subdomain": "api.github.com",
  "certificate_id": 12345678,
  "issuer_name": "C=US, O=DigiCert Inc, CN=DigiCert TLS RSA SHA256 2020 CA1",
  "common_name": "github.com",
  "serial_number": "0a1b2c3d4e5f",
  "not_before": "2026-01-15T00:00:00+00:00",
  "not_after": "2027-01-15T23:59:59+00:00",
  "entry_timestamp": "2026-01-15T08:12:03+00:00",
  "days_until_expiry": 132,
  "is_expired": false,
  "is_wildcard": false
}

💰 Pricing

Pay-Per-Event — you pay only when these events fire:

Event	USD	What it is
actor-start	$0.20	One-off warm-up charge per run
certificate-scraped	$0.006	Per certificate/subdomain row written to the dataset

Example: 1 000 results at the rates above ≈ $6.20. No subscription, no minimum, no card to start — Apify gives every new account $5 of free credit.

🚧 Limitations

• Certificate Transparency logs only see certificates issued by CAs that log to CT — self-signed or non-logged internal certificates never appear.
• No DNS resolution or liveness check — a discovered subdomain in the dataset may no longer resolve to a live host.
• No historical diffing between runs in v1 — compare successive dataset exports yourself to see what changed.

❓ FAQ

Is this legal?

Yes — Certificate Transparency logs are public by design (RFC 9162); every certificate a public CA issues is logged forever, and crt.sh exposes that log through a free, keyless, publicly documented endpoint.

Does this scan or connect to the discovered subdomains?

No — this Actor reads Certificate Transparency log metadata only. It never resolves DNS or connects to any subdomain it discovers.

What happens if a domain has zero certificates?

The run still succeeds — a clean, no-match answer is a completed search, not an error.

Why is a domain's row count capped?

Certificate Transparency logs can return tens of thousands of rows for a popular domain (google.com, for instance). maxCertsPerDomain is the egress guard that keeps your bill predictable — raise it if you need more.

💬 Your feedback

Spotted a bug, hit a weird edge case, or need a new field? Open an issue on the Actor's Issues tab on Apify Console — we ship fixes weekly and we read every report.

---