# Certificate Transparency Subdomain Scraper (`devilscrapes/crt-sh-certificate-transparency-scraper`) Actor Enumerate subdomains and audit certificate hygiene via crt.sh's public Certificate Transparency log — no API key. Get one row per certificate or unique subdomain with computed expiry, wildcard, and freshness signals, capped so a popular domain never blows your bill. - **URL**: https://apify.com/devilscrapes/crt-sh-certificate-transparency-scraper.md - **Developed by:** [DevilScrapes](https://apify.com/devilscrapes) (community) - **Categories:** Developer tools - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing Pay per event This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. - **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use). - **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`. - **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`. - **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`. - **Any other language** — the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README
Devil Scrapes mark ## Certificate Transparency Subdomain Scraper **💰 $6.20 / 1 000 results**  ·  pay only for results  ·  no credit card to try *We do the dirty work so your dataset stays clean.* 😈 Enumerate subdomains and audit certificate hygiene via crt.sh's public Certificate Transparency log — no API key. Get one row per certificate or unique subdomain with computed expiry, wildcard, and freshness signals, capped so a popular domain never blows your bill.
*** ### 🎯 What this scrapes Every TLS certificate ever issued for a domain is logged forever in public Certificate Transparency logs — a firehose that crt.sh exposes as a free, keyless JSON endpoint. The catch: the raw payload is large, un-deduplicated, and carries no computed security signal. This Actor queries crt.sh per root domain and turns that firehose into a clean, capped dataset: one row per certificate (or per unique subdomain, your choice) with expiry countdown, expired/wildcard flags, and issuer metadata already computed — ready for an attack-surface inventory, a cert-hygiene sweep, or a daily subdomain-discovery job. ### 🔥 What we handle for you - 🔎 **One query per domain, whole log in one shot** — crt.sh has no pagination; we fetch the complete result and process it in memory, so results land in one pass. - 🧮 **Computed security signals** — `days_until_expiry`, `is_expired`, and `is_wildcard` come pre-computed, not left for you to derive from raw timestamps. - 🧊 **Deduplication, your way** — one row per unique subdomain (keeping the newest cert), or one row per raw certificate/SAN entry if you want the closer-to-raw shape. - 🚧 **Egress guard built in** — `maxCertsPerDomain` caps the response before a single row is built, so a domain with 10,000+ historical certificates can't blow your bill. - 🔁 **Retries with exponential backoff** on `408 / 429 / 503` and network errors — a slow moment on crt.sh's shared database doesn't fail your run. - 💰 **Pay-Per-Event pricing** — you only pay for certificates that land in your dataset. No data, no charge beyond the warm-up fee. ### 💡 Use cases - Attack-surface enumeration — pentest and bug bounty teams discovering subdomains from CT logs instead of brute-force DNS. - Certificate hygiene monitoring — schedule daily runs across your own fleet and catch expiring or duplicate certificates. - Brand protection — spot lookalike-domain certificate issuance for enforcement leads. - Security research — track issuer patterns and wildcard usage across a domain's certificate history. ### ⚙️ How to use it 1. Click **Try for free** at the top of the page. 2. Fill in the input form — most fields have sensible defaults. 3. Click **Start**. Output streams into the run's dataset. 4. Export from **Storage → Dataset** as JSON, CSV, or Excel — or fetch via the API. ### 📥 Input | Field | Type | Required | Default | Notes | |---|---|:--:|---|---| | `domains` | `array` | **yes** | \['github.com', 'gitlab.com'] | Root domains to query against crt.sh, e.g. example.com. One crt.sh query per domain. | | `maxCertsPerDomain` | `integer` | no | 200 | Hard cap on rows emitted per domain — the egress guard. crt.sh can return tens of thousands of records for a large… | | `deduplicateSubdomains` | `boolean` | no | True | When true, emit one row per unique subdomain (keeping the newest certificate's metadata). When false, emit one row per… | | `includeExpired` | `boolean` | no | True | When false, drop certificates whose validity end date is in the past. | | `proxyConfiguration` | `object` | no | {'useApifyProxy': True} | Apify Proxy configuration. crt.sh is a public keyless JSON API with no anti-bot behavior, so no RESIDENTIAL group is… | #### Example input ```json { "domains": [ "github.com", "gitlab.com" ], "maxCertsPerDomain": 10, "deduplicateSubdomains": true, "includeExpired": true, "proxyConfiguration": { "useApifyProxy": true } } ``` ### 📤 Output Every row is one dataset item. | Field | Type | Notes | |---|---|---| | `domain` | `string` | The queried root domain. | | `subdomain` | `string` | Parsed from name\_value (or common\_name fallback). | | `certificate_id` | `integer` | crt.sh certificate id. | | `issuer_name` | `string` | crt.sh issuer\_name. | | `common_name` | `string` | crt.sh common\_name. | | `serial_number` | `string` | crt.sh serial\_number, when published. | | `not_before` | `string` | Certificate validity start, ISO-8601. | | `not_after` | `string` | Certificate validity end, ISO-8601. | | `entry_timestamp` | `string` | crt.sh CT-log entry timestamp, ISO-8601, when published. | | `days_until_expiry` | `integer` | (not\_after - now).days — negative when already expired. | | `is_expired` | `boolean` | True when not\_after is in the past. | | `is_wildcard` | `boolean` | True when the subdomain starts with '\*.'. | #### Example output ```json { "domain": "github.com", "subdomain": "api.github.com", "certificate_id": 12345678, "issuer_name": "C=US, O=DigiCert Inc, CN=DigiCert TLS RSA SHA256 2020 CA1", "common_name": "github.com", "serial_number": "0a1b2c3d4e5f", "not_before": "2026-01-15T00:00:00+00:00", "not_after": "2027-01-15T23:59:59+00:00", "entry_timestamp": "2026-01-15T08:12:03+00:00", "days_until_expiry": 132, "is_expired": false, "is_wildcard": false } ``` ### 💰 Pricing Pay-Per-Event — you pay only when these events fire: | Event | USD | What it is | |---|---:|---| | `actor-start` | $0.20 | One-off warm-up charge per run | | `certificate-scraped` | $0.006 | Per certificate/subdomain row written to the dataset | Example: 1 000 results at the rates above ≈ **$6.20**. No subscription, no minimum, no card to start — Apify gives every new account $5 of free credit. ### 🚧 Limitations - Certificate Transparency logs only see certificates issued by CAs that log to CT — self-signed or non-logged internal certificates never appear. - No DNS resolution or liveness check — a discovered subdomain in the dataset may no longer resolve to a live host. - No historical diffing between runs in v1 — compare successive dataset exports yourself to see what changed. ### ❓ FAQ **Is this legal?** Yes — Certificate Transparency logs are public by design (RFC 9162); every certificate a public CA issues is logged forever, and crt.sh exposes that log through a free, keyless, publicly documented endpoint. **Does this scan or connect to the discovered subdomains?** No — this Actor reads Certificate Transparency log metadata only. It never resolves DNS or connects to any subdomain it discovers. **What happens if a domain has zero certificates?** The run still succeeds — a clean, no-match answer is a completed search, not an error. **Why is a domain's row count capped?** Certificate Transparency logs can return tens of thousands of rows for a popular domain (google.com, for instance). `maxCertsPerDomain` is the egress guard that keeps your bill predictable — raise it if you need more. ### 💬 Your feedback Spotted a bug, hit a weird edge case, or need a new field? Open an issue on the Actor's **Issues** tab on Apify Console — we ship fixes weekly and we read every report. ***
Built by **[Devil Scrapes](https://apify.com/DevilScrapes)** 😈 — a small fleet of opinionated public-data Actors. Honest pricing, real engineering, zero fine print.
# Actor input Schema ## `domains` (type: `array`): Root domains to query against crt.sh, e.g. example.com. One crt.sh query per domain. ## `maxCertsPerDomain` (type: `integer`): Hard cap on rows emitted per domain — the egress guard. crt.sh can return tens of thousands of records for a large domain. ## `deduplicateSubdomains` (type: `boolean`): When true, emit one row per unique subdomain (keeping the newest certificate's metadata). When false, emit one row per raw certificate/SAN entry. ## `includeExpired` (type: `boolean`): When false, drop certificates whose validity end date is in the past. ## `proxyConfiguration` (type: `object`): Apify Proxy configuration. crt.sh is a public keyless JSON API with no anti-bot behavior, so proxying is OFF by default: measured 2026-09-05, a proxied cloud fetch moved ~17 KB/s and timed out on gitlab.com's 1.4 MB response, while a direct fetch of the same URL took 4.7s. Enable it only if you need a specific exit IP. ## Actor input object example ```json { "domains": [ "github.com", "gitlab.com" ], "maxCertsPerDomain": 200, "deduplicateSubdomains": true, "includeExpired": true, "proxyConfiguration": { "useApifyProxy": false } } ``` # Actor output Schema ## `datasetItems` (type: `string`): All dataset items as JSON. ## `datasetItemsCsv` (type: `string`): Same data exported to CSV. ## `datasetView` (type: `string`): Open the run dataset in the Console. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "domains": [ "github.com", "gitlab.com" ], "maxCertsPerDomain": 200, "deduplicateSubdomains": true, "includeExpired": true, "proxyConfiguration": { "useApifyProxy": false } }; // Run the Actor and wait for it to finish const run = await client.actor("devilscrapes/crt-sh-certificate-transparency-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "domains": [ "github.com", "gitlab.com", ], "maxCertsPerDomain": 200, "deduplicateSubdomains": True, "includeExpired": True, "proxyConfiguration": { "useApifyProxy": False }, } # Run the Actor and wait for it to finish run = client.actor("devilscrapes/crt-sh-certificate-transparency-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "domains": [ "github.com", "gitlab.com" ], "maxCertsPerDomain": 200, "deduplicateSubdomains": true, "includeExpired": true, "proxyConfiguration": { "useApifyProxy": false } }' | apify call devilscrapes/crt-sh-certificate-transparency-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,devilscrapes/crt-sh-certificate-transparency-scraper" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/Qa3R43ObTCWdKdzlp/builds/TdezhGQSEq1MywYEj/openapi.json