theHarvester Domain OSINT Collector

Collect public certificate-transparency subdomains, hosts, resolved IP addresses, DNS aliases, and related URL candidates for one or many domains.

This Actor provides a focused theHarvester OSINT workflow online without installing command-line tools. It queries the public Cert Spotter certificate-transparency index, deduplicates in-scope hostnames, enriches them with public DNS-over-HTTPS data, and exports source evidence with every finding.

Email discovery, credentials, paid OSINT services, brute-force enumeration, port scanning, screenshots, and vulnerability testing are intentionally excluded.

What does this Actor do?

For each submitted root domain, the Actor:

1. queries Cert Spotter's public certificate issuance index;
2. extracts DNS names that belong to the submitted domain;
3. normalizes wildcard names and deduplicates hostnames;
4. preserves certificate IDs, issuer names, source query URL, and observed dates;
5. optionally resolves A, AAAA, and CNAME records through Cloudflare DNS-over-HTTPS;
6. derives HTTP and HTTPS URL candidates for downstream checks;
7. writes one typed dataset row per hostname.

The default dataset is ready for JSON, CSV, Excel, API, webhook, and scheduled-run integrations.

Who is it for?

• Security teams building an authorized inventory of internet-facing names.
• Bug bounty researchers collecting passive, public reconnaissance before manual validation.
• Attack-surface analysts comparing certificate hostnames across recurring runs.
• IT and cloud teams finding legacy or unexpected DNS names associated with their domains.
• Due-diligence teams gathering source-attributed public infrastructure clues.
• Data engineers feeding normalized certificate and DNS findings into a pipeline.

This is passive discovery. It does not prove that a hostname is owned, active, reachable, safe, or vulnerable.

Why use this online theHarvester OSINT workflow?

A full theHarvester installation covers many heterogeneous sources, some of which require accounts or API keys. This Actor takes a narrower, repeatable approach:

• no API keys or user accounts;
• no browser or residential proxy; certificate queries use bounded datacenter proxy rotation;
• explicit public-source attribution;
• deterministic root-domain scope;
• typed DNS and IP fields;
• bounded request retries and limits;
• multi-domain batch input;
• output designed for recurring comparisons.

The Actor is inspired by the domain-reconnaissance job associated with theHarvester. It is not affiliated with or a full hosted distribution of the upstream theHarvester project.

What data is extracted?

Field	Meaning
domain	Root domain supplied in the input.
hostname	Deduplicated root or subdomain from certificate data.
isApex	Whether the finding equals the root domain.
seenAsWildcard	Whether certificate evidence included a *. form.
sources	Public indexes that attributed the hostname.
certificateEvidence	Source query URL, certificate IDs, issuers, and date range.
ipv4Addresses	A records returned by public DNS.
ipv6Addresses	AAAA records returned by public DNS.
cnameRecords	CNAME targets returned by public DNS.
relatedUrls	Derived HTTP and HTTPS candidates; not a reachability claim.
dnsResolved	Whether A, AAAA, or CNAME data was returned.
collectedAt	ISO timestamp when the row was assembled.

Certificate transparency is historical and can contain retired, wildcard, internal-looking, or no-longer-resolving names. Treat findings as leads for authorized analysis.

Getting started

1. Open the Actor in Apify Console.
2. Enter one or more root domains such as apify.com.
3. Keep Resolve DNS enabled if you need IP and CNAME enrichment.
4. Set a per-domain and total finding limit appropriate for your workflow.
5. Click Start.
6. Open the default dataset and choose table, JSON, CSV, Excel, RSS, or XML output.
7. Schedule the Actor if you want recurring snapshots.

A small first run is the easiest way to verify that the data contract fits your pipeline.

Input parameters

domains

Required array of 1–20 root domains. Use domain names without URL paths or wildcard prefixes.

Valid:

["apify.com", "github.com"]

Invalid inputs such as https://bad domain/path fail closed instead of silently changing scope.

resolveDns

Defaults to true. Resolves A, AAAA, and CNAME records through Cloudflare's public DNS-over-HTTPS endpoint.

Disable it when you only need certificate evidence and want the lightest possible run.

includeApex

Defaults to true. Disable it to return only subdomains when the root domain also appears in certificate data.

maxResultsPerDomain

Maximum deduplicated findings saved for each domain. Allowed range: 1–1,000. Default: 100.

maxItems

Maximum findings saved across the entire run. Allowed range: 1–5,000. Default: 500.

maxConcurrency

Number of hostname DNS lookups processed concurrently. Allowed range: 1–20. Default: 5.

requestTimeoutSecs

Timeout for each upstream request. Allowed range: 5–60 seconds. Default: 20.

Example input: one-time domain scan

{
  "domains": ["apify.com"],
  "resolveDns": true,
  "includeApex": true,
  "maxResultsPerDomain": 25,
  "maxItems": 25
}

A current local verification of this exact workflow returned 20 typed hostname findings, 16 of which resolved through public DNS.

Example input: recurring portfolio reconnaissance

{
  "domains": ["apify.com", "github.com"],
  "resolveDns": true,
  "includeApex": true,
  "maxResultsPerDomain": 20,
  "maxItems": 40,
  "maxConcurrency": 5
}

Use Apify schedules to run this input daily or weekly, then compare datasets by domain and hostname in your own datastore or automation.

Output example

The following shortened record reflects the current output shape:

{
  "domain": "apify.com",
  "hostname": "affiliate.apify.com",
  "isApex": false,
  "seenAsWildcard": false,
  "sources": ["certspotter"],
  "certificateEvidence": [{
    "source": "certspotter",
    "certificateIds": ["15288112961", "16385737358"],
    "issuers": ["C=US, O=Let's Encrypt, CN=YR2"],
    "firstSeen": "2026-06-09T01:15:00.000Z",
    "lastSeen": "2026-08-09T03:25:05.000Z"
  }],
  "ipv4Addresses": ["216.150.16.129", "216.150.1.129"],
  "ipv6Addresses": [],
  "cnameRecords": ["cname.firstpromoter.com"],
  "relatedUrls": [
    "https://affiliate.apify.com",
    "http://affiliate.apify.com"
  ],
  "dnsResolved": true,
  "collectedAt": "2026-08-28T20:14:30.459Z"
}

The complete certificateEvidence object also includes the exact public source query URL.

How much does it cost to collect domain OSINT findings?

The Actor uses pay-per-event pricing:

• $0.005 once when a run starts;
• $0.0092 BRONZE tier for each hostname finding saved to the dataset;
• DNS records, IPs, URL candidates, and certificate evidence are included in the finding event and are not charged separately.

Illustrative BRONZE totals:

Findings	BRONZE calculation
10	$0.005 + (10 × $0.0092)
25	$0.005 + (25 × $0.0092)
100	$0.005 + (100 × $0.0092)
500	$0.005 + (500 × $0.0092)

Higher usage tiers receive lower per-finding prices. Platform compute charges are covered by the pay-per-event model rather than billed as a separate Actor event.

Recurring monitoring and comparison

Certificate data changes over time as organizations request and renew certificates. To monitor it:

1. create a Task with stable input;
2. set a daily or weekly schedule;
3. export each run's dataset to your database or cloud storage;
4. compare the composite key domain + hostname;
5. use certificateEvidence.lastSeen and DNS fields as context;
6. manually validate unexpected changes before taking action.

The Actor returns snapshots. It does not maintain history, generate diffs, send security alerts, or certify ownership on its own.

Integration ideas

• Send new hostname rows to Slack after comparing them in Make or Zapier.
• Load CSV results into a spreadsheet for an authorized asset review.
• Store snapshots in BigQuery, Snowflake, PostgreSQL, or object storage.
• Join ipv4Addresses with an approved internal CMDB.
• Queue relatedUrls for a separate, authorized status-check Actor.
• Use webhooks to start downstream enrichment after a run succeeds.

Always preserve sources and certificateEvidence when moving findings into another system.

Run through the Apify API with cURL

Replace YOUR_TOKEN with an Apify API token:

curl -X POST \
  "https://api.apify.com/v2/acts/automation-lab~theharvester-domain-osint-collector/runs?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "domains": ["apify.com"],
    "resolveDns": true,
    "maxItems": 25
  }'

To wait for completion and receive dataset items directly, use the synchronous dataset-items endpoint supported by the Apify API.

Run with JavaScript

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/theharvester-domain-osint-collector').call({
  domains: ['apify.com'],
  resolveDns: true,
  maxItems: 25,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Run with Python

from apify_client import ApifyClient
import os

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("automation-lab/theharvester-domain-osint-collector").call(
    run_input={
        "domains": ["apify.com"],
        "resolveDns": True,
        "maxItems": 25,
    }
)

items = client.dataset(run["defaultDatasetId"]).list_items().items
print(items)

Use with Apify MCP

Add the Actor to Claude Code:

claude mcp add --transport http apify \
  "https://mcp.apify.com?tools=automation-lab/theharvester-domain-osint-collector"

Claude Desktop, Cursor, and VS Code setup

Add the same remote MCP server URL in Claude Desktop, Cursor, or VS Code. A compatible JSON configuration is:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=automation-lab/theharvester-domain-osint-collector"
    }
  }
}

Example prompts:

• “Collect up to 25 public certificate hostnames and resolved IPs for apify.com.”
• “Run passive domain reconnaissance for apify.com and github.com, then summarize unresolved names.”
• “Export the source evidence and CNAME records from this domain OSINT run.”

Reliability and retry behavior

Cert Spotter is the only certificate source in v1 because it produced useful structured output during implementation discovery. The Actor applies bounded retries to transient HTTP statuses and request timeouts.

DNS failures are isolated per hostname and record type. A finding can still be useful when dnsResolved is false because certificate evidence is independent of current DNS state.

The run fails when the certificate source cannot be queried successfully for a submitted domain. It does not fabricate empty success output after a total upstream failure.

Cert Spotter requests use Apify datacenter proxy rotation to avoid coupling a run to one shared source quota. DNS-over-HTTPS requests stay direct. The Actor does not expose a proxy input because proxy choice does not expand the public data source.

Limits and interpretation

• Cert Spotter coverage is not guaranteed to include every certificate or hostname.
• Certificate names can be stale, malformed, wildcarded, or no longer controlled by the same party.
• A DNS answer can change immediately after collection.
• relatedUrls are candidates derived from names, not proof of a responding web service.
• The Actor does not crawl discovered URLs.
• The Actor does not collect email addresses.
• The Actor does not test ports, services, vulnerabilities, or takeover conditions.
• The Actor does not query credentialed or paid sources.
• Results are limited and sorted deterministically by hostname after normalization.

Increase limits only when the resulting per-finding cost and dataset size suit your workflow.

Responsible and legal use

Use the Actor only for domains and workflows you are authorized to investigate. Certificate-transparency and DNS records are public, but how you combine, retain, or act on them can still be governed by contracts, privacy law, security policy, and local regulation.

Do not use results to access systems, evade controls, harass people, or claim a vulnerability without validation. Follow coordinated disclosure rules and the scope of any bug bounty program.

This Actor performs passive collection and DNS resolution. Authorization for further scanning or interaction is your responsibility.

Troubleshooting

Why did I receive fewer findings than my limit?

Limits are ceilings, not targets. The public source may contain fewer unique in-scope names, and wildcard or duplicate certificate entries collapse into one hostname row.

Why is dnsResolved false?

The certificate may be historical, the name may have no current A/AAAA/CNAME record, or the public DNS request may have failed. Check the run log and validate important names independently.

Why did the run fail instead of returning an empty dataset?

A total certificate-source failure is different from a successful query with no findings. The Actor fails closed so scheduled workflows can detect an upstream problem.

Can I discover emails with this Actor?

No. Email discovery is excluded from v1. Output focuses on certificate hostnames, DNS aliases, resolved IPs, and related URL candidates.

Does it run the complete theHarvester tool?

No. It delivers a focused online workflow for public certificate-transparency domain reconnaissance. It does not claim full source or CLI parity.

Related Automation Lab Actors

• Bulk DMARC Record Checker for auditing DMARC DNS policy across a domain list.
• Sitemap URL Status Auditor for checking URLs that your organization deliberately publishes in sitemaps.

Use separate Actors only where their scope and authorization match your workflow.

FAQ

Is this useful for the importance of OSINT in recurring asset management?

Yes. Its primary value is repeatable, source-attributed passive discovery that can reveal changes worth validating. It is one input to an asset-management process, not a complete security assessment.

Can I submit URLs instead of domains?

No. Submit root domains only. The Actor deliberately rejects paths and unsupported input forms to keep scope predictable.

Are HTTP and HTTPS candidates checked?

No. Both candidates are derived from each certificate hostname. Use an authorized URL status workflow if reachability matters.

Can I disable DNS requests?

Yes. Set resolveDns to false to collect certificate evidence without A, AAAA, or CNAME enrichment.

Can I process multiple domains?

Yes. Submit up to 20 root domains and use maxResultsPerDomain plus maxItems to control output.

How do I export results?

Open the run's default dataset and choose JSON, CSV, Excel, XML, or RSS, or fetch items through the dataset API.