# theHarvester Domain OSINT Collector (`automation-lab/theharvester-domain-osint-collector`) Actor

Collect source-attributed public certificate hostnames, resolved IPs, DNS aliases, and related URL candidates for recurring domain reconnaissance.

- **URL**: https://apify.com/automation-lab/theharvester-domain-osint-collector.md
- **Developed by:** [Stas Persiianenko](https://apify.com/automation-lab) (community)
- **Categories:** Other
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $5.52 / 1,000 finding extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## theHarvester Domain OSINT Collector

Collect public certificate-transparency subdomains, hosts, resolved IP addresses, DNS aliases, and related URL candidates for one or many domains.

This Actor provides a focused **theHarvester OSINT** workflow online without installing command-line tools. It queries the public Cert Spotter certificate-transparency index, deduplicates in-scope hostnames, enriches them with public DNS-over-HTTPS data, and exports source evidence with every finding.

Email discovery, credentials, paid OSINT services, brute-force enumeration, port scanning, screenshots, and vulnerability testing are intentionally excluded.

### What does this Actor do?

For each submitted root domain, the Actor:

1. queries Cert Spotter's public certificate issuance index;
2. extracts DNS names that belong to the submitted domain;
3. normalizes wildcard names and deduplicates hostnames;
4. preserves certificate IDs, issuer names, source query URL, and observed dates;
5. optionally resolves A, AAAA, and CNAME records through Cloudflare DNS-over-HTTPS;
6. derives HTTP and HTTPS URL candidates for downstream checks;
7. writes one typed dataset row per hostname.

The default dataset is ready for JSON, CSV, Excel, API, webhook, and scheduled-run integrations.

### Who is it for?

- **Security teams** building an authorized inventory of internet-facing names.
- **Bug bounty researchers** collecting passive, public reconnaissance before manual validation.
- **Attack-surface analysts** comparing certificate hostnames across recurring runs.
- **IT and cloud teams** finding legacy or unexpected DNS names associated with their domains.
- **Due-diligence teams** gathering source-attributed public infrastructure clues.
- **Data engineers** feeding normalized certificate and DNS findings into a pipeline.

This is passive discovery. It does not prove that a hostname is owned, active, reachable, safe, or vulnerable.

### Why use this online theHarvester OSINT workflow?

A full theHarvester installation covers many heterogeneous sources, some of which require accounts or API keys. This Actor takes a narrower, repeatable approach:

- no API keys or user accounts;
- no browser or residential proxy; certificate queries use bounded datacenter proxy rotation;
- explicit public-source attribution;
- deterministic root-domain scope;
- typed DNS and IP fields;
- bounded request retries and limits;
- multi-domain batch input;
- output designed for recurring comparisons.

The Actor is inspired by the domain-reconnaissance job associated with theHarvester. It is not affiliated with or a full hosted distribution of the upstream theHarvester project.

### What data is extracted?

| Field | Meaning |
| --- | --- |
| `domain` | Root domain supplied in the input. |
| `hostname` | Deduplicated root or subdomain from certificate data. |
| `isApex` | Whether the finding equals the root domain. |
| `seenAsWildcard` | Whether certificate evidence included a `*.` form. |
| `sources` | Public indexes that attributed the hostname. |
| `certificateEvidence` | Source query URL, certificate IDs, issuers, and date range. |
| `ipv4Addresses` | A records returned by public DNS. |
| `ipv6Addresses` | AAAA records returned by public DNS. |
| `cnameRecords` | CNAME targets returned by public DNS. |
| `relatedUrls` | Derived HTTP and HTTPS candidates; not a reachability claim. |
| `dnsResolved` | Whether A, AAAA, or CNAME data was returned. |
| `collectedAt` | ISO timestamp when the row was assembled. |

Certificate transparency is historical and can contain retired, wildcard, internal-looking, or no-longer-resolving names. Treat findings as leads for authorized analysis.

### Getting started

1. Open the Actor in Apify Console.
2. Enter one or more root domains such as `apify.com`.
3. Keep **Resolve DNS** enabled if you need IP and CNAME enrichment.
4. Set a per-domain and total finding limit appropriate for your workflow.
5. Click **Start**.
6. Open the default dataset and choose table, JSON, CSV, Excel, RSS, or XML output.
7. Schedule the Actor if you want recurring snapshots.

A small first run is the easiest way to verify that the data contract fits your pipeline.

### Input parameters

#### `domains`

Required array of 1–20 root domains. Use domain names without URL paths or wildcard prefixes.

Valid:

```json
["apify.com", "github.com"]
```

Invalid inputs such as `https://bad domain/path` fail closed instead of silently changing scope.

#### `resolveDns`

Defaults to `true`. Resolves A, AAAA, and CNAME records through Cloudflare's public DNS-over-HTTPS endpoint.

Disable it when you only need certificate evidence and want the lightest possible run.

#### `includeApex`

Defaults to `true`. Disable it to return only subdomains when the root domain also appears in certificate data.

#### `maxResultsPerDomain`

Maximum deduplicated findings saved for each domain. Allowed range: 1–1,000. Default: 100.

#### `maxItems`

Maximum findings saved across the entire run. Allowed range: 1–5,000. Default: 500.

#### `maxConcurrency`

Number of hostname DNS lookups processed concurrently. Allowed range: 1–20. Default: 5.

#### `requestTimeoutSecs`

Timeout for each upstream request. Allowed range: 5–60 seconds. Default: 20.

### Example input: one-time domain scan

```json
{
  "domains": ["apify.com"],
  "resolveDns": true,
  "includeApex": true,
  "maxResultsPerDomain": 25,
  "maxItems": 25
}
```

A current local verification of this exact workflow returned 20 typed hostname findings, 16 of which resolved through public DNS.

### Example input: recurring portfolio reconnaissance

```json
{
  "domains": ["apify.com", "github.com"],
  "resolveDns": true,
  "includeApex": true,
  "maxResultsPerDomain": 20,
  "maxItems": 40,
  "maxConcurrency": 5
}
```

Use Apify schedules to run this input daily or weekly, then compare datasets by `domain` and `hostname` in your own datastore or automation.

### Output example

The following shortened record reflects the current output shape:

```json
{
  "domain": "apify.com",
  "hostname": "affiliate.apify.com",
  "isApex": false,
  "seenAsWildcard": false,
  "sources": ["certspotter"],
  "certificateEvidence": [{
    "source": "certspotter",
    "certificateIds": ["15288112961", "16385737358"],
    "issuers": ["C=US, O=Let's Encrypt, CN=YR2"],
    "firstSeen": "2026-06-09T01:15:00.000Z",
    "lastSeen": "2026-08-09T03:25:05.000Z"
  }],
  "ipv4Addresses": ["216.150.16.129", "216.150.1.129"],
  "ipv6Addresses": [],
  "cnameRecords": ["cname.firstpromoter.com"],
  "relatedUrls": [
    "https://affiliate.apify.com",
    "http://affiliate.apify.com"
  ],
  "dnsResolved": true,
  "collectedAt": "2026-08-28T20:14:30.459Z"
}
```

The complete `certificateEvidence` object also includes the exact public source query URL.

### How much does it cost to collect domain OSINT findings?

The Actor uses pay-per-event pricing:

- **$0.005** once when a run starts;
- **$0.0092 BRONZE tier** for each hostname finding saved to the dataset;
- DNS records, IPs, URL candidates, and certificate evidence are included in the finding event and are not charged separately.

Illustrative BRONZE totals:

| Findings | BRONZE calculation |
| ---: | --- |
| 10 | $0.005 + (10 × $0.0092) |
| 25 | $0.005 + (25 × $0.0092) |
| 100 | $0.005 + (100 × $0.0092) |
| 500 | $0.005 + (500 × $0.0092) |

Higher usage tiers receive lower per-finding prices. Platform compute charges are covered by the pay-per-event model rather than billed as a separate Actor event.

### Recurring monitoring and comparison

Certificate data changes over time as organizations request and renew certificates. To monitor it:

1. create a Task with stable input;
2. set a daily or weekly schedule;
3. export each run's dataset to your database or cloud storage;
4. compare the composite key `domain + hostname`;
5. use `certificateEvidence.lastSeen` and DNS fields as context;
6. manually validate unexpected changes before taking action.

The Actor returns snapshots. It does not maintain history, generate diffs, send security alerts, or certify ownership on its own.

### Integration ideas

- Send new hostname rows to Slack after comparing them in Make or Zapier.
- Load CSV results into a spreadsheet for an authorized asset review.
- Store snapshots in BigQuery, Snowflake, PostgreSQL, or object storage.
- Join `ipv4Addresses` with an approved internal CMDB.
- Queue `relatedUrls` for a separate, authorized status-check Actor.
- Use webhooks to start downstream enrichment after a run succeeds.

Always preserve `sources` and `certificateEvidence` when moving findings into another system.

### Run through the Apify API with cURL

Replace `YOUR_TOKEN` with an Apify API token:

```bash
curl -X POST \
  "https://api.apify.com/v2/acts/automation-lab~theharvester-domain-osint-collector/runs?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "domains": ["apify.com"],
    "resolveDns": true,
    "maxItems": 25
  }'
```

To wait for completion and receive dataset items directly, use the synchronous dataset-items endpoint supported by the Apify API.

### Run with JavaScript

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/theharvester-domain-osint-collector').call({
  domains: ['apify.com'],
  resolveDns: true,
  maxItems: 25,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);
```

### Run with Python

```python
from apify_client import ApifyClient
import os

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("automation-lab/theharvester-domain-osint-collector").call(
    run_input={
        "domains": ["apify.com"],
        "resolveDns": True,
        "maxItems": 25,
    }
)

items = client.dataset(run["defaultDatasetId"]).list_items().items
print(items)
```

### Use with Apify MCP

Add the Actor to Claude Code:

```bash
claude mcp add --transport http apify \
  "https://mcp.apify.com?tools=automation-lab/theharvester-domain-osint-collector"
```

**Claude Desktop, Cursor, and VS Code setup**

Add the same remote MCP server URL in Claude Desktop, Cursor, or VS Code. A compatible JSON configuration is:

```json
{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=automation-lab/theharvester-domain-osint-collector"
    }
  }
}
```

Example prompts:

- “Collect up to 25 public certificate hostnames and resolved IPs for apify.com.”
- “Run passive domain reconnaissance for apify.com and github.com, then summarize unresolved names.”
- “Export the source evidence and CNAME records from this domain OSINT run.”

### Reliability and retry behavior

Cert Spotter is the only certificate source in v1 because it produced useful structured output during implementation discovery. The Actor applies bounded retries to transient HTTP statuses and request timeouts.

DNS failures are isolated per hostname and record type. A finding can still be useful when `dnsResolved` is false because certificate evidence is independent of current DNS state.

The run fails when the certificate source cannot be queried successfully for a submitted domain. It does not fabricate empty success output after a total upstream failure.

Cert Spotter requests use Apify datacenter proxy rotation to avoid coupling a run to one shared source quota. DNS-over-HTTPS requests stay direct. The Actor does not expose a proxy input because proxy choice does not expand the public data source.

### Limits and interpretation

- Cert Spotter coverage is not guaranteed to include every certificate or hostname.
- Certificate names can be stale, malformed, wildcarded, or no longer controlled by the same party.
- A DNS answer can change immediately after collection.
- `relatedUrls` are candidates derived from names, not proof of a responding web service.
- The Actor does not crawl discovered URLs.
- The Actor does not collect email addresses.
- The Actor does not test ports, services, vulnerabilities, or takeover conditions.
- The Actor does not query credentialed or paid sources.
- Results are limited and sorted deterministically by hostname after normalization.

Increase limits only when the resulting per-finding cost and dataset size suit your workflow.

### Responsible and legal use

Use the Actor only for domains and workflows you are authorized to investigate. Certificate-transparency and DNS records are public, but how you combine, retain, or act on them can still be governed by contracts, privacy law, security policy, and local regulation.

Do not use results to access systems, evade controls, harass people, or claim a vulnerability without validation. Follow coordinated disclosure rules and the scope of any bug bounty program.

This Actor performs passive collection and DNS resolution. Authorization for further scanning or interaction is your responsibility.

### Troubleshooting

#### Why did I receive fewer findings than my limit?

Limits are ceilings, not targets. The public source may contain fewer unique in-scope names, and wildcard or duplicate certificate entries collapse into one hostname row.

#### Why is `dnsResolved` false?

The certificate may be historical, the name may have no current A/AAAA/CNAME record, or the public DNS request may have failed. Check the run log and validate important names independently.

#### Why did the run fail instead of returning an empty dataset?

A total certificate-source failure is different from a successful query with no findings. The Actor fails closed so scheduled workflows can detect an upstream problem.

#### Can I discover emails with this Actor?

No. Email discovery is excluded from v1. Output focuses on certificate hostnames, DNS aliases, resolved IPs, and related URL candidates.

#### Does it run the complete theHarvester tool?

No. It delivers a focused online workflow for public certificate-transparency domain reconnaissance. It does not claim full source or CLI parity.

### Related Automation Lab Actors

- [Bulk DMARC Record Checker](https://apify.com/automation-lab/bulk-dmarc-record-checker) for auditing DMARC DNS policy across a domain list.
- [Sitemap URL Status Auditor](https://apify.com/automation-lab/sitemap-url-status-auditor) for checking URLs that your organization deliberately publishes in sitemaps.

Use separate Actors only where their scope and authorization match your workflow.

### FAQ

#### Is this useful for the importance of OSINT in recurring asset management?

Yes. Its primary value is repeatable, source-attributed passive discovery that can reveal changes worth validating. It is one input to an asset-management process, not a complete security assessment.

#### Can I submit URLs instead of domains?

No. Submit root domains only. The Actor deliberately rejects paths and unsupported input forms to keep scope predictable.

#### Are HTTP and HTTPS candidates checked?

No. Both candidates are derived from each certificate hostname. Use an authorized URL status workflow if reachability matters.

#### Can I disable DNS requests?

Yes. Set `resolveDns` to `false` to collect certificate evidence without A, AAAA, or CNAME enrichment.

#### Can I process multiple domains?

Yes. Submit up to 20 root domains and use `maxResultsPerDomain` plus `maxItems` to control output.

#### How do I export results?

Open the run's default dataset and choose JSON, CSV, Excel, XML, or RSS, or fetch items through the dataset API.

# Actor input Schema

## `domains` (type: `array`):

Root domains to investigate. Enter domain names only, without paths or wildcard prefixes.

## `resolveDns` (type: `boolean`):

Resolve A, AAAA, and CNAME records through Cloudflare's public DNS-over-HTTPS endpoint.

## `includeApex` (type: `boolean`):

Include the submitted root domain when it appears in certificate records.

## `maxResultsPerDomain` (type: `integer`):

Maximum deduplicated hostname findings saved for each domain.

## `maxItems` (type: `integer`):

Maximum findings saved across all submitted domains.

## `maxConcurrency` (type: `integer`):

Maximum hostnames resolved concurrently. Lower this if a public DNS service is rate-limiting requests.

## `requestTimeoutSecs` (type: `integer`):

Timeout for each certificate-transparency or DNS request before bounded retry/fallback handling.

## Actor input object example

```json
{
  "domains": [
    "apify.com"
  ],
  "resolveDns": true,
  "includeApex": true,
  "maxResultsPerDomain": 20,
  "maxItems": 20,
  "maxConcurrency": 5,
  "requestTimeoutSecs": 20
}
```

# Actor output Schema

## `dataset` (type: `string`):

Default dataset containing deduplicated domain reconnaissance findings.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "domains": [
        "apify.com"
    ],
    "resolveDns": true,
    "includeApex": true,
    "maxResultsPerDomain": 20,
    "maxItems": 20
};

// Run the Actor and wait for it to finish
const run = await client.actor("automation-lab/theharvester-domain-osint-collector").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "domains": ["apify.com"],
    "resolveDns": True,
    "includeApex": True,
    "maxResultsPerDomain": 20,
    "maxItems": 20,
}

# Run the Actor and wait for it to finish
run = client.actor("automation-lab/theharvester-domain-osint-collector").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "domains": [
    "apify.com"
  ],
  "resolveDns": true,
  "includeApex": true,
  "maxResultsPerDomain": 20,
  "maxItems": 20
}' |
apify call automation-lab/theharvester-domain-osint-collector --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,automation-lab/theharvester-domain-osint-collector"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/IxAi5UFA87EKfkJG2/builds/4wc4JgCbsmuPteIs7/openapi.json