Onion Directory & Contact Metadata Scraper

Collect public onion directory and contact metadata without crawling page bodies. The actor merges OnionTree, dark.fail and tor.taxi entries by v3 onion domain, keeps provenance, adds neutral categories, descriptions and PGP contact fields, and can measure liveness through a bundled Tor daemon. A separate search mode returns Ahmia index-result metadata only.

Scope and acceptable use

This actor is metadata-only by design. It reads public directory pages, OnionTree YAML records and Ahmia index results. It does not collect page text or HTML, credentials, API keys, passwords, cryptocurrency wallet addresses, marketplace listings, vendor/product data, breach data or illegal material. Search-mode result pages are never opened. Liveness checks only fetch the root of a directory entry's onion site to record status and title.

Use it for lawful security research, OSINT triage, directory quality checks and threat-intelligence enrichment. Tor use and onward use of the data may be regulated in your jurisdiction; you are responsible for complying with applicable law and with each source's terms. Do not use it to facilitate crime, to target individuals, or to build a marketplace index. The marketplace name denylist is a best-effort safety filter, not a guarantee.

Modes

Mode	What it does	Opens onion pages?
directories (default)	Aggregates OnionTree, dark.fail and tor.taxi, dedupes by v3 onion domain and merges provenance.	Only when verifyLiveness is true, one root GET per domain.
search	Queries the Ahmia index and returns title, URL, rank, snippet and last-seen claim.	Never.
byUrls	Reads directory/index pages you supply and extracts their v3 onion entries plus adjacent structured contact links.	Fetches the directory pages themselves, never the linked onion entries.

Legacy mode values are mapped in code: crawl becomes byUrls, searchAndCrawl becomes directories, and an unknown mode falls back to directories.

Output fields

Every record is omit-empty: a field the source genuinely lacks is absent, never null or an empty array.

Directory rows (recordType: "directoryEntry")

• name, title — entry name (title is the legacy alias)
• onionUrl, url, onionDomain — primary URL and its v3 host (url is the legacy alias)
• urls[] — all v3 mirror URLs for the entry (up to 10)
• links[], onionLinks[] — legacy aliases of urls[]
• description — directory description, capped at 1,000 characters
• categories[] — neutral OnionTree categories such as news, mail, securedrop
• source, sourceUrl — primary provenance (onionTree, darkFail, torTaxi, byUrls)
• alsoListedIn[] — other sources that listed the same onion domain
• sourceStatus — dark.fail's own claim (online, offline, unknown), never used as measured status
• status, httpStatus, pageTitle, lastCheckedAt — measured through Tor when verifyLiveness is true
• pgpFingerprint, pgpKeyId, contactEmail — OnionTree public-key contact metadata
• emails[], phones[], misc — structured contact values only (mailto:, tel:, explicit social profile links)
• scrapedAt — ISO-8601 UTC timestamp

Search rows (recordType: "searchResult")

query, rank, title, onionUrl, onionDomain, snippet, lastSeenClaim, totalResults, searchKeywordFound, source, sourceUrl, scrapedAt.

There is no cryptoAddresses field and no misc.apiKeys field. Both were removed for scope compliance and are never populated.

Input

Field	Type	Default	What it controls
mode	select	directories	directories, search or byUrls
source	select	onionTree	all, onionTree, darkFail or torTaxi (directories mode)
categories	string list	[]	OnionTree category filter; empty means all neutral categories
searchQueries	string list	[]	Ahmia queries (search mode; max 20)
maxResultsPerQuery	integer 1–1000	20	Per-query Ahmia cap
startUrls	URL list	[]	Directory/index pages (byUrls mode)
maxItems	integer 1–5000	100	Global emitted-record cap
maxPages	integer 1–50	5	Directory-page cap (byUrls mode)
verifyLiveness	boolean	true	Measure each unique onion domain through Tor
livenessTimeoutSecs	integer 5–60	12	Per-domain liveness timeout
includeUnreachable	boolean	true	Keep entries measured as timeout/unreachable
extractEmails	boolean	true	Structured emails from PGP user IDs and mailto: links
extractPhones	boolean	true	Phones from explicit tel: links only
extractSocialHandles	boolean	true	Handles from explicit social profile links only
extractPgpContacts	boolean	true	PGP fingerprint, key ID and contact email

No cookies, API keys or proxy configuration are required or accepted.

Example: daily directory run (cheapest)

{
    "mode": "directories",
    "source": "onionTree",
    "maxItems": 3,
    "verifyLiveness": false,
    "extractEmails": false,
    "extractPgpContacts": false
}

Example: merge all directories with liveness

{
    "mode": "directories",
    "source": "all",
    "categories": ["news", "securedrop", "mail"],
    "maxItems": 50,
    "verifyLiveness": true,
    "livenessTimeoutSecs": 12,
    "includeUnreachable": true
}

Example: Ahmia index metadata

{
    "mode": "search",
    "searchQueries": ["securedrop", "whistleblowing"],
    "maxResultsPerQuery": 10,
    "maxItems": 20
}

Example: read your own directory page

{
    "mode": "byUrls",
    "startUrls": [{ "url": "https://example.org/onion-directory" }],
    "maxPages": 2,
    "maxItems": 25
}

How liveness works

When verifyLiveness is true the actor starts the bundled Tor daemon and makes one GET request per unique onion domain, with a concurrency of four and a short jitter between requests. Example valid v3 entry: http://bbcnewsd73hkzno2ini43t4gblxvycyac5aw4gnv7t2rccijh7745uqd.onion. Results are classified as live, challenge (Cloudflare/DDoS-Guard or a 403/429), timeout or unreachable. After six consecutive transport failures the circuit breaker stops checking and marks the remaining entries unknown. A liveness failure never discards an already-built directory row.

Limitations

• Legacy v2 onions are dead in modern Tor. They are validated out and counted as skipped; only 56-character v3 addresses are emitted.
• Directory data is stale by nature. Many listed services are offline; status is only meaningful when you enable verifyLiveness.
• dark.fail and tor.taxi do not expose category tags, so the category filter applies to OnionTree rows only. The marketplace denylist for untagged sources is heuristic and covers known market, drug-tracking, exploit-market and darknet-market-guide brands.
• Ahmia's search form uses a rotating hidden token and a session cookie. If the token cannot be read, search returns a clear status message instead of looping.
• OnionTree's own website can be unreachable; the actor reads its public GitHub tarball mirror instead.
• No actor can guarantee that a third-party directory entry is safe, lawful or current. Verify before use.

FAQ

Do I need to install Tor or configure a proxy? No. The container includes Tor. The actor starts it only when liveness is enabled or a .onion directory page is supplied.

Does this actor crawl onion pages or collect page content? No. It reads directory pages and index-result metadata. A liveness check fetches only the root of an onion site to measure status and title; page bodies are not stored or emitted.

Why are some directory entries missing? Marketplace and vendor-tagged OnionTree records are excluded by scope, payment tags are stripped from categories, legacy v2-only records are skipped, and duplicate domains are merged into one row. A source that is blocked or unreachable is skipped without failing the run.

Why is a site marked challenge instead of live? The site answered with a Cloudflare/DDoS-Guard interstitial or a 403/429. It is reachable but not serving its normal page, so the actor reports the measured condition rather than assuming success.

What does sourceStatus mean? It is dark.fail's own claim about a site. It is kept separate from the measured status because directory claims can disagree with a live Tor check.

Why are PGP keys not included? Only the fingerprint, key ID and contact email are public contact metadata. The full public-key block is intentionally not stored.

Can I search more than one keyword? Yes. searchQueries accepts up to 20 queries; each query is one Ahmia request, and maxResultsPerQuery plus maxItems bound the output.

Why did an old task with crawl or search still run? Legacy input keys and mode names are mapped in code. crawl now means byUrls, searchAndCrawl now means directories, and extractCryptoAddresses/extractApiKeys are accepted but ignored with a status note because those data types are outside this actor's scope.

Are empty fields returned as null? No. Fields with no value are omitted at every depth.