First public release. Everything below 1.0 was pre-release development; 0.4 was never deployed.
People, organisations, IP geolocation, and WHOIS history — closes the TruTrace spec gaps.
-
Named people and organisations extracted from archived pages, with roles. Two
layers, neither tied to a platform:
- Declared — schema.org JSON-LD and microdata (
Person, Organization, founder,
employee, worksFor, jobTitle), <meta name="author">, rel="author", and the
copyright footer (© 2015 Theranos, Inc.). The page states the type; nothing is guessed.
- Named-entity recognition — a multilingual token-classification model
(
bert-base-multilingual-cased-ner-hrl, int8, 173MB, baked into the image) reads the
rendered page's innerText line by line. A job title on the same visible line becomes
the person's role ("Geschäftsführer: Dirk Hünten, Michael Knippel" labels both), an
organisation on the same line becomes worksFor, and the line itself is kept as a
citable snippet. Lines repeated across a site's pages (header, nav, footer) are
inferred once and replayed as sightings, so cost scales with unique text, not pages.
A first version used capitalised-word regexes plus three word blocklists. It reached 4
people / 0 false positives on the test WordPress site, then emitted "Kosten", "Buch" and
"Rabatt" as people on the first German page it met — every blocklist entry was a patch
for one site that guaranteed a different failure on the next. Bake-off across a WordPress
blog, a hand-built Rails site and a German Impressum (13 ground-truth entities): rule-based
compromise missed every non-Western name; English bert-base-NER scored 12/13 but leaked
6 of 7 known false positives; the multilingual model scored 12/13 and leaked 1 (a lone first
name, removed by the two-word rule). The one miss was declared in JSON-LD. Filters that
remain are shape and frequency only: two words minimum for a person, score ≥ 0.9, an
organisation found on a Title-Case-only line is a menu label unless it is a ≥2-word legal
entity, and worksFor is taken only from short byline lines, never from biography prose.
Verified end-to-end on theranos.com (dead corporate site, 2013–2015 archive) with no
site-specific code: the full board — Elizabeth Holmes [Founder], Donald L. Lucas
[Chairman], Robert B. Shapiro, Channing Robertson, Peter Thomas, William K. Bowes — plus
the NPR host who interviewed Holmes, correctly left as a mention.
-
relation: site | mention on every person and organisation, list sorted site-first.
Correct NER on a blog about college admissions returns fifty universities; they are real
organisations and real noise. site means declared in structured data, or (people) seen
on 3+ distinct pages or carrying a job title, or (organisations) present on at least half
the pages the way a footer name is. pagesSeenOn is exposed so the threshold is auditable.
snippet carries the visible line each entity was read from.
-
Cost measured on the platform, not just the laptop. Per-page timers log fetch /
render / extraction / entity time, summed over the 3 parallel workers. On an M-series
laptop the model was ~3% of a 30-page quick run (people+orgs 6s of 77s). The first live
run on Apify at the 2048MB minimum told a different story: 531s total, people+orgs 679s -
1.47s per line against 12ms locally. Cause: onnxruntime sizes its thread pool from the
HOST's core count, the container's cgroup quota is invisible to it, and a dozen threads
fought over a half-core share while also starving Chromium (render 346s). Fix: Apify gives
one CPU per 4096MB and publishes APIFY_MEMORY_MBYTES, so intraOpNumThreads is
round(memory / 4096), minimum 1; unset locally, nothing changes. Same run, same 2048MB:
182s total, people+orgs 75s, render 162s, 0.10 compute units (was 0.30). For
comparison, 4096MB without the fix: 275s, 0.31 CU - doubling memory halved the time at
the same cost; fixing the threads cut the cost. Output was identical across all three runs
and the laptop (same 40 contacts, same 4 site-level entities). Chromium rendering is now
the largest CPU cost; archive.org fetch is 56s and outside our control.
-
Output layout: the contacts table is the dataset. One row per contact per source, with
the URL and capture date that produced it; people and organisations also carry role,
worksFor, relation and the evidence snippet. One summary row per domain closes each
block with the coverage verdict, so a run is never an empty table. The full report is the
run's OUTPUT record (Console's Output tab) and REPORT-<domain>. The previous layout put
the report in the dataset and the contacts in a separate named dataset, so the Store preview
showed one JSON blob per domain and the evidence table lived elsewhere. Now every output is
inside the run's own default storages, which is the contract the Actor's limited-permissions
declaration is meant for, the Store preview is the contacts table itself, and pay-per-result
maps directly onto "one row per contact".
-
Store listing is deployed from the repo. apify push never updates title, description,
SEO fields, categories, permissions or the example input on an existing Actor, so those were
empty on the platform. scripts/store-metadata.mjs runs after every deploy, validates the
portfolio shipping rules (title 63, seoTitle 60, seoDescription 200, description 300, no
em dashes, 3 categories max) and PUTs only what differs. Logo lives at .actor/logo.svg /
.actor/logo.png; the icon field is not writable through the API and is uploaded once in
Console.
-
Pricing: pay-per-event, sized from measured cost. Developer cost is $0.20 per compute
unit (Apify's FREE/BRONZE rate; measured $0.205 across five platform runs): a Quick scan
costs $0.02, a Standard scan $0.04. The lesson from the portfolio's nmap incident (per-record
pricing only, no floor, -769% margin) is that the per-run fee must cover the compute
ceiling, so the run defaults are set to 2 GB and 1800 s (ceiling 1 CU = $0.20) and the
apify-actor-start event, which Apify bills once per GB, is $0.125: $0.25 per scan at the
defaults, of which the developer keeps 80% = $0.20, break-even even on a run that hits the
timeout with nothing found. Rows are charged from code with two events so a corporate
archive naming 140 universities is not billed like 140 contacts: contact $0.02 (email,
phone, a person or organisation that is part of the site, or any registration, WHOIS
history, certificate or code source) and mention $0.002 (a name that only appears in the
content). The summary row is free and the synthetic dataset-item event is deliberately not
priced. Under pay-per-event, Actor.pushData(rows, event) charges within the user's spend
limit and writes only what was charged; the summary row says so if that happens. Three
Standard scans measured on the platform (4 GB, then priced at the 2 GB default): a small
dead site, 6 contacts + 36 mentions, $0.44 to the user on $0.044 of compute; a sparse
result, 3 contacts, $0.31 on $0.046; a large corporate archive, 231 rows, 47 contacts +
183 mentions, $1.56 on $0.038. Developer profit 72-78% of price, in line with the
portfolio. Memory barely moves compute (4 GB finishes in half the time for about the same
CU) but doubles the per-GB scan fee, which is why 2 GB is the default.
Applied by CI on first deploy (store.json apply: true); later changes go through
Console because Apify allows one pricing change per 30 days. Paid test on the platform:
chargedEventCounts came back apify-actor-start: 2, contact: 6, mention: 34 for a run
that wrote 6 contact rows, 34 mention rows and one free summary row, $0.44 to the user.
Note for the next person who tests this: the counts settle a minute or so after the run
finishes; reading them the instant the run ends shows zeros for the custom events. The run
logs the pricing view it received at startup, so that question is answerable from the log.
-
Pricing-change rules, established against the API. The pricing history is append-only
in time, records can never be removed or edited, and only one future-dated record may exist
at a time. Increases, new paid events and model changes need 14 days' notice and are limited
to once a month; decreases and description edits take effect immediately. On this private
Actor an increase was accepted 50 minutes after the first set, where the portfolio's public
Actors were refused 11 days after theirs, so the monthly lock appears to attach to public or
in-use Actors; Apify does not document that. The test left one scheduled record, contact
$0.021 from 2026-09-25, that the API will not let anyone cancel; the deploy workflow runs
once on that date and the metadata script appends the decrease back to $0.02. The script
now reconciles decreases from store.json on every deploy and never automates an increase.
-
Quick is the default depth; published to the Store. Apify's daily quality check runs
every public Actor with its default input and requires it to finish within 5 minutes, or
after 3 failing days the Actor is marked under maintenance. The default input (theranos.com
at Standard) took 351 s at the 2 GB default, so the pre-selected depth is now Quick: about
30 archived pages in about 3 minutes, which on theranos.com still returns the founder and
the board. Standard and Deep are one click away. Publication (isPublic) is declared in
store.json and applied by the same CI step as every other listing field.
-
Nothing waits unbounded any more. A Quick run on theranos.com hung for 15 minutes on
the platform at 0% CPU after "30 sampled": a promise that never settled, with every worker
queued behind it, heading for the 30-minute timeout, a charged start fee and no report.
Two candidates fit the idle silence, and both are closed: page.content() and
page.evaluate() had no timeout (a wedged renderer holds them forever), and the archive
limiter honoured Retry-After up to 600 s without logging. Now every Playwright call is
bounded, two consecutive render failures relaunch the browser (killing the process if
close() hangs), each page task has a 150 s deadline and can no longer take the stage
down with it, the limiter announces any wait over 15 s and caps backoff at 120 s, and the
archive stage has a time budget per depth (Quick 180 s, Standard 600 s, Deep 1200 s;
Quick was 120 s for one build and clipped 4 of 30 pages on a healthy heavy site).
When the budget trips, the unfetched pages are counted in old_pages.skippedForTime, the
section is marked incomplete with "not absent, not read yet", and the run still finishes
with its summary row and report.
-
License is MIT in both LICENSE and package.json (was Apache-2.0 in the latter).
-
IP geolocation via ip-api.com (free, no key). Every historical IP from passive DNS
is now auto-enriched with country, city, ISP, org and AS number using the free batch
endpoint (up to 100 IPs in a single request). Zero config — runs automatically.
-
SecurityTrails integration (BYOK, free tier: 2500 queries/month). New optional
securityTrailsApiKey input field. Free signup at securitytrails.com, no credit card.
Provides dated DNS history (A and NS records with first/last seen) and WHOIS history
with full registrant details going back to 2008. Contacts from WHOIS records are fed
into the ContactStore with provenance.
-
WHOIS history via Whoxy API (BYOK). New optional whoisHistoryApiKey input field.
Costs $0.005 per lookup, no monthly fee, data from November 2012. Pre-GDPR registrant
names, emails, phones, addresses and organisations extracted as high-confidence contacts.
Privacy-guard placeholders filtered out.
-
Contact breakdown includes people and orgs. The contacts section breakdown
now counts person and organization types alongside email and phone.
-
Source count: 14. 12 run with zero config, 2 optional BYOK sources (SecurityTrails
and Whoxy) unlock WHOIS history when keys are provided.
Checkbox-based section picker and restructured JSON output.
Five correctness fixes, every one found by running the actor on real domains rather
than by reading the code. Baseline output kept at
runs/2026-09-09_baseline_emoji-cafe.json so the change is diffable.
First build.
