Store Radar — Shopify new-store detector (Railway-ready)

Always-on service that watches the Certificate Transparency firehose, verifies which newly-issued certs belong to Shopify stores, and writes one CSV per day. A built-in web page lets you watch status and download the leads from anywhere — no SSH.

It needs a certstream firehose to read from. The public one (certstream.calidog.io) is offline most of the time, so you run your own. Total setup is two Railway services and about five minutes.

Architecture

┌─────────────────────────┐        ┌────────────────────────────┐
  │  certstream (service 1) │  ws    │  store-radar (service 2)   │
  │  reloading01/           │───────▶│  this repo                 │
  │  certstream-server-rust │        │  • DNS-verifies vs Shopify │
  │  (public Docker image)  │        │  • writes daily CSV        │
  └─────────────────────────┘        │  • serves status + CSV     │
                                      └────────────────────────────┘
                                                  │  https
                                                  ▼
                                         you: open the URL,
                                         download today.csv

Step 1 — Deploy the certstream firehose (service 1)

1. In your Railway project: New → Deploy from Docker Image.
2. Image: ghcr.io/reloading01/certstream-server-rust:latest
3. Under Settings → Networking, click Generate Domain (public URL). The server listens on 8080; set the public port to 8080 if asked.
4. Copy the public domain, e.g. certstream-production.up.railway.app.

That's the whole service — no code. (Optional: set a bearer token via its env vars if you don't want the stream open to the world. See its docs.)

Why a public domain instead of Railway's private network? Private networking is IPv6-only and the rust server binds IPv4 by default, so the public URL is the reliable path. The traffic is public CT data anyway.

Step 2 — Deploy this detector (service 2)

1. Push this folder to a GitHub repo (or use the Railway CLI).

2. Railway: New → Deploy from GitHub repo → pick it. It builds from the Dockerfile automatically (config in railway.json).

3. Variables → add:

   Variable	Value
   CERTSTREAM_URL	wss://certstream-production.up.railway.app/ (your service-1 domain, wss://)
   OUT_DIR	/data
   SMART_FILTER	1
   HTTP_VERIFY	0 (set 1 to also grab .myshopify.com handles — slower)
   NICHE	(optional) apparel,bespoke,jewel to log only matching domains
   WORKERS	200 (concurrent DNS-verification workers)
   DNS_THREADS	256 (getaddrinfo thread pool; raise for more DNS throughput)
   DNS_TIMEOUT	5 (seconds before a slow DNS lookup is given up on)
   NEW_ONLY	0 (set 1 to log only new domains, dropping cert renewals of existing stores)
   NEW_WINDOW_DAYS	7 (a domain is "new" if its earliest-ever cert is within this many days)
   KEEP_UNKNOWN	1 (in NEW_ONLY mode, still log rows whose age couldn't be determined; set 0 to drop them)
   CRTSH_CONCURRENCY	3 (max simultaneous crt.sh history lookups)

4. Settings → Networking → Generate Domain for this service too.

5. Settings → Volumes → New Volume, mount path /data. (Railway disks are ephemeral; the volume keeps your CSVs across restarts and redeploys.)

Step 3 — Use it

Open the detector's public URL:

• / — live status: certs seen, stores found, finds/min, recent hits (now with a country column). Auto-refreshes.
• /today.csv — download today's leads.
• /shopify_new_stores_YYYY-MM-DD.csv — any past day (listed on the status page).
• /stores.json — last 200 finds as JSON (handy to wire into your pipeline).
• /countries — deduped leads split by country. One downloadable CSV per country plus a combined deduped file.

Deduped, country-split leads

The raw daily CSVs log every detected host, so the same store often appears several times: bellemo.shop, www.bellemo.shop, account.bellemo.shop. The organizer collapses every host to its registrable domain (so those three become the single store bellemo.shop) and splits the result by country.

Country comes from the domain's TLD: country-code TLDs (.co.uk → UK, .de → Germany, .com.au → Australia, .ca → Canada …) map to a country; generic TLDs (.com, .shop, .store, .online …) and vanity ccTLDs (.co, .io, .me …) go to a GENERIC bucket. (GeoIP on the resolved IP can't help — every store points at Shopify's own IP block, so the IP only tells you where Shopify is.)

From the web UI: open /countries, click Rebuild from raw CSVs, then download per-country files (/by-country/UK.csv, /by-country/AU.csv, …) or the whole deduped set (/deduped.csv).

From the command line:

python organize_stores.py                 # organize every CSV in ./out
python organize_stores.py --in-dir ./out --out-dir ./out/organized
python organize_stores.py "out/200000+ stores.csv"   # one specific file
python organize_stores.py --colombia-real # treat .co/.io/.me as their real country

Output (under --out-dir, default ./out/organized):

Columns:

registrable_domain, country_code, country, seen_host, subdomains, detected_at, myshopify_handle, resolved_ip, type, signal, freshness

Run locally (to test before deploying)

pip install -r requirements.txt

# Point at your own certstream, or try the public one (often down):
export CERTSTREAM_URL="wss://YOUR-certstream-host/"
export OUT_DIR="./out"
python store_radar_service.py
# open http://localhost:8080

Run certstream locally too if you want a fully local stack:

docker run -d -p 8080:8080 ghcr.io/reloading01/certstream-server-rust:latest
export CERTSTREAM_URL="ws://localhost:8080/"

What it catches (and what it doesn't)

• Strong: stores that connect a custom domain and enable managed SSL — your best leads, caught within seconds of going live.
• Weak: free *.myshopify.com stores often sit behind a wildcard cert and won't always appear; Cloudflare-fronted stores won't resolve to Shopify's IPs.
• SMART_FILTER narrows to Let's-Encrypt + few-SAN certs (the managed-SSL pattern) to keep DNS volume sane. Turn it off to check everything (much heavier).
• This is a high-signal daily stream, not a complete census of every new store.

New vs renewal

The CT firehose fires on certificate issuance, which mixes brand-new stores with existing stores whose SSL cert just auto-renewed. To tell them apart, each confirmed custom-domain store is checked against crt.sh: we take the earliest certificate ever issued for that domain and classify it.

• new — earliest cert is within NEW_WINDOW_DAYS (default 7) → likely a store that just came online.
• renewal — earliest cert is older → an existing store renewing SSL.
• unknown — crt.sh lookup failed, timed out, or returned nothing. crt.sh is slow and rate-limited, so this happens; *.myshopify.com (cert-SAN) rows are always unknown because wildcard certs make crt.sh unreliable for them.

Set NEW_ONLY=1 to write only new (and unknown, unless KEEP_UNKNOWN=0) rows. Lookups run behind a small concurrency cap (CRTSH_CONCURRENCY) with a 10s timeout, one retry, and an in-memory cache, so they never starve DNS verification.

Caveat: this detects a new domain, not strictly a new Shopify store. A domain that existed on another platform (Wix, WooCommerce, a static site…) and later migrated to Shopify will have old certs and so look like a renewal, even though the store is brand new on Shopify. Treat new as high-confidence and renewal/unknown as worth a second look rather than a hard exclusion.

Output columns

detected_at, domain, myshopify_handle, resolved_ip, type, signal, freshness

type is myshopify (cert SAN) or custom; signal is how it was confirmed (cert, dns, or fingerprint); freshness is new, renewal, or unknown (see New vs renewal).