# Shopify New Store Scraper – Daily Leads by Pixels (`diplomatic_skull112/pixelspiece`) Actor Shopify store scraper that detects newly launched stores in real time. Get a deduplicated list of new Shopify store URLs with country and first-seen date, updated daily. Ideal for ecommerce lead generation, cold outreach, and Shopify app marketing. Export to CSV, JSON, or Excel. - **URL**: https://apify.com/diplomatic\_skull112/pixelspiece.md - **Developed by:** [Mohit Shyani](https://apify.com/diplomatic_skull112) (community) - **Categories:** Lead generation, E-commerce - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 2 bookmarks - **User rating**: 5.00 out of 5 stars ## Pricing from $5.00 / 1,000 store detecteds This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Store Radar — Shopify new-store detector (Railway-ready) Always-on service that watches the Certificate Transparency firehose, verifies which newly-issued certs belong to Shopify stores, and writes one CSV per day. A built-in web page lets you watch status and download the leads from anywhere — no SSH. It needs a **certstream firehose** to read from. The public one (`certstream.calidog.io`) is offline most of the time, so you run your own. Total setup is two Railway services and about five minutes. --- ### Architecture ```` ┌─────────────────────────┐ ┌────────────────────────────┐ │ certstream (service 1) │ ws │ store-radar (service 2) │ │ reloading01/ │───────▶│ this repo │ │ certstream-server-rust │ │ • DNS-verifies vs Shopify │ │ (public Docker image) │ │ • writes daily CSV │ └─────────────────────────┘ │ • serves status + CSV │ └────────────────────────────┘ │ https ▼ you: open the URL, download today.csv ```` --- ### Step 1 — Deploy the certstream firehose (service 1) 1. In your Railway project: **New → Deploy from Docker Image**. 2. Image: `ghcr.io/reloading01/certstream-server-rust:latest` 3. Under **Settings → Networking**, click **Generate Domain** (public URL). The server listens on **8080**; set the public port to 8080 if asked. 4. Copy the public domain, e.g. `certstream-production.up.railway.app`. That's the whole service — no code. (Optional: set a bearer token via its env vars if you don't want the stream open to the world. See its docs.) > Why a public domain instead of Railway's private network? Private networking is > IPv6-only and the rust server binds IPv4 by default, so the public URL is the > reliable path. The traffic is public CT data anyway. ### Step 2 — Deploy this detector (service 2) 1. Push this folder to a GitHub repo (or use the Railway CLI). 2. Railway: **New → Deploy from GitHub repo** → pick it. It builds from the `Dockerfile` automatically (config in `railway.json`). 3. **Variables** → add: | Variable | Value | |---|---| | `CERTSTREAM_URL` | `wss://certstream-production.up.railway.app/` (your service-1 domain, `wss://`) | | `OUT_DIR` | `/data` | | `SMART_FILTER` | `1` | | `HTTP_VERIFY` | `0` (set `1` to also grab `.myshopify.com` handles — slower) | | `NICHE` | *(optional)* `apparel,bespoke,jewel` to log only matching domains | | `WORKERS` | `200` (concurrent DNS-verification workers) | | `DNS_THREADS` | `256` (getaddrinfo thread pool; raise for more DNS throughput) | | `DNS_TIMEOUT` | `5` (seconds before a slow DNS lookup is given up on) | | `NEW_ONLY` | `0` (set `1` to log only *new* domains, dropping cert renewals of existing stores) | | `NEW_WINDOW_DAYS` | `7` (a domain is "new" if its earliest-ever cert is within this many days) | | `KEEP_UNKNOWN` | `1` (in `NEW_ONLY` mode, still log rows whose age couldn't be determined; set `0` to drop them) | | `CRTSH_CONCURRENCY` | `3` (max simultaneous crt.sh history lookups) | 4. **Settings → Networking → Generate Domain** for this service too. 5. **Settings → Volumes → New Volume**, mount path `/data`. (Railway disks are ephemeral; the volume keeps your CSVs across restarts and redeploys.) ### Step 3 — Use it Open the detector's public URL: - `/` — live status: certs seen, stores found, finds/min, recent hits (now with a country column). Auto-refreshes. - `/today.csv` — download today's leads. - `/shopify_new_stores_YYYY-MM-DD.csv` — any past day (listed on the status page). - `/stores.json` — last 200 finds as JSON (handy to wire into your pipeline). - `/countries` — **deduped leads split by country.** One downloadable CSV per country plus a combined deduped file. --- ### Deduped, country-split leads The raw daily CSVs log **every** detected host, so the same store often appears several times: `bellemo.shop`, `www.bellemo.shop`, `account.bellemo.shop`. The organizer collapses every host to its **registrable domain** (so those three become the single store `bellemo.shop`) and splits the result by country. Country comes from the domain's TLD: country-code TLDs (`.co.uk` → UK, `.de` → Germany, `.com.au` → Australia, `.ca` → Canada …) map to a country; generic TLDs (`.com`, `.shop`, `.store`, `.online` …) and vanity ccTLDs (`.co`, `.io`, `.me` …) go to a **GENERIC** bucket. (GeoIP on the resolved IP can't help — every store points at Shopify's own IP block, so the IP only tells you where Shopify is.) **From the web UI:** open `/countries`, click **Rebuild from raw CSVs**, then download per-country files (`/by-country/UK.csv`, `/by-country/AU.csv`, …) or the whole deduped set (`/deduped.csv`). **From the command line:** ```bash python organize_stores.py # organize every CSV in ./out python organize_stores.py --in-dir ./out --out-dir ./out/organized python organize_stores.py "out/200000+ stores.csv" # one specific file python organize_stores.py --colombia-real # treat .co/.io/.me as their real country ```` Output (under `--out-dir`, default `./out/organized`): | File | What | |---|---| | `all_stores_deduped.csv` | every unique store, with `registrable_domain`, `country_code`, `country`, and the `subdomains` that were collapsed | | `by_country/.csv` | one file per country (`UK.csv`, `AU.csv`, `DE.csv`, `GENERIC.csv`, …) | | `by_country/_index.csv` | country, store count, file — a summary | Columns: `registrable_domain, country_code, country, seen_host, subdomains, detected_at, myshopify_handle, resolved_ip, type, signal, freshness`. *** ### Run locally (to test before deploying) ```bash pip install -r requirements.txt ## Point at your own certstream, or try the public one (often down): export CERTSTREAM_URL="wss://YOUR-certstream-host/" export OUT_DIR="./out" python store_radar_service.py ## open http://localhost:8080 ``` Run certstream locally too if you want a fully local stack: ```bash docker run -d -p 8080:8080 ghcr.io/reloading01/certstream-server-rust:latest export CERTSTREAM_URL="ws://localhost:8080/" ``` *** ### What it catches (and what it doesn't) - **Strong:** stores that connect a custom domain and enable managed SSL — your best leads, caught within seconds of going live. - **Weak:** free `*.myshopify.com` stores often sit behind a wildcard cert and won't always appear; Cloudflare-fronted stores won't resolve to Shopify's IPs. - `SMART_FILTER` narrows to Let's-Encrypt + few-SAN certs (the managed-SSL pattern) to keep DNS volume sane. Turn it off to check everything (much heavier). - This is a high-signal daily stream, not a complete census of every new store. ### New vs renewal The CT firehose fires on certificate **issuance**, which mixes brand-new stores with existing stores whose SSL cert just auto-renewed. To tell them apart, each confirmed custom-domain store is checked against [crt.sh](https://crt.sh): we take the earliest certificate ever issued for that domain and classify it. - **`new`** — earliest cert is within `NEW_WINDOW_DAYS` (default 7) → likely a store that just came online. - **`renewal`** — earliest cert is older → an existing store renewing SSL. - **`unknown`** — crt.sh lookup failed, timed out, or returned nothing. crt.sh is slow and rate-limited, so this happens; `*.myshopify.com` (cert-SAN) rows are always `unknown` because wildcard certs make crt.sh unreliable for them. Set `NEW_ONLY=1` to write only `new` (and `unknown`, unless `KEEP_UNKNOWN=0`) rows. Lookups run behind a small concurrency cap (`CRTSH_CONCURRENCY`) with a 10s timeout, one retry, and an in-memory cache, so they never starve DNS verification. > **Caveat:** this detects a *new domain*, not strictly a new Shopify store. A domain > that existed on another platform (Wix, WooCommerce, a static site…) and later migrated > to Shopify will have old certs and so look like a `renewal`, even though the *store* is > brand new on Shopify. Treat `new` as high-confidence and `renewal`/`unknown` as worth > a second look rather than a hard exclusion. ### Output columns `detected_at, domain, myshopify_handle, resolved_ip, type, signal, freshness` `type` is `myshopify` (cert SAN) or `custom`; `signal` is how it was confirmed (`cert`, `dns`, or `fingerprint`); `freshness` is `new`, `renewal`, or `unknown` (see [New vs renewal](#new-vs-renewal)). # Actor input Schema ## `maxItems` (type: `integer`): Stop the run after this many stores are detected. Keeps your cost predictable. Set 0 for no limit (then the time limit ends the run). ## `maxRuntimeSecs` (type: `integer`): Hard stop after this many seconds (includes ~20s firehose warm-up). Set 0 for no time limit (then the store limit ends the run). If both limits are 0, the run defaults to 600s so it can never run forever. ## `niche` (type: `array`): Only keep domains whose name contains one of these words, e.g. apparel, jewel, print. Leave empty to get every detected Shopify store. ## `countries` (type: `array`): Keep only these country-code TLDs, e.g. UK, AU, DE, CA. Leave empty for all countries. Generic TLDs (.com, .shop, .store) are labelled GENERIC. ## `newOnly` (type: `boolean`): Cross-check crt.sh and keep only domains whose earliest certificate is recent (a genuinely new store, not an SSL renewal of an existing one). Higher signal, slower. ## `httpVerify` (type: `boolean`): Also load each store's homepage to confirm Shopify and capture its .myshopify.com handle. Richer data, slower. ## `workers` (type: `integer`): Concurrent DNS lookups. Higher drains bursts faster; 200 is a good default. ## Actor input object example ```json { "maxItems": 100, "maxRuntimeSecs": 600, "niche": [ "apparel", "jewel", "print" ], "countries": [], "newOnly": false, "httpVerify": false, "workers": 200 } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "niche": [ "apparel", "jewel", "print" ] }; // Run the Actor and wait for it to finish const run = await client.actor("diplomatic_skull112/pixelspiece").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "niche": [ "apparel", "jewel", "print", ] } # Run the Actor and wait for it to finish run = client.actor("diplomatic_skull112/pixelspiece").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "niche": [ "apparel", "jewel", "print" ] }' | apify call diplomatic_skull112/pixelspiece --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=diplomatic_skull112/pixelspiece", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Shopify New Store Scraper – Daily Leads by Pixels", "description": "Shopify store scraper that detects newly launched stores in real time. Get a deduplicated list of new Shopify store URLs with country and first-seen date, updated daily. Ideal for ecommerce lead generation, cold outreach, and Shopify app marketing. Export to CSV, JSON, or Excel.", "version": "0.1", "x-build-id": "D1vPQORPl2KyLk0ys" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/diplomatic_skull112~pixelspiece/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-diplomatic_skull112-pixelspiece", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/diplomatic_skull112~pixelspiece/runs": { "post": { "operationId": "runs-sync-diplomatic_skull112-pixelspiece", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/diplomatic_skull112~pixelspiece/run-sync": { "post": { "operationId": "run-sync-diplomatic_skull112-pixelspiece", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "maxItems": { "title": "Max stores to collect", "minimum": 0, "type": "integer", "description": "Stop the run after this many stores are detected. Keeps your cost predictable. Set 0 for no limit (then the time limit ends the run).", "default": 100 }, "maxRuntimeSecs": { "title": "Max runtime (seconds)", "minimum": 0, "type": "integer", "description": "Hard stop after this many seconds (includes ~20s firehose warm-up). Set 0 for no time limit (then the store limit ends the run). If both limits are 0, the run defaults to 600s so it can never run forever.", "default": 600 }, "niche": { "title": "Niche keywords (optional)", "type": "array", "description": "Only keep domains whose name contains one of these words, e.g. apparel, jewel, print. Leave empty to get every detected Shopify store.", "default": [], "items": { "type": "string" } }, "countries": { "title": "Countries (optional)", "type": "array", "description": "Keep only these country-code TLDs, e.g. UK, AU, DE, CA. Leave empty for all countries. Generic TLDs (.com, .shop, .store) are labelled GENERIC.", "default": [], "items": { "type": "string" } }, "newOnly": { "title": "Only brand-new domains", "type": "boolean", "description": "Cross-check crt.sh and keep only domains whose earliest certificate is recent (a genuinely new store, not an SSL renewal of an existing one). Higher signal, slower.", "default": false }, "httpVerify": { "title": "Fetch homepage to grab .myshopify handle", "type": "boolean", "description": "Also load each store's homepage to confirm Shopify and capture its .myshopify.com handle. Richer data, slower.", "default": false }, "workers": { "title": "DNS verification workers", "minimum": 10, "maximum": 800, "type": "integer", "description": "Concurrent DNS lookups. Higher drains bursts faster; 200 is a good default.", "default": 200 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```