OptiRefine TikTok Shop Product & Trend Tracker (v0.3.1 private pilot)

New: SandBase keyword discovery adapter with strict one-request-per-keyword behavior, offline tests and a clear unknown-response error. This build has not been tested with an authenticated SandBase V2 response or deployed to Apify from this environment. The provider may change its response schema.

SandBase quick start

1. Revoke any previously shared key; set a fresh SANDBASE_API_KEY in Apify Source > Environment variables, marked secret.
2. Deploy this whole project to the existing Actor, not only the input schema.
3. Configure the example examples/sandbase-private-pilot.json; confirm rights only if they apply.
4. Select Build (for Web IDE) or apify push (CLI), then run privately.
5. View output Dataset and RUN_SUMMARY in key-value store.

SandBase has no dollar-spend cap in this Actor: sandbaseMaxRequests limits calls per run, not total account usage. Verify the free-call eligibility, plan and terms before repeated requests, and get redistribution approval for commercial publishing. Only keyword search (US) is supported in SandBase mode, first page only; ID lookup is not implemented for SandBase.

Run npm install --include=dev && npm run build && npm test after installing dependencies. Offline-only tests without Apify dependencies: npm run test:offline when TypeScript is installed.

See docs/DEPLOY_CURRENT_FIX.md for the v0.2.0-to-v0.3.x deployment mismatch and precise deployment instructions, plus docs/SANDBASE_SETUP.md.

---

OptiRefine — TikTok Shop Product & Price Trend Tracker

Historical v0.2.0 notes (retained for compatibility):

Enter product keywords or exact product IDs and analyze product snapshots, prices, sellers, and disclosed sold indicators. This release adds a permission-gated connection to an upstream Apify Actor, while preserving authorized dataset imports and historical comparisons.

Important: This is not an independently authorized TikTok Shop scraper. The live mode calls a third-party Apify Actor. That Actor's Store listing does not automatically grant OptiRefine the rights to redistribute or sell its data, nor does it supersede TikTok's terms. Only enable live mode if your use is permitted. Get written authorization before publishing commercial redistribution. No authorized real-world run, licensing review, or cloud smoke test has been completed for this release.

What's new

• sourceMode: "externalActor" delegates discovery to the fixed source openclawai/tiktok-shop-scraper using documented shop_search / shop_product inputs.
• mode: "discover" accepts search keywords; mode: "products" accepts exact numeric productIds when external mode is selected.
• upstreamMaxCostUsd (default $0.25, max $5) passed to Actor.call as a per-upstream-run spending cap. The upstream run may incur costs even if the downstream tracker fails.
• Safely maps documented item_type: "product" records, skipping hints, errors, reviews, duplicate or invalid rows; excludes the original raw object.
• Existing normalized dataset mode and snapshot trend engine remain intact. No credible daily sales, revenue, or trend score can be derived from a single unverified sales indicator.
• sourceDatasetId and previousDatasetId are Apify storage pickers declaring READ scope to support limited-permission runs.

Configure a private pilot on Apify

1. Confirm your rights to run the upstream Actor and to use its output for the intended purpose; commercial resale requires appropriate authorization from relevant rights holders. Leave the rights checkboxes unchecked until you actually have this permission. No checkbox creates rights by itself.
2. Extract this project, then in its root folder run npm install --include=dev, npm run test:core, npm run build, and apify push. You are updating the existing optirefine/tiktok-shop-trend-tracker if authenticated into the same OptiRefine account.
3. Open the Actor's Input tab and select Live search via third-party Actor. Select Discover products by keyword and enter a search term such as skincare.
4. Set maxProducts to 5 for the first permitted run, upstreamMaxCostUsd to 0.25, and select the two data-rights confirmation boxes only when accurate. The example files intentionally leave these false.
5. Run privately, open the RUN_SUMMARY record and product Dataset. The summary should contain upstreamRunId, upstreamDatasetId, returned/filtered counts, and any normalization errors.
6. Review the upstream run's actual spend in Apify. It is separate from any charge to customers for this tracker. The cap limits upstream charges, not the full cost of the surrounding workflow.
7. To compare historical snapshots, run the same query with the same watchKey at least 24 hours later, select monitor, and choose the prior OptiRefine tracker run dataset in previousDatasetId. A single run has insufficient_history; momentum requires 3 comparable dated observations and trustworthy consistent counters.

Basic live input format

examples/live-search.json is ready to edit, but dataRightsConfirmed and upstreamReuseConfirmed are intentionally false. Do not set them to true unless the statements are true.

{
  "sourceMode": "externalActor",
  "mode": "discover",
  "market": "US",
  "keywords": ["skincare"],
  "maxProducts": 5,
  "upstreamMaxCostUsd": 0.25,
  "watchKey": "my-us-products",
  "dataRightsConfirmed": false,
  "upstreamReuseConfirmed": false
}

Use your own existing authorized dataset (no upstream call)

{
  "sourceMode": "authorizedDataset",
  "mode": "products",
  "market": "US",
  "sourceDatasetId": "YOUR_EXISTING_PRODUCT_DATASET_ID",
  "dataRightsConfirmed": true,
  "maxProducts": 20,
  "watchKey": "daily-watch"
}

Use the dataset picker in Apify, rather than typing an Actor or run ID. The source feed should follow examples/feed-records.json: productId must be a string, priceDecimal must be a decimal string, and sourceProvider must consistently identify a provider for comparisons.

Source limitations and billing

• The upstream source's README describes mode: "shop_search", queries, region, and maxItems, with advertised per-result pricing. Its implementation may change without notice. Input/output contracts were checked against public documentation, not against a real authenticated run.
• We only process item_type: "product" and item_id stored as a string. Known output fields include title, price, currency, sold_count, rating, and cover_url. Missing details remain null; the raw blob and other record types are not propagated.
• Sold count is labeled a displayed, unverified counter with unknown period and possible rounding. We do not infer daily sales, sales velocity, or GMV from it. Historical price comparisons can still be informative when price/currency/variants match.
• All displayed times are UTC. Data sources may fail due to upstream changes or account limitations; check the upstream run ID in the error or summary.
• Actor.pushData(product, "product-result") is inherited from v0.1. Before monetizing, configure and verify the product-result pay-per-event event in Apify Console and perform a charging-limit test. Upstream costs are additional to the pricing event.
• Supported market in this release is US only. The upstream Actor may claim others, but that does not mean our normalized schema supports them.
• Product keys are market + product ID. Comparisons use your chosen watchKey, tenant scope, provider identity, currency, and compatible timestamps. Select previous run results only from your own authorized history.
• The upstream provider is a fixed allowlist value, not an arbitrary user-supplied Actor ID; this reduces accidental delegation to unrelated Actors.

Commands

npm install --include=dev
npm run test:core
npm run build
apify validate-schema
apify push

Docker installs development dependencies before compiling TypeScript, then prunes them. This corrects the prior tsc: not found cloud build error.

Validation and deployment limitations

The project has local offline contract tests for input gating, mocked upstream product mapping, normalization, history, and comparisons. We have not performed a live upstream call, an Apify cloud run, verified API spend, or confirmed written resale rights. Don't advertise it as commercially ready yet.

Technical layout

• src/core/input.ts: mode/source/permission/cost/input validation
• src/sources/openClawMapper.ts: pure upstream input/output translation
• src/sources/externalActor.ts: downstream invocation and dataset reads, with testable injection seam
• src/sources/dataset.ts: authorized feed and historical dataset reads
• src/core/normalize.ts, src/core/history.ts: normalized snapshots and conservative trend math
• src/main.ts: orchestration, output, billing summary
• .actor/: Apify actor metadata, input/dataset/output schemas
• tests/: deterministic offline contract tests

Release blockers for public monetized listing

1. Permission to commercially redistribute or use upstream output in a publicly accessible paid Actor.
2. Live run verifying source schema, returned fields, actual US coverage, and output quality.
3. Test limited-permission Actor-to-Actor calls and resulting dataset access on the deployed runtime.
4. Verify upstream price, platform expenses, PPE price configuration, contribution margin and charge-limit behavior.
5. Verify no synthetic or vendor raw/private record leaks and that previous-run histories remain tenant-isolated.

References: Apify Actor.call, Apify Actor permissions, upstream Actor.