A/B Test Significance Calculator — Google Optimize Replacement

Drop-in replacement for Google Optimize's statistical significance workflow. Get p-values, confidence intervals, lift, statistical power, and required sample sizes — via API, no browser, no account, no monthly fee.

Why this exists: Google Optimize was shut down on September 30, 2023. Google pushed users toward paid enterprise solutions (GA4 + Optimize 360 partners, VWO+AB Tasty starting at $250+/mo). Open-source alternatives like GrowthBook and PostHog require self-hosting. The simple "paste variant data, get p-value" workflow that 80% of SMB users actually needed was never replaced.

This actor does that one thing, exceptionally well. Built for marketers, product managers, engineers, and data scientists who need answer "did my test win?" on demand — inside Zapier workflows, CI/CD pipelines, Slack bots, or notebooks.

🔑 Features

• Conversion-rate tests — two-sample z-test for proportions (signup rate, purchase rate, click-through rate)
• Continuous metrics — Welch's t-test for unequal variances (revenue per user, session length, page views)
• Multi-variant support — up to 6 variants with automatic Bonferroni correction for familywise error rate
• Full output — p-value, 95% CI, absolute + relative lift, statistical power achieved, per-variant stats
• Sample size calculator — tells you how many visitors per variant you need for a given MDE and power
• Winner flag + human recommendation — one-line "ship Variant A" or "keep testing" verdict
• Zero scraping, zero JS — pure Python scipy.stats. Runs in under 2 seconds. Lowest-compute actor in the fleet.

💼 Common Use Cases

• Post-test analysis — drop-in replacement for Google Optimize's significance display
• Zapier / Make.com workflows — auto-analyze Mixpanel/Amplitude test results nightly
• CI/CD integration — gate deployments on experimental success
• Slack bots — /significance variant_a=250/5000 variant_b=310/5000 → auto-response
• Notebooks / BI dashboards — trigger via API, render results alongside visualizations
• Marketing team Slack — daily digest of running test statuses
• Startup PMs — quickly sanity-check whether the "10% lift" from the latest test is real

📥 Input Example — Conversion Test

{
  "metricType": "conversion",
  "variants": [
    {"name": "Control", "visitors": 5000, "conversions": 250},
    {"name": "Variant A (new CTA)", "visitors": 5000, "conversions": 310},
    {"name": "Variant B (redesign)", "visitors": 5000, "conversions": 295}
  ],
  "alpha": "0.05",
  "power": "0.80",
  "minDetectableEffect": "0.05"
}

📥 Input Example — Continuous Metric

{
  "metricType": "continuous",
  "variants": [
    {"name": "Control", "n": 1000, "mean": 42.50, "std": 18.30},
    {"name": "Variant A", "n": 1000, "mean": 46.80, "std": 19.10}
  ],
  "alpha": "0.05"
}

📤 Output

{
  "metric_type": "conversion",
  "significance_level_alpha": 0.05,
  "alpha_bonferroni_corrected": 0.025,
  "num_variants": 3,
  "num_comparisons": 2,
  "control_variant": "Control",
  "variants_summary": [
    {"name": "Control", "visitors": 5000, "conversions": 250, "conversion_rate": 0.05},
    {"name": "Variant A", "visitors": 5000, "conversions": 310, "conversion_rate": 0.062}
  ],
  "comparisons": [
    {
      "variant": "Variant A (new CTA)",
      "vs": "Control",
      "significant": true,
      "p_value": 0.009234,
      "z_score": 2.6078,
      "lift_absolute": 0.012,
      "lift_relative": 0.24,
      "ci_95_lower": 0.003,
      "ci_95_upper": 0.021,
      "statistical_power": 0.84
    }
  ],
  "winner": "Variant A (new CTA)",
  "required_sample_size_per_variant": 6162,
  "recommendation": "Ship Variant A (new CTA). It beats Control with 24.0% relative lift at p<0.025 (Bonferroni-corrected)."
}

🐍 Python SDK Example

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("nexgendata/ab-test-calculator").call(run_input={
    "metricType": "conversion",
    "variants": [
        {"name": "Control", "visitors": 10000, "conversions": 520},
        {"name": "Variant A", "visitors": 10000, "conversions": 605}
    ]
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print("Winner:", item["winner"])
    print("Recommendation:", item["recommendation"])

🌐 cURL Example

curl -X POST "https://api.apify.com/v2/acts/nexgendata~ab-test-calculator/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "variants": [
      {"name": "Control", "visitors": 5000, "conversions": 250},
      {"name": "Variant A", "visitors": 5000, "conversions": 310}
    ]
  }'

🔗 Zapier / Make.com Integration

Perfect for automating test analysis. Trigger: "New row in Experiments Google Sheet" → Action: "Run Actor (A/B Test Calculator)" with variant data → Output: Post winner + lift to Slack, Notion, or email.

❓ FAQ

Q: What test is used for conversion metrics? Two-sample z-test for proportions with pooled standard error. Matches Google Optimize's "Relative improvement" calculation and what Evan Miller's calculator uses.

Q: What about continuous metrics (revenue, session length)? Welch's t-test with Welch-Satterthwaite degrees of freedom. Handles unequal variances, which is the realistic case for revenue data.

Q: How are multiple variants handled? All treatments are compared against the first variant (control). Alpha is Bonferroni-corrected by dividing by the number of comparisons to control familywise error rate.

Q: Is this Bayesian or frequentist? Frequentist. This matches Google Optimize's default behavior and what most teams already reason about (p-values, CIs). Bayesian support is on the roadmap.

Q: Does it work for sequential testing / peeking? Not natively — use only once per test, after test concludes. Sequential testing requires different statistical treatment (alpha spending, mSPRT). Coming in v1.1.

Q: How is this different from Evan Miller's free calculator? Same math, but API-first. Automate across many tests instead of copy-pasting into a web form. Great for portfolios of experiments.

💰 Pricing (Pay-Per-Event)

• Actor start: $0.005
• Test analyzed: $0.002

Typical run cost: $0.007 per test analyzed. Effectively free for individual use. At 1,000 tests/month you're still under $7.

🔗 Related NexGenData Actors

• Facebook Ads Library Scraper — measure ad performance against A/B groups
• Google Trends Scraper — trend-adjust your conversion rates
• SaaS Pricing Tracker — track competitor pricing during test windows

🚀 Apify Affiliate Program

New to Apify? Sign up with our referral link for free platform credits.

---

A Google Optimize replacement for marketers, PMs, and data scientists who miss the simple workflow. Built by NexGenData.