Pricing

from $2.00 / 1,000 test analyzeds

🧪 A/B Test Significance Calculator — Google Optimize Alt

Drop-in Google Optimize replacement. Calculate p-values, confidence intervals, lift, sample sizes, and winners for A/B tests. Two-proportion z-test + Welch's t-test + Bonferroni multi-variant. API-first, no account, $0.007/test.

Pricing

from $2.00 / 1,000 test analyzeds

Rating

0.0

(0)

Developer

Stephan Corbeil

Actor stats

Bookmarked

Total users

Monthly active users

12 days ago

Last modified

A/B Test Significance Calculator — Google Optimize Replacement

Drop-in replacement for Google Optimize's statistical significance workflow. Get p-values, confidence intervals, lift, statistical power, and required sample sizes — via API, no browser, no account, no monthly fee.

Why this exists: Google Optimize was shut down on September 30, 2023. Google pushed users toward paid enterprise solutions (GA4 + Optimize 360 partners, VWO+AB Tasty starting at $250+/mo). Open-source alternatives like GrowthBook and PostHog require self-hosting. The simple "paste variant data, get p-value" workflow that 80% of SMB users actually needed was never replaced.

This actor does that one thing, exceptionally well. Built for marketers, product managers, engineers, and data scientists who need answer "did my test win?" on demand — inside Zapier workflows, CI/CD pipelines, Slack bots, or notebooks.

🔑 Features

Conversion-rate tests — two-sample z-test for proportions (signup rate, purchase rate, click-through rate)
Continuous metrics — Welch's t-test for unequal variances (revenue per user, session length, page views)
Multi-variant support — up to 6 variants with automatic Bonferroni correction for familywise error rate
Full output — p-value, 95% CI, absolute + relative lift, statistical power achieved, per-variant stats
Sample size calculator — tells you how many visitors per variant you need for a given MDE and power
Winner flag + human recommendation — one-line "ship Variant A" or "keep testing" verdict
Zero scraping, zero JS — pure Python scipy.stats. Runs in under 2 seconds. Lowest-compute actor in the fleet.

💼 Common Use Cases

Post-test analysis — drop-in replacement for Google Optimize's significance display
Zapier / Make.com workflows — auto-analyze Mixpanel/Amplitude test results nightly
CI/CD integration — gate deployments on experimental success
Slack bots — /significance variant_a=250/5000 variant_b=310/5000 → auto-response
Notebooks / BI dashboards — trigger via API, render results alongside visualizations
Marketing team Slack — daily digest of running test statuses
Startup PMs — quickly sanity-check whether the "10% lift" from the latest test is real

📥 Input Example — Conversion Test

{
  "metricType": "conversion",
  "variants": [
    {"name": "Control", "visitors": 5000, "conversions": 250},
    {"name": "Variant A (new CTA)", "visitors": 5000, "conversions": 310},
    {"name": "Variant B (redesign)", "visitors": 5000, "conversions": 295}
  ],
  "alpha": "0.05",
  "power": "0.80",
  "minDetectableEffect": "0.05"
}

📥 Input Example — Continuous Metric

{
  "metricType": "continuous",
  "variants": [
    {"name": "Control", "n": 1000, "mean": 42.50, "std": 18.30},
    {"name": "Variant A", "n": 1000, "mean": 46.80, "std": 19.10}
  ],
  "alpha": "0.05"
}

📤 Output

{
  "metric_type": "conversion",
  "significance_level_alpha": 0.05,
  "alpha_bonferroni_corrected": 0.025,
  "num_variants": 3,
  "num_comparisons": 2,
  "control_variant": "Control",
  "variants_summary": [
    {"name": "Control", "visitors": 5000, "conversions": 250, "conversion_rate": 0.05},
    {"name": "Variant A", "visitors": 5000, "conversions": 310, "conversion_rate": 0.062}
  ],
  "comparisons": [
    {
      "variant": "Variant A (new CTA)",
      "vs": "Control",
      "significant": true,
      "p_value": 0.009234,
      "z_score": 2.6078,
      "lift_absolute": 0.012,
      "lift_relative": 0.24,
      "ci_95_lower": 0.003,
      "ci_95_upper": 0.021,
      "statistical_power": 0.84
    }
  ],
  "winner": "Variant A (new CTA)",
  "required_sample_size_per_variant": 6162,
  "recommendation": "Ship Variant A (new CTA). It beats Control with 24.0% relative lift at p<0.025 (Bonferroni-corrected)."
}

🐍 Python SDK Example

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("nexgendata/ab-test-calculator").call(run_input={
    "metricType": "conversion",
    "variants": [
        {"name": "Control", "visitors": 10000, "conversions": 520},
        {"name": "Variant A", "visitors": 10000, "conversions": 605}
    ]
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print("Winner:", item["winner"])
    print("Recommendation:", item["recommendation"])

🌐 cURL Example

curl -X POST "https://api.apify.com/v2/acts/nexgendata~ab-test-calculator/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "variants": [
      {"name": "Control", "visitors": 5000, "conversions": 250},
      {"name": "Variant A", "visitors": 5000, "conversions": 310}
    ]
  }'

🔗 Zapier / Make.com Integration

Perfect for automating test analysis. Trigger: "New row in Experiments Google Sheet" → Action: "Run Actor (A/B Test Calculator)" with variant data → Output: Post winner + lift to Slack, Notion, or email.

❓ FAQ

Q: What test is used for conversion metrics? Two-sample z-test for proportions with pooled standard error. Matches Google Optimize's "Relative improvement" calculation and what Evan Miller's calculator uses.

Q: What about continuous metrics (revenue, session length)? Welch's t-test with Welch-Satterthwaite degrees of freedom. Handles unequal variances, which is the realistic case for revenue data.

Q: How are multiple variants handled? All treatments are compared against the first variant (control). Alpha is Bonferroni-corrected by dividing by the number of comparisons to control familywise error rate.

Q: Is this Bayesian or frequentist? Frequentist. This matches Google Optimize's default behavior and what most teams already reason about (p-values, CIs). Bayesian support is on the roadmap.

Q: Does it work for sequential testing / peeking? Not natively — use only once per test, after test concludes. Sequential testing requires different statistical treatment (alpha spending, mSPRT). Coming in v1.1.

Q: How is this different from Evan Miller's free calculator? Same math, but API-first. Automate across many tests instead of copy-pasting into a web form. Great for portfolios of experiments.

💰 Pricing (Pay-Per-Event)

Actor start: $0.005
Test analyzed: $0.002

Typical run cost: $0.007 per test analyzed. Effectively free for individual use. At 1,000 tests/month you're still under $7.

Facebook Ads Library Scraper — measure ad performance against A/B groups
Google Trends Scraper — trend-adjust your conversion rates
SaaS Pricing Tracker — track competitor pricing during test windows

🚀 Apify Affiliate Program

New to Apify? Sign up with our referral link for free platform credits.

A Google Optimize replacement for marketers, PMs, and data scientists who miss the simple workflow. Built by NexGenData.

💻 Code Example — Python

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("nexgendata/ab-test-calculator").call(run_input={
    # Fill in the input shape from the actor's input_schema
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

🌐 Code Example — cURL

curl -X POST "https://api.apify.com/v2/acts/nexgendata~ab-test-calculator/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ /* input schema */ }'

❓ FAQ

Q: How do I get started? Sign up at apify.com, grab your API token from Settings → Integrations, and run the actor via the Apify console, API, Python SDK, or any integration (Zapier, Make.com, n8n).

Q: What's the typical cost per run? See the pricing section below. Most runs finish under $0.10 for typical batches.

Q: Is this actor maintained? Yes. NexGenData maintains 165+ Apify actors and ships updates regularly. Bug reports via the Apify console issues tab get responses within 24 hours.

Q: Can I use the output commercially? Yes — you own the output data. Check the target site's Terms of Service for any usage restrictions on the scraped content itself.

Q: How do I handle rate limits? Apify manages concurrency and retries automatically. For very large batches (10K+ items), run multiple smaller jobs in parallel instead of one mega-job for better reliability.

💰 Pricing

Pay-per-event pricing — you only pay for what you actually extract.

Actor Start: $0.0001
result: $0.0050

🚀 Apify Affiliate Program

New to Apify? Sign up with our referral link — you get free platform credits on signup, and you help fund the maintenance of this actor fleet.

📚 More From NexGenData

Explore the full catalog, tutorials, Gumroad data packs, and newsletter at thenextgennexus.com — the brand home for everything we ship.

📖 Tutorials & how-to guides
🗂️ Full actor catalog with usage examples
📦 Gumroad data packs (one-time purchases)
📬 Newsletter — monthly drops of new actors and revenue experiments

Built and maintained by NexGenData — 165+ actors covering scraping, enrichment, MCP servers, and automation. 🏠 Home: thenextgennexus.com

Why A/B Test Calculator Beats VWO, AB Tasty & Optimizely

Feature	NexGenData A/B Test Calculator	VWO	AB Tasty	Optimizely
Cost	$0.005 / calculation, pay-per-result	$199-749+ / month	Custom (typically $1000+/mo)	Enterprise ($25K+/yr)
Statistical engine	scipy.stats — same math, no UI	Proprietary	Proprietary	Proprietary
Conversion-rate test	Two-sample z-test	Yes	Yes	Yes
Continuous-metric test	Welch's t-test	Yes	Yes	Yes
Multi-variant + Bonferroni	Yes	Yes	Yes	Yes
Sample-size calculator	Yes	Yes	Yes	Yes
API access	Apify REST + JSON	Plan-gated	Plan-gated	Plan-gated
Auth required	Apify token	Account + plan	Account + plan	Enterprise contract
Free tier	Free Apify credits	30-day trial	Demo only	Demo only

Marketers, PMs, and growth engineers who already store variant data in Mixpanel / Amplitude / a warehouse pick this actor instead of paying $199-1000+/month for VWO or AB Tasty just to read the final p-value. It is a drop-in alternative to the discontinued Google Optimize significance display — same workflow, no monthly fee. Cheaper than Optimizely's $25K+/year enterprise tier by 1000×+, and compared with self-hosted GrowthBook / PostHog you skip the deployment overhead entirely.

Use case	Actor
Shopify storefront teardown	Shopify Store Analyzer
SaaS pricing-page change tracker	SaaS Pricing Tracker
Bulk Lighthouse / Web Vitals	Bulk Lighthouse Checker
Page-speed analyzer	Page Speed Analyzer
WCAG accessibility auditor	WCAG Accessibility Auditor
Competitor price tracking	Competitor Price Monitor
G2 SaaS reviews	G2 Reviews Scraper
Apple App Store rankings	Apple App Store Scraper
Hiring-signal detector	Hiring Signal Detector