# 🧪 A/B Test Significance Calculator — Google Optimize Alt (`nexgendata/ab-test-calculator`) Actor

Drop-in Google Optimize replacement. Calculate p-values, confidence intervals, lift, sample sizes, and winners for A/B tests. Two-proportion z-test + Welch's t-test + Bonferroni multi-variant. API-first, no account, $0.007/test.

- **URL**: https://apify.com/nexgendata/ab-test-calculator.md
- **Developed by:** [Stephan Corbeil](https://apify.com/nexgendata) (community)
- **Categories:** Marketing, Automation, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, NaN bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage

## What's an Apify Actor?

Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md):

```bash
npm install apify-client
```

In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md):

```bash
pip install apify-client
```

In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md):

````bash
# MacOS / Linux
curl -fsSL https://apify.com/install-cli.sh | bash
# Windows
irm https://apify.com/install-cli.ps1 | iex
```bash

In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md).

If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).


# README

## A/B Test Significance Calculator — Google Optimize Replacement

> **Drop-in replacement for Google Optimize's statistical significance workflow.** Get p-values, confidence intervals, lift, statistical power, and required sample sizes — via API, no browser, no account, no monthly fee.

**Why this exists:** Google Optimize was shut down on September 30, 2023. Google pushed users toward paid enterprise solutions (GA4 + Optimize 360 partners, VWO+AB Tasty starting at $250+/mo). Open-source alternatives like GrowthBook and PostHog require self-hosting. The simple "paste variant data, get p-value" workflow that 80% of SMB users actually needed was never replaced.

This actor does that one thing, exceptionally well. Built for marketers, product managers, engineers, and data scientists who need answer "did my test win?" on demand — inside Zapier workflows, CI/CD pipelines, Slack bots, or notebooks.

### 🔑 Features

- **Conversion-rate tests** — two-sample z-test for proportions (signup rate, purchase rate, click-through rate)
- **Continuous metrics** — Welch's t-test for unequal variances (revenue per user, session length, page views)
- **Multi-variant support** — up to 6 variants with automatic Bonferroni correction for familywise error rate
- **Full output** — p-value, 95% CI, absolute + relative lift, statistical power achieved, per-variant stats
- **Sample size calculator** — tells you how many visitors per variant you need for a given MDE and power
- **Winner flag + human recommendation** — one-line "ship Variant A" or "keep testing" verdict
- **Zero scraping, zero JS** — pure Python scipy.stats. Runs in under 2 seconds. Lowest-compute actor in the fleet.

### 💼 Common Use Cases

- **Post-test analysis** — drop-in replacement for Google Optimize's significance display
- **Zapier / Make.com workflows** — auto-analyze Mixpanel/Amplitude test results nightly
- **CI/CD integration** — gate deployments on experimental success
- **Slack bots** — `/significance variant_a=250/5000 variant_b=310/5000` → auto-response
- **Notebooks / BI dashboards** — trigger via API, render results alongside visualizations
- **Marketing team Slack** — daily digest of running test statuses
- **Startup PMs** — quickly sanity-check whether the "10% lift" from the latest test is real

### 📥 Input Example — Conversion Test

```json
{
  "metricType": "conversion",
  "variants": [
    {"name": "Control", "visitors": 5000, "conversions": 250},
    {"name": "Variant A (new CTA)", "visitors": 5000, "conversions": 310},
    {"name": "Variant B (redesign)", "visitors": 5000, "conversions": 295}
  ],
  "alpha": "0.05",
  "power": "0.80",
  "minDetectableEffect": "0.05"
}
````

### 📥 Input Example — Continuous Metric

```json
{
  "metricType": "continuous",
  "variants": [
    {"name": "Control", "n": 1000, "mean": 42.50, "std": 18.30},
    {"name": "Variant A", "n": 1000, "mean": 46.80, "std": 19.10}
  ],
  "alpha": "0.05"
}
```

### 📤 Output

```json
{
  "metric_type": "conversion",
  "significance_level_alpha": 0.05,
  "alpha_bonferroni_corrected": 0.025,
  "num_variants": 3,
  "num_comparisons": 2,
  "control_variant": "Control",
  "variants_summary": [
    {"name": "Control", "visitors": 5000, "conversions": 250, "conversion_rate": 0.05},
    {"name": "Variant A", "visitors": 5000, "conversions": 310, "conversion_rate": 0.062}
  ],
  "comparisons": [
    {
      "variant": "Variant A (new CTA)",
      "vs": "Control",
      "significant": true,
      "p_value": 0.009234,
      "z_score": 2.6078,
      "lift_absolute": 0.012,
      "lift_relative": 0.24,
      "ci_95_lower": 0.003,
      "ci_95_upper": 0.021,
      "statistical_power": 0.84
    }
  ],
  "winner": "Variant A (new CTA)",
  "required_sample_size_per_variant": 6162,
  "recommendation": "Ship Variant A (new CTA). It beats Control with 24.0% relative lift at p<0.025 (Bonferroni-corrected)."
}
```

### 🐍 Python SDK Example

```python
from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("nexgendata/ab-test-calculator").call(run_input={
    "metricType": "conversion",
    "variants": [
        {"name": "Control", "visitors": 10000, "conversions": 520},
        {"name": "Variant A", "visitors": 10000, "conversions": 605}
    ]
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print("Winner:", item["winner"])
    print("Recommendation:", item["recommendation"])
```

### 🌐 cURL Example

```bash
curl -X POST "https://api.apify.com/v2/acts/nexgendata~ab-test-calculator/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "variants": [
      {"name": "Control", "visitors": 5000, "conversions": 250},
      {"name": "Variant A", "visitors": 5000, "conversions": 310}
    ]
  }'
```

### 🔗 Zapier / Make.com Integration

Perfect for automating test analysis. Trigger: "New row in Experiments Google Sheet" → Action: "Run Actor (A/B Test Calculator)" with variant data → Output: Post winner + lift to Slack, Notion, or email.

### ❓ FAQ

**Q: What test is used for conversion metrics?**
Two-sample z-test for proportions with pooled standard error. Matches Google Optimize's "Relative improvement" calculation and what Evan Miller's calculator uses.

**Q: What about continuous metrics (revenue, session length)?**
Welch's t-test with Welch-Satterthwaite degrees of freedom. Handles unequal variances, which is the realistic case for revenue data.

**Q: How are multiple variants handled?**
All treatments are compared against the first variant (control). Alpha is Bonferroni-corrected by dividing by the number of comparisons to control familywise error rate.

**Q: Is this Bayesian or frequentist?**
Frequentist. This matches Google Optimize's default behavior and what most teams already reason about (p-values, CIs). Bayesian support is on the roadmap.

**Q: Does it work for sequential testing / peeking?**
Not natively — use only once per test, after test concludes. Sequential testing requires different statistical treatment (alpha spending, mSPRT). Coming in v1.1.

**Q: How is this different from Evan Miller's free calculator?**
Same math, but API-first. Automate across many tests instead of copy-pasting into a web form. Great for portfolios of experiments.

### 💰 Pricing (Pay-Per-Event)

- **Actor start:** $0.005
- **Test analyzed:** $0.002

Typical run cost: $0.007 per test analyzed. Effectively free for individual use. At 1,000 tests/month you're still under $7.

### 🔗 Related NexGenData Actors

- [Facebook Ads Library Scraper](https://apify.com/nexgendata/facebook-ads-library-scraper?fpr=2ayu9b) — measure ad performance against A/B groups
- [Google Trends Scraper](https://apify.com/nexgendata/google-trends-scraper?fpr=2ayu9b) — trend-adjust your conversion rates
- [SaaS Pricing Tracker](https://apify.com/nexgendata/saas-pricing-tracker?fpr=2ayu9b) — track competitor pricing during test windows

### 🚀 Apify Affiliate Program

New to Apify? Sign up with our [referral link](https://www.apify.com/?fpr=2ayu9b) for free platform credits.

***

*A Google Optimize replacement for marketers, PMs, and data scientists who miss the simple workflow. Built by NexGenData.*

# Actor input Schema

## `metricType` (type: `string`):

Type of metric. 'conversion' for binary outcomes (signups, purchases). 'continuous' for numeric metrics (revenue per user, session length).

## `variants` (type: `array`):

Array of 2-6 test variants. First variant is treated as control. For conversion: {name, visitors, conversions}. For continuous: {name, n, mean, std}.

## `alpha` (type: `string`):

Significance threshold (type I error rate). Common values: 0.05 (95% confidence) or 0.01 (99% confidence).

## `power` (type: `string`):

Desired statistical power (1 - type II error rate). Standard value: 0.80.

## `minDetectableEffect` (type: `string`):

Relative effect size you want to detect (e.g. 0.05 = 5% relative lift). Used to compute required sample size.

## Actor input object example

```json
{
  "metricType": "conversion",
  "variants": [
    {
      "name": "Control",
      "visitors": 5000,
      "conversions": 250
    },
    {
      "name": "Variant A",
      "visitors": 5000,
      "conversions": 310
    }
  ],
  "alpha": "0.05",
  "power": "0.80",
  "minDetectableEffect": "0.05"
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "metricType": "conversion",
    "variants": [
        {
            "name": "Control",
            "visitors": 5000,
            "conversions": 250
        },
        {
            "name": "Variant A",
            "visitors": 5000,
            "conversions": 310
        }
    ],
    "alpha": "0.05",
    "power": "0.80",
    "minDetectableEffect": "0.05"
};

// Run the Actor and wait for it to finish
const run = await client.actor("nexgendata/ab-test-calculator").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "metricType": "conversion",
    "variants": [
        {
            "name": "Control",
            "visitors": 5000,
            "conversions": 250,
        },
        {
            "name": "Variant A",
            "visitors": 5000,
            "conversions": 310,
        },
    ],
    "alpha": "0.05",
    "power": "0.80",
    "minDetectableEffect": "0.05",
}

# Run the Actor and wait for it to finish
run = client.actor("nexgendata/ab-test-calculator").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "metricType": "conversion",
  "variants": [
    {
      "name": "Control",
      "visitors": 5000,
      "conversions": 250
    },
    {
      "name": "Variant A",
      "visitors": 5000,
      "conversions": 310
    }
  ],
  "alpha": "0.05",
  "power": "0.80",
  "minDetectableEffect": "0.05"
}' |
apify call nexgendata/ab-test-calculator --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=nexgendata/ab-test-calculator",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

```json
{
    "openapi": "3.0.1",
    "info": {
        "title": "🧪 A/B Test Significance Calculator — Google Optimize Alt",
        "description": "Drop-in Google Optimize replacement. Calculate p-values, confidence intervals, lift, sample sizes, and winners for A/B tests. Two-proportion z-test + Welch's t-test + Bonferroni multi-variant. API-first, no account, $0.007/test.",
        "version": "0.0",
        "x-build-id": "UnFgbt1Sa0SQxMYYt"
    },
    "servers": [
        {
            "url": "https://api.apify.com/v2"
        }
    ],
    "paths": {
        "/acts/nexgendata~ab-test-calculator/run-sync-get-dataset-items": {
            "post": {
                "operationId": "run-sync-get-dataset-items-nexgendata-ab-test-calculator",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK"
                    }
                }
            }
        },
        "/acts/nexgendata~ab-test-calculator/runs": {
            "post": {
                "operationId": "runs-sync-nexgendata-ab-test-calculator",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor and returns information about the initiated run in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK",
                        "content": {
                            "application/json": {
                                "schema": {
                                    "$ref": "#/components/schemas/runsResponseSchema"
                                }
                            }
                        }
                    }
                }
            }
        },
        "/acts/nexgendata~ab-test-calculator/run-sync": {
            "post": {
                "operationId": "run-sync-nexgendata-ab-test-calculator",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK"
                    }
                }
            }
        }
    },
    "components": {
        "schemas": {
            "inputSchema": {
                "type": "object",
                "required": [
                    "variants"
                ],
                "properties": {
                    "metricType": {
                        "title": "Metric type",
                        "enum": [
                            "conversion",
                            "continuous"
                        ],
                        "type": "string",
                        "description": "Type of metric. 'conversion' for binary outcomes (signups, purchases). 'continuous' for numeric metrics (revenue per user, session length).",
                        "default": "conversion"
                    },
                    "variants": {
                        "title": "Variants",
                        "type": "array",
                        "description": "Array of 2-6 test variants. First variant is treated as control. For conversion: {name, visitors, conversions}. For continuous: {name, n, mean, std}."
                    },
                    "alpha": {
                        "title": "Significance level (alpha)",
                        "type": "string",
                        "description": "Significance threshold (type I error rate). Common values: 0.05 (95% confidence) or 0.01 (99% confidence).",
                        "default": "0.05"
                    },
                    "power": {
                        "title": "Target statistical power",
                        "type": "string",
                        "description": "Desired statistical power (1 - type II error rate). Standard value: 0.80.",
                        "default": "0.80"
                    },
                    "minDetectableEffect": {
                        "title": "Minimum detectable effect (for sample-size calc)",
                        "type": "string",
                        "description": "Relative effect size you want to detect (e.g. 0.05 = 5% relative lift). Used to compute required sample size."
                    }
                }
            },
            "runsResponseSchema": {
                "type": "object",
                "properties": {
                    "data": {
                        "type": "object",
                        "properties": {
                            "id": {
                                "type": "string"
                            },
                            "actId": {
                                "type": "string"
                            },
                            "userId": {
                                "type": "string"
                            },
                            "startedAt": {
                                "type": "string",
                                "format": "date-time",
                                "example": "2025-01-08T00:00:00.000Z"
                            },
                            "finishedAt": {
                                "type": "string",
                                "format": "date-time",
                                "example": "2025-01-08T00:00:00.000Z"
                            },
                            "status": {
                                "type": "string",
                                "example": "READY"
                            },
                            "meta": {
                                "type": "object",
                                "properties": {
                                    "origin": {
                                        "type": "string",
                                        "example": "API"
                                    },
                                    "userAgent": {
                                        "type": "string"
                                    }
                                }
                            },
                            "stats": {
                                "type": "object",
                                "properties": {
                                    "inputBodyLen": {
                                        "type": "integer",
                                        "example": 2000
                                    },
                                    "rebootCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "restartCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "resurrectCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "computeUnits": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            },
                            "options": {
                                "type": "object",
                                "properties": {
                                    "build": {
                                        "type": "string",
                                        "example": "latest"
                                    },
                                    "timeoutSecs": {
                                        "type": "integer",
                                        "example": 300
                                    },
                                    "memoryMbytes": {
                                        "type": "integer",
                                        "example": 1024
                                    },
                                    "diskMbytes": {
                                        "type": "integer",
                                        "example": 2048
                                    }
                                }
                            },
                            "buildId": {
                                "type": "string"
                            },
                            "defaultKeyValueStoreId": {
                                "type": "string"
                            },
                            "defaultDatasetId": {
                                "type": "string"
                            },
                            "defaultRequestQueueId": {
                                "type": "string"
                            },
                            "buildNumber": {
                                "type": "string",
                                "example": "1.0.0"
                            },
                            "containerUrl": {
                                "type": "string"
                            },
                            "usage": {
                                "type": "object",
                                "properties": {
                                    "ACTOR_COMPUTE_UNITS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_WRITES": {
                                        "type": "integer",
                                        "example": 1
                                    },
                                    "KEY_VALUE_STORE_LISTS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_INTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_EXTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_SERPS": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            },
                            "usageTotalUsd": {
                                "type": "number",
                                "example": 0.00005
                            },
                            "usageUsd": {
                                "type": "object",
                                "properties": {
                                    "ACTOR_COMPUTE_UNITS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_WRITES": {
                                        "type": "number",
                                        "example": 0.00005
                                    },
                                    "KEY_VALUE_STORE_LISTS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_INTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_EXTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_SERPS": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}
```