Actor A/B Tester -- Compare Two Actors Side by Side

Run two Apify actors with the exact same input in parallel and get a structured comparison report covering result count, field coverage, execution speed, and compute cost. Actor A/B Tester tells you which actor delivers more data, faster, and cheaper -- so you can make informed decisions about which scraper, enrichment tool, or data pipeline to use in production.

Features

• Runs two actors simultaneously with identical input for a fair comparison
• Compares result count, execution duration, compute cost, and output field coverage
• Identifies shared fields and unique fields between both actors' output schemas
• Declares a winner based on a weighted scoring system (results, speed, cost, fields, status)
• Returns a sample record from each actor so you can visually inspect data quality
• Works with any Apify actor -- scrapers, enrichment tools, API wrappers, or custom actors

Use Cases

• Scraper evaluation: Compare two competing scrapers (e.g., Cheerio vs. Playwright-based) on the same target URL to see which extracts more data fields faster.
• Cost optimization: Test whether a cheaper actor produces comparable results to a premium one before committing to a production pipeline.
• Migration validation: When replacing one actor with another, verify the new actor returns equivalent or better results before switching.
• Actor development: Test your custom actor against an existing Store actor to benchmark performance during development.
• Vendor selection: Compare third-party actors from different developers on the Apify Store to pick the best one for your use case.

How to Use

1. Go to Actor A/B Tester on the Apify Store.
2. Click Try for free.
3. Enter your Apify API Token.
4. Enter the two actor IDs or names (e.g., apify/web-scraper and apify/cheerio-scraper).
5. Provide a Test Input JSON that both actors accept.
6. Optionally adjust timeout and memory settings.
7. Click Start and wait for both actors to finish.
8. View the comparison report in the Dataset tab.

Input Parameters

Parameter	Type	Required	Default	Description
apiToken	string	Yes	--	Your Apify API token for authenticating actor runs
actorA	string	Yes	--	Actor ID or username~name for the first actor (e.g., apify/web-scraper)
actorB	string	Yes	--	Actor ID or username~name for the second actor (e.g., apify/cheerio-scraper)
testInput	object	Yes	--	JSON input to pass to both actors. Must be compatible with both actors' input schemas.
timeout	integer	No	120	Maximum seconds to wait for each actor to complete
memory	integer	No	512	Memory allocation in MB for each actor run

Output Example

{
    "actorA": {
        "name": "apify/web-scraper",
        "status": "SUCCEEDED",
        "results": 15,
        "duration": 8.2,
        "cost": 0.003,
        "fields": ["url", "title", "email"],
        "sampleRecord": {
            "url": "https://example.com/page1",
            "title": "Example Page",
            "email": "info@example.com"
        }
    },
    "actorB": {
        "name": "apify/cheerio-scraper",
        "status": "SUCCEEDED",
        "results": 12,
        "duration": 12.5,
        "cost": 0.005,
        "fields": ["url", "name", "contact_email", "phone"],
        "sampleRecord": {
            "url": "https://example.com/page1",
            "name": "Example Company",
            "contact_email": "info@example.com",
            "phone": "+1-555-0100"
        }
    },
    "comparison": {
        "winner": "actorA",
        "reasons": [
            "25% more results (A: 15, B: 12)",
            "34% faster (A: 8.2s, B: 12.5s)",
            "40% cheaper (A: $0.003, B: $0.005)"
        ],
        "sharedFields": ["url"],
        "uniqueToA": ["title", "email"],
        "uniqueToB": ["name", "contact_email", "phone"],
        "resultCountDiff": "+25%",
        "costDiff": "-40%",
        "speedDiff": "-34%"
    },
    "testedAt": "2026-03-18T10:30:00.000Z"
}

Output Fields

Field	Type	Description
actorA	object	Full run results for the first actor
actorB	object	Full run results for the second actor
actorA/B.name	string	Actor ID or name as provided in input
actorA/B.status	string	Run status: SUCCEEDED, FAILED, TIMED-OUT, ABORTED, ERROR, or FAILED_TO_START
actorA/B.results	number	Number of items in the output dataset
actorA/B.duration	number	Run duration in seconds (from Apify timestamps)
actorA/B.cost	number	Total compute cost in USD
actorA/B.fields	array	List of all unique field names found across output records
actorA/B.sampleRecord	object	First record from the output dataset for visual inspection
comparison.winner	string	actorA, actorB, or tie
comparison.reasons	array	Human-readable explanations for the winner decision
comparison.sharedFields	array	Fields present in both actors' output
comparison.uniqueToA	array	Fields only in Actor A's output
comparison.uniqueToB	array	Fields only in Actor B's output
comparison.resultCountDiff	string	Percentage difference in result count (A vs B)
comparison.costDiff	string	Percentage difference in cost (negative = A is cheaper)
comparison.speedDiff	string	Percentage difference in speed (negative = A is faster)
testedAt	string	ISO 8601 timestamp of when the test was run

How It Works

1. Parallel execution. Both actors are started simultaneously via the Apify API with identical input, timeout, and memory settings. This ensures a fair comparison under the same conditions.

2. Polling. If either actor is still running after the initial request, the tester polls every 3 seconds until the actor completes or the timeout expires.

3. Data collection. For each actor, the tester fetches up to 1,000 items from the output dataset, extracts all unique field names, and records the first item as a sample record.

4. Cost retrieval. The total USD cost is fetched from the Apify run details API, which includes compute and platform usage.

5. Comparison scoring. The winner is determined by a point system:

   • Results count: 2 points to the actor with more results
   • Speed: 1 point to the faster actor
   • Cost: 1 point to the cheaper actor
   • Field coverage: 1 point to the actor with more output fields
   • Status: 3 points if one actor succeeded and the other did not

6. Report generation. The comparison includes percentage differences, field overlap analysis, and human-readable reasons for the winner.

How Much Does It Cost?

Actor A/B Tester uses Pay-Per-Event pricing at $0.15 per A/B test. This covers the orchestration and comparison logic. The two sub-actor runs are billed separately at their own rates.

Scenario	A/B Tests	Orchestration Cost	Sub-actor Cost
Single comparison	1	$0.15	Varies by actor
Weekly benchmarks (4/month)	4	$0.60	Varies by actor
Evaluation of 10 actor pairs	10	$1.50	Varies by actor

The Apify Free plan includes $5 of monthly credits, enough for approximately 30 A/B tests (orchestration only, excluding sub-actor compute costs).

Tips

• Use the same memory and timeout for fair comparisons. Different memory allocations can significantly affect both speed and cost. The A/B tester applies the same settings to both actors by design.
• Start with small inputs. Use a limited test input (e.g., 1-2 URLs) to keep sub-actor costs low during initial evaluation.
• Check the sampleRecord. Raw result counts do not tell the full story. The sample record lets you compare actual data quality -- field richness, data completeness, and value formats.
• Compare field overlap. Two actors might return different field names for the same data (e.g., email vs contact_email). The shared/unique field analysis helps you spot this.
• Run multiple tests. A single run can be noisy due to network conditions or API rate limits. Run 3-5 tests for reliable benchmarks.

Frequently Asked Questions

Q: Can I compare actors with different input schemas? A: The test input must be valid for both actors. If Actor A expects startUrls and Actor B expects urls, you will need to structure the input to satisfy both schemas, or test with an input format both accept.

Q: What happens if one actor fails? A: The tester still produces a full report. The failed actor will show status FAILED (or ERROR/TIMED-OUT), zero results, and the successful actor will receive 3 bonus winner points for completing successfully.

Q: Does the $0.15 fee include the sub-actor run costs? A: No. The $0.15 covers the A/B Tester orchestration. Each sub-actor run is billed separately based on that actor's pricing model (compute time, PPE events, etc.).

Q: Can I compare more than two actors at once? A: Not in a single run. Run multiple A/B tests to create a tournament bracket -- compare pairs and then compare the winners.

Q: Is it legal to compare actors from other developers? A: Yes. You are running actors through the standard Apify API using your own token and credits. This is no different from running any public actor on the Apify Store.

Q: Why is the winner sometimes "tie"? A: A tie occurs when both actors score equally across all comparison dimensions. This can happen when both produce the same number of results with similar speed and cost.

Integrations

Use this actor with:

• Zapier for automated weekly benchmarking
• Make for complex evaluation pipelines
• Google Sheets for tracking comparison results over time
• The Apify API for programmatic access

Programmatic Access

Python

from apify_client import ApifyClient

client = ApifyClient("apify_api_xxxxxxxxxxxxxxxxxxxxxxxxxxxx")

run = client.actor("ryanclinton/actor-ab-tester").call(
    run_input={
        "apiToken": "apify_api_xxxxxxxxxxxxxxxxxxxxxxxxxxxx",
        "actorA": "apify/web-scraper",
        "actorB": "apify/cheerio-scraper",
        "testInput": {"startUrls": [{"url": "https://example.com"}]},
        "timeout": 120,
        "memory": 512
    }
)

dataset_items = client.dataset(run["defaultDatasetId"]).list_items().items
result = dataset_items[0]

print(f"Winner: {result['comparison']['winner']}")
for reason in result["comparison"]["reasons"]:
    print(f"  {reason}")

JavaScript

import { ApifyClient } from "apify-client";

const client = new ApifyClient({
    token: "apify_api_xxxxxxxxxxxxxxxxxxxxxxxxxxxx",
});

const run = await client.actor("ryanclinton/actor-ab-tester").call({
    apiToken: "apify_api_xxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    actorA: "apify/web-scraper",
    actorB: "apify/cheerio-scraper",
    testInput: { startUrls: [{ url: "https://example.com" }] },
    timeout: 120,
    memory: 512,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
const result = items[0];

console.log(`Winner: ${result.comparison.winner}`);
result.comparison.reasons.forEach((r) => console.log(`  ${r}`));

cURL

curl -X POST "https://api.apify.com/v2/acts/ryanclinton~actor-ab-tester/runs?token=YOUR_API_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
        "apiToken": "apify_api_xxxxxxxxxxxxxxxxxxxxxxxxxxxx",
        "actorA": "apify/web-scraper",
        "actorB": "apify/cheerio-scraper",
        "testInput": {"startUrls": [{"url": "https://example.com"}]},
        "timeout": 120,
        "memory": 512
    }'

Related Actors

• ryanclinton/apifyforge-quality-monitor -- Score actor quality across five dimensions
• ryanclinton/actor-competitor-scanner -- Find competing actors in the Apify Store
• ryanclinton/actor-revenue-analytics -- Track actor revenue and usage metrics
• ryanclinton/cost-watchdog -- Monitor spending and detect cost anomalies