# AI E2E Test Runner (`rainminer/ai-e2e-test-runner`) Actor

Write E2E tests in plain English. Turns steps into Playwright actions, runs them against your public site, and returns pass/fail with screenshots, video, console logs, and network errors. Built for indie hackers, small teams, and PMs who need QA without writing test code.

- **URL**: https://apify.com/rainminer/ai-e2e-test-runner.md
- **Developed by:** [rainminer](https://apify.com/rainminer) (community)
- **Categories:** AI, Automation, Other
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $250.00 / 1,000 test run completeds

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## AI E2E Test Runner

Write end-to-end tests in **plain English**. No Playwright code required. This Actor turns natural-language steps into browser actions, runs them against a public website, and returns pass/fail results with screenshots, session video, console logs, and network errors.

Paste a URL, list the steps a user would take, and start the run. Indie hackers, small product teams, and non-technical PMs can smoke-test signup, search, checkout, and dashboard flows without standing up a CI pipeline.

![Playwright](https://playwright.dev/img/playwright-logo.svg)

***

### What does AI E2E Test Runner do?

Give it a website URL and a list of instructions such as “Click Sign Up”, “Fill email field with test@example.com”, or “Verify that Welcome appears”. The Actor:

- Parses each instruction into a structured action (navigate, click, fill, verify, screenshot, and more)
- Opens Chromium (Firefox/WebKit when available) with smart waits and self-healing locators
- Retries a failed step up to three times
- Saves screenshots and an optional WebM recording
- Writes a single JSON result with per-step status, duration, and artifacts

Unrecognized phrasing is handled with a rule-based parser first. When an OpenRouter key is configured on the Actor, leftover instructions can be interpreted by a small language model. The runner **never executes generated JavaScript** — only structured actions.

***

### Why run E2E tests in plain English?

- **Ship faster** — product people can describe a flow instead of waiting on a test engineer
- **Catch broken CTAs** — verify that buttons, forms, and welcome copy still appear after a deploy
- **Keep evidence** — screenshots and video make failures easy to share in Slack or a ticket
- **Automate on Apify** — schedule nightly smoke tests, call the Actor from CI, or fan out URLs via API
- **No test repo required** — useful when you do not yet have Playwright in the application codebase

***

### Who is it for?

- Indie hackers smoke-testing marketing sites and SaaS onboarding
- Small engineering teams that want a scheduled Playwright check without maintaining fixtures
- PMs and QA generalists who can write steps but not selectors
- Agencies running the same “happy path” against many client staging URLs

***

### How to run a plain-English Playwright test

1. Open the Actor in [Apify Console](https://console.apify.com/)
2. Set **Website URL** to a publicly reachable page
3. Add **Test instructions** in order (navigate, click, fill, verify, screenshot)
4. Keep **Maximum steps** small for a first run
5. Click **Start**
6. Open the dataset for pass/fail, then the key-value store for PNG screenshots and WebM video

***

### What instructions are supported?

| You write | What happens |
| --- | --- |
| Navigate to the homepage | Opens the website URL |
| Navigate to https://example.com/pricing | Opens that URL |
| Click on ‘Sign Up’ button | Clicks a button, link, or matching text |
| Fill email field with ‘test@example.com’ | Fills the labeled or placeholder field |
| Press Enter | Sends a keyboard key |
| Verify that ‘Welcome to My App’ appears | Fails the run if the text is not visible |
| Take a screenshot of the dashboard | Saves a full-page PNG |
| Wait 2 seconds / Wait for Checkout | Pause or wait for text |
| Hover over Pricing / Check Remember me | Hover or toggle a checkbox |

Locators try accessible names, labels, placeholders, and visible text. If one strategy misses, the next is tried automatically.

***

### Input

```json
{
  "websiteUrl": "https://demo.playwright.dev/todomvc",
  "instructions": [
    "Navigate to the homepage",
    "Fill 'What needs to be done?' field with 'Buy groceries'",
    "Press Enter",
    "Verify that 'Buy groceries' appears",
    "Take a screenshot of the todo list"
  ],
  "maxItems": 5,
  "browser": "chromium",
  "viewportWidth": 1920,
  "viewportHeight": 1080,
  "timeout": 30000,
  "screenshots": true,
  "video": true,
  "continueOnFailure": false,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `websiteUrl` | string | — | **Required.** Public http(s) URL to test |
| `instructions` | string\[] | — | **Required.** Plain-English steps, in order |
| `maxItems` | integer | `20` | Maximum instructions to execute |
| `browser` | string | `chromium` | `chromium`, `firefox`, or `webkit` |
| `viewportWidth` / `viewportHeight` | integer | `1920` / `1080` | Browser viewport |
| `timeout` | integer | `30000` | Per-step timeout in milliseconds |
| `screenshots` | boolean | `true` | Save screenshots for capture steps and failures |
| `video` | boolean | `true` | Record a WebM session video |
| `continueOnFailure` | boolean | `false` | Keep going after a failed step |
| `proxyConfiguration` | object | no proxy | Enable Apify Proxy only if the site blocks datacenter IPs |
| `startUrls` | array | — | Optional fallback when `websiteUrl` is omitted |

This Actor only drives **public** pages. Do not use it to log into other people’s accounts or to bypass paywalls. For your own staging site, prefer a dedicated test user that you control.

***

### Output

One dataset item is written per run (also stored as `OUTPUT` in the key-value store):

```json
{
  "status": "passed",
  "duration": 4523,
  "websiteUrl": "https://demo.playwright.dev/todomvc",
  "finalUrl": "https://demo.playwright.dev/todomvc/#/",
  "imageUrl": "https://api.apify.com/v2/key-value-stores/.../records/screenshot-todo_list",
  "videoUrl": "https://api.apify.com/v2/key-value-stores/.../records/video-test-recording",
  "steps": [
    {
      "action": "navigate",
      "instruction": "Navigate to the homepage",
      "status": "passed",
      "duration": 1200,
      "url": "https://demo.playwright.dev/todomvc"
    },
    {
      "action": "click",
      "selector": "button|link|text='Sign Up'",
      "status": "passed",
      "duration": 340
    }
  ],
  "artifacts": {
    "screenshots": ["screenshot-todo_list"],
    "video": "video-test-recording",
    "consoleLogs": ["info: App initialized"]
  },
  "error": null
}
```

Download the dataset as JSON, CSV, or Excel. Screenshot files use `image/png`; recordings use `video/webm` (Playwright’s native format).

***

### How much does a test run cost?

Pricing is **pay per test run**. You are charged one `test-run-completed` event after the Actor saves the result and artifacts — not per instruction.

A typical smoke test (a handful of steps, screenshots on, video on, 1 GB memory) finishes in well under a minute of compute. Enable Apify Proxy only when the target site requires it; proxy traffic is billed separately by Apify.

***

### Use cases

- Nightly smoke test of marketing homepage CTAs
- Staging signup form still submits and shows a welcome message
- Todo, search, or filter UI still accepts typed input
- Pricing page still shows the advertised plan names
- Docs site search still returns a known article title
- Agency client launch checklist before go-live
- Post-deploy check from GitHub Actions via the Apify API
- Capture a short video of a flaky flow for a bug report
- Verify cookie-banner copy without writing selectors
- Regression check after a CMS content change
- Multi-viewport screenshot of a landing page (set width/height)
- Onboarding checklist for a non-technical founder

***

### Tips

- Start with 3–5 instructions and a known-good public demo before pointing at production
- Quote visible labels exactly: `Fill 'What needs to be done?' field with 'Buy groceries'`
- Prefer **Verify that ‘…’ appears** over vague “the page looks good” steps
- Turn **Continue on failure** on when you want later screenshots even if an early click misses
- The default Chromium image is enough for most sites; Firefox/WebKit may fall back to Chromium
- Failed steps store a `screenshot-failed-step-N` record (PNG) so you can see what the browser showed

***

### FAQ

**Is this a replacement for a full Playwright suite?**\
No. It is a fast, instruction-driven smoke runner. Keep unit tests and committed Playwright specs for complex application logic.

**Can it log into Gmail, a bank, or a customer’s private account?**\
No. Only test pages that are publicly reachable or that you already made available without harvesting third-party credentials.

**Why is the video WebM instead of MP4?**\
Playwright records WebM. Store players and most browsers play it; convert downstream if you need MP4.

**What if a selector is missing?**\
The step is retried three times with alternate locators. The error message names the instruction, and a failure screenshot is saved when screenshots are enabled.

**Do I need an OpenRouter key?**\
No. Common English patterns run without an LLM. A configured `OPENROUTER_API_KEY` only helps with unusual phrasing.

***

### Other rainminer Actors

If you need diagnostics instead of a scripted user flow, try [URL to Console Logs](https://apify.com/rainminer/url-console-logs-inspector?fpr=uuazcu) to capture JavaScript errors from a page load.

***

### Image Credit

Image credit: [Playwright](https://playwright.dev/)

# Actor input Schema

## `websiteUrl` (type: `string`):

Public website to test. The first Navigate step uses this URL when the instruction says homepage or does not include a full URL.

## `startUrls` (type: `array`):

Optional extra entry URLs. When websiteUrl is empty, the first start URL is used.

## `instructions` (type: `array`):

Plain-English steps to run in order. Examples: Navigate to the homepage, Click Sign Up, Fill email field with 'test@example.com', Verify that Welcome appears, Take a screenshot.

## `maxItems` (type: `integer`):

Maximum number of instructions to execute in this run.

## `browser` (type: `string`):

Playwright browser engine. Chromium is always available. Firefox and WebKit fall back to Chromium if they are not installed in the container.

## `viewportWidth` (type: `integer`):

Browser viewport width in pixels.

## `viewportHeight` (type: `integer`):

Browser viewport height in pixels.

## `timeout` (type: `integer`):

Default timeout for each Playwright action, including smart waits for the target element.

## `screenshots` (type: `boolean`):

Save a screenshot after explicit screenshot steps and after any failed step.

## `video` (type: `boolean`):

Record the browser session. Saved as WebM in the key-value store.

## `continueOnFailure` (type: `boolean`):

Keep running remaining steps after a failed step. Verification failures still mark the run as failed.

## `proxyConfiguration` (type: `object`):

Optional proxy for the test browser. Leave Apify Proxy off unless the target site blocks datacenter IPs.

## Actor input object example

```json
{
  "websiteUrl": "https://demo.playwright.dev/todomvc",
  "instructions": [
    "Navigate to the homepage",
    "Fill 'What needs to be done?' field with 'Buy groceries'",
    "Press Enter",
    "Verify that 'Buy groceries' appears",
    "Take a screenshot of the todo list"
  ],
  "maxItems": 20,
  "browser": "chromium",
  "viewportWidth": 1920,
  "viewportHeight": 1080,
  "timeout": 30000,
  "screenshots": true,
  "video": true,
  "continueOnFailure": false,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `overview` (type: `string`):

No description

## `screenshots` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "websiteUrl": "https://demo.playwright.dev/todomvc",
    "instructions": [
        "Navigate to the homepage",
        "Fill 'What needs to be done?' field with 'Buy groceries'",
        "Press Enter",
        "Verify that 'Buy groceries' appears",
        "Take a screenshot of the todo list"
    ],
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("rainminer/ai-e2e-test-runner").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "websiteUrl": "https://demo.playwright.dev/todomvc",
    "instructions": [
        "Navigate to the homepage",
        "Fill 'What needs to be done?' field with 'Buy groceries'",
        "Press Enter",
        "Verify that 'Buy groceries' appears",
        "Take a screenshot of the todo list",
    ],
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("rainminer/ai-e2e-test-runner").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "websiteUrl": "https://demo.playwright.dev/todomvc",
  "instructions": [
    "Navigate to the homepage",
    "Fill '\''What needs to be done?'\'' field with '\''Buy groceries'\''",
    "Press Enter",
    "Verify that '\''Buy groceries'\'' appears",
    "Take a screenshot of the todo list"
  ],
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call rainminer/ai-e2e-test-runner --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,rainminer/ai-e2e-test-runner"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/iyu26GcitrdHzVlNN/builds/weXcXOr3LdmjLlfpm/openapi.json