AI E2E Test Runner

Write end-to-end tests in plain English. No Playwright code required. This Actor turns natural-language steps into browser actions, runs them against a public website, and returns pass/fail results with screenshots, session video, console logs, and network errors.

Paste a URL, list the steps a user would take, and start the run. Indie hackers, small product teams, and non-technical PMs can smoke-test signup, search, checkout, and dashboard flows without standing up a CI pipeline.

---

What does AI E2E Test Runner do?

Give it a website URL and a list of instructions such as “Click Sign Up”, “Fill email field with test@example.com”, or “Verify that Welcome appears”. The Actor:

• Parses each instruction into a structured action (navigate, click, fill, verify, screenshot, and more)
• Opens Chromium (Firefox/WebKit when available) with smart waits and self-healing locators
• Retries a failed step up to three times
• Saves screenshots and an optional WebM recording
• Writes a single JSON result with per-step status, duration, and artifacts

Unrecognized phrasing is handled with a rule-based parser first. When an OpenRouter key is configured on the Actor, leftover instructions can be interpreted by a small language model. The runner never executes generated JavaScript — only structured actions.

---

Why run E2E tests in plain English?

• Ship faster — product people can describe a flow instead of waiting on a test engineer
• Catch broken CTAs — verify that buttons, forms, and welcome copy still appear after a deploy
• Keep evidence — screenshots and video make failures easy to share in Slack or a ticket
• Automate on Apify — schedule nightly smoke tests, call the Actor from CI, or fan out URLs via API
• No test repo required — useful when you do not yet have Playwright in the application codebase

---

Who is it for?

• Indie hackers smoke-testing marketing sites and SaaS onboarding
• Small engineering teams that want a scheduled Playwright check without maintaining fixtures
• PMs and QA generalists who can write steps but not selectors
• Agencies running the same “happy path” against many client staging URLs

---

How to run a plain-English Playwright test

1. Open the Actor in Apify Console
2. Set Website URL to a publicly reachable page
3. Add Test instructions in order (navigate, click, fill, verify, screenshot)
4. Keep Maximum steps small for a first run
5. Click Start
6. Open the dataset for pass/fail, then the key-value store for PNG screenshots and WebM video

---

What instructions are supported?

You write	What happens
Navigate to the homepage	Opens the website URL
Navigate to https://example.com/pricing	Opens that URL
Click on ‘Sign Up’ button	Clicks a button, link, or matching text
Fill email field with ‘test@example.com’	Fills the labeled or placeholder field
Press Enter	Sends a keyboard key
Verify that ‘Welcome to My App’ appears	Fails the run if the text is not visible
Take a screenshot of the dashboard	Saves a full-page PNG
Wait 2 seconds / Wait for Checkout	Pause or wait for text
Hover over Pricing / Check Remember me	Hover or toggle a checkbox

Locators try accessible names, labels, placeholders, and visible text. If one strategy misses, the next is tried automatically.

---

Input

{
  "websiteUrl": "https://demo.playwright.dev/todomvc",
  "instructions": [
    "Navigate to the homepage",
    "Fill 'What needs to be done?' field with 'Buy groceries'",
    "Press Enter",
    "Verify that 'Buy groceries' appears",
    "Take a screenshot of the todo list"
  ],
  "maxItems": 5,
  "browser": "chromium",
  "viewportWidth": 1920,
  "viewportHeight": 1080,
  "timeout": 30000,
  "screenshots": true,
  "video": true,
  "continueOnFailure": false,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}

Field	Type	Default	Description
websiteUrl	string	—	Required. Public http(s) URL to test
instructions	string[]	—	Required. Plain-English steps, in order
maxItems	integer	20	Maximum instructions to execute
browser	string	chromium	chromium, firefox, or webkit
viewportWidth / viewportHeight	integer	1920 / 1080	Browser viewport
timeout	integer	30000	Per-step timeout in milliseconds
screenshots	boolean	true	Save screenshots for capture steps and failures
video	boolean	true	Record a WebM session video
continueOnFailure	boolean	false	Keep going after a failed step
proxyConfiguration	object	no proxy	Enable Apify Proxy only if the site blocks datacenter IPs
startUrls	array	—	Optional fallback when websiteUrl is omitted

This Actor only drives public pages. Do not use it to log into other people’s accounts or to bypass paywalls. For your own staging site, prefer a dedicated test user that you control.

---

Output

One dataset item is written per run (also stored as OUTPUT in the key-value store):

{
  "status": "passed",
  "duration": 4523,
  "websiteUrl": "https://demo.playwright.dev/todomvc",
  "finalUrl": "https://demo.playwright.dev/todomvc/#/",
  "imageUrl": "https://api.apify.com/v2/key-value-stores/.../records/screenshot-todo_list",
  "videoUrl": "https://api.apify.com/v2/key-value-stores/.../records/video-test-recording",
  "steps": [
    {
      "action": "navigate",
      "instruction": "Navigate to the homepage",
      "status": "passed",
      "duration": 1200,
      "url": "https://demo.playwright.dev/todomvc"
    },
    {
      "action": "click",
      "selector": "button|link|text='Sign Up'",
      "status": "passed",
      "duration": 340
    }
  ],
  "artifacts": {
    "screenshots": ["screenshot-todo_list"],
    "video": "video-test-recording",
    "consoleLogs": ["info: App initialized"]
  },
  "error": null
}

Download the dataset as JSON, CSV, or Excel. Screenshot files use image/png; recordings use video/webm (Playwright’s native format).

---

How much does a test run cost?

Pricing is pay per test run. You are charged one test-run-completed event after the Actor saves the result and artifacts — not per instruction.

A typical smoke test (a handful of steps, screenshots on, video on, 1 GB memory) finishes in well under a minute of compute. Enable Apify Proxy only when the target site requires it; proxy traffic is billed separately by Apify.

---

Use cases

• Nightly smoke test of marketing homepage CTAs
• Staging signup form still submits and shows a welcome message
• Todo, search, or filter UI still accepts typed input
• Pricing page still shows the advertised plan names
• Docs site search still returns a known article title
• Agency client launch checklist before go-live
• Post-deploy check from GitHub Actions via the Apify API
• Capture a short video of a flaky flow for a bug report
• Verify cookie-banner copy without writing selectors
• Regression check after a CMS content change
• Multi-viewport screenshot of a landing page (set width/height)
• Onboarding checklist for a non-technical founder

---

Tips

• Start with 3–5 instructions and a known-good public demo before pointing at production
• Quote visible labels exactly: Fill 'What needs to be done?' field with 'Buy groceries'
• Prefer Verify that ‘…’ appears over vague “the page looks good” steps
• Turn Continue on failure on when you want later screenshots even if an early click misses
• The default Chromium image is enough for most sites; Firefox/WebKit may fall back to Chromium
• Failed steps store a screenshot-failed-step-N record (PNG) so you can see what the browser showed

---

FAQ

Is this a replacement for a full Playwright suite?
No. It is a fast, instruction-driven smoke runner. Keep unit tests and committed Playwright specs for complex application logic.

Can it log into Gmail, a bank, or a customer’s private account?
No. Only test pages that are publicly reachable or that you already made available without harvesting third-party credentials.

Why is the video WebM instead of MP4?
Playwright records WebM. Store players and most browsers play it; convert downstream if you need MP4.

What if a selector is missing?
The step is retried three times with alternate locators. The error message names the instruction, and a failure screenshot is saved when screenshots are enabled.

Do I need an OpenRouter key?
No. Common English patterns run without an LLM. A configured OPENROUTER_API_KEY only helps with unusual phrasing.

---

Other rainminer Actors

If you need diagnostics instead of a scripted user flow, try URL to Console Logs to capture JavaScript errors from a page load.

---

Image Credit

Image credit: Playwright