Automate Web Page Using OpenAI

An Apify Actor that controls a real browser with an OpenAI model to interact with web pages and extract structured data — no CSS selectors or page-specific scraping code required.

You give it a URL, tell it what to do (e.g. "search for xiaomi and click the first result"), and describe what data to extract. The agent takes screenshots at each step, sends them to OpenAI, performs the requested actions, and finally saves the extracted data to a dataset.

Built on Crawlee + Puppeteer and the Apify SDK.

---

How It Works

1. The browser opens the Start URL.
2. The Interaction Prompt is sent to the OpenAI model along with a screenshot.
3. The model issues browser actions (click, type, scroll, etc.) and the agent executes them — repeating until the task is done or Max Interaction Steps is reached.
4. A final screenshot and the cleaned page HTML are sent to the model.
5. The model extracts the data described in Data to Extract and saves it to the dataset.

Each intermediate screenshot is also saved to the key-value store so you can inspect every step the agent took.

---

Input

Field	Type	Required	Description
startUrls	array	✅	Web pages to open. Each item must have a url key.
prompt	string	✅	What the AI should do on the page (clicks, searches, form fills, etc.).
expectedOutput	string	✅	What data to extract from the final page state.
outputSchema	JS function	—	Zod schema function for typed, validated output.
openAiModel	string	—	OpenAI model to use. Default: gpt-5.4.
maxSteps	integer	—	Max browser actions before stopping. Default: 10.
countryCode	string	—	Proxy country code (US, DE, VN, FR, GB). Default: US.
proxyConfig	object	—	Advanced proxy settings (Apify Residential by default).

Basic Example — eBay Product

{
    "startUrls": [{ "url": "https://tradingeconomics.com/" }],
    "prompt": "Find the gold price. Close any popup if it appears.",
    "expectedOutput": "Extract the gold price."
}

Advanced Example — eBay Search with Typed Schema

{
    "startUrls": [{ "url": "https://www.ebay.com" }],
    "prompt": "Search by keyword 'xiaomi' and click on the first item in the search results. Close any popup if it appears.",
    "expectedOutput": "Extract the title, price and condition of the eBay item.",
    "outputSchema": "(z) => z.object({ title: z.string(), price: z.string(), condition: z.string() })",
    "maxSteps": 15,
    "countryCode": "US"
}

Output

The dataset receives one record per agent step, plus one final record with the extracted data.

Intermediate step record (written after every browser action):

{
    "url": "https://www.ebay.com/sch/i.html?_nkw=xiaomi",
    "title": "xiaomi items for sale | eBay",
    "screenshotSentToOpenAiUrl": "https://api.apify.com/v2/key-value-stores/xxx/records/screenshot_<uuid>.png"
}

Final record (written after extraction is complete):

{
    "url": "https://www.ebay.com/p/3072579174?iid=186372216016&var=694422418597",
    "title": "Samsung Galaxy S22 - 128 GB - Phantom Black (Unlocked)",
    "screenshotSentToOpenAiUrl": "https://api.apify.com/v2/key-value-stores/xxx/records/final_screenshot_<uuid>.png",
    "data": {
        "title": "Samsung Galaxy S22 - 128GB - Phantom Black (Unlocked)",
        "price": "$156.99",
        "condition": "Very Good – Refurbished"
    }
}

So a run with 5 interaction steps will produce 6 dataset records total (5 step records + 1 final record). The screenshotSentToOpenAiUrl in each record links directly to the screenshot the model saw at that step, letting you replay exactly what the agent did.

Tips

• Start simple. Use a direct product URL and a short prompt first, then add complexity.
• Be specific in expectedOutput. The more precise your description, the better the extraction (e.g. "Extract the price as a number without the currency symbol" vs "Extract the price").
• Use outputSchema when you need consistent field types across many pages — Zod will validate and coerce the model's output.
• Increase maxSteps for multi-page flows (search → click → detail page needs at least 5–10 steps).
• Prefer shorter runs. Each Actor startup has overhead; one longer run is more efficient than many short ones.

Related actors

OpenAI Web Scraper dtrungtin/openai-web-scraper

Epilogue

Thank you for trying my actor. I will be very glad for a feedback that you can send to my email dtrungtin@gmail.com.