Guide · Open source

How to use Midscene: write E2E tests in natural language

Updated 2026-10-29 · written with AI from the project's own documentation and checked against our catalogue data

Midscene is an open-source framework for end-to-end testing of web, mobile and desktop apps. Instead of fixed selectors, you describe steps and expected results in plain language, and an AI vision model uses screenshots to act and check. This guide shows you how to set it up, write your first test in YAML or inside Playwright, and decide whether it fits your project.

Midsceneweb-infra-dev/midscene · ★ 15.5k · MIT Install options & downloads

Quick start

1Clone the repository. Run git clone on the Midscene repository to get the source locally.
2Check Node.js. Confirm Node.js is installed; the required version is not stated in the README excerpt.
3Configure an AI model. Follow the Model Configuration guide to connect a vision-capable model.
4Create a Midscene Test project. Follow the Create and use test projects guide and configure the Web runtime.
5Write your first YAML case. Add gotoUrl and steps using aiAct, aiWaitFor and aiAssert, then replace the example URL with your app's address.
6Or add Midscene to Playwright. Create a PlaywrightAgent from your page and call aiAct, aiWaitFor and aiAssert in your existing test.
7Review the HTML report. Open the generated report to check screenshots, actions and assertion results, and refine instructions where needed.

What you need before you start

Midscene needs Node.js. The README excerpt does not state a required version, so check the project documentation before installing.

You also need an AI model configured as described in the Model Configuration guide. Midscene sends screenshots to the model, so the model provider must be able to receive them, or you must set up a self-hosted model.

Node.js (version not stated in the README excerpt)
An AI model set up via the Model Configuration guide
A Playwright or Puppeteer setup for web tests, or ADB, WebDriverAgent or HDC for mobile

Get the Midscene source code

The README's install step clones the repository. Run the command below to get a local copy of the project.

git clone https://github.com/web-infra-dev/midscene

Write a test in YAML with Midscene Test

Midscene Test (@midscene/test) lets you express tests as YAML cases. After creating a Midscene Test project and configuring a model and the Web runtime, write steps with aiAct, aiWaitFor and aiAssert. Replace the example URL with your app's address.

The three keys shown in the example are aiAct for actions, aiWaitFor for waiting on a visible state, and aiAssert for checks.

beforeEach runs gotoUrl to open your app
Each case has a name and a list of steps
Use aiAct for actions, aiWaitFor for waiting, aiAssert for checks

beforeEach:
  - gotoUrl: https://your-shop.example

cases:
  - name: Find headphones under $100
    steps:
      - aiAct: Search for headphones and filter the results to prices under $100
      - aiWaitFor: The filtered search results are visible
      - aiAssert: There are products in the search results, and each costs less than $100

Use Midscene inside an existing Playwright test

If you already have Playwright tests, you can add Midscene without moving to YAML. Configure a model, open your app in a Playwright page, create a PlaywrightAgent, and call the same kinds of methods in code.

The README's Puppeteer integration is the alternative for Puppeteer suites; see the Puppeteer guide on the Midscene site.

import { PlaywrightAgent } from '@midscene/web/playwright';

const agent = new PlaywrightAgent(page);

await agent.aiAct('Search for headphones and filter the results to prices under $100');
await agent.aiWaitFor('The filtered search results are visible');
await agent.aiAssert('There are products in the search results, and each costs less than $100');

Check visual details and canvas content

Midscene locates elements by appearance and position, so it can target icon-only buttons, custom controls, canvas content and elements inside cross-origin iframes without selectors. You can also assert visual details such as colors, selection highlights and layout in plain language.

await agent.aiAssert('The selected plan has a blue border and a checkmark');
await agent.aiAssert('The error message is visible below the email field');

Read the report and debug failures

Midscene generates an HTML report. Open it to inspect screenshots, actions and assertion results. Interactive reports also show element locations and the AI decision process. The Playground can be used to try and refine instructions against your interface before you commit them to tests.

Screenshots of each step
Element locations and the AI decision process
Action and assertion results
Playground for refining instructions

Pick a model that fits your budget

The README lists supported models including Qwen3.x, Doubao-Seed-2.1, DeepSeek V4 Flash, GLM-4.6V, gemini-3.5-flash and UI-TARS, with open-source options you can self-host. Use the Model Selection Guide to choose and the Model Configuration guide to set one up.

The README reports one AppControlBench run that passed 58 of 60 tasks with $0.59 total model cost. Treat that as a single reported run and check the linked report for per-task costs.

What people use it for

Web teams writing E2E tests that need less selector maintenance
Adding visual checks such as highlights and layout to existing Playwright or Puppeteer suites
Testing canvas content or icon-only controls that selectors cannot reach easily
QA engineers testing Android or iOS apps through ADB or WebDriverAgent

Frequently asked questions

Is Midscene free to use?

Midscene is released under the MIT licence, as shown in the README. The AI model you use may have its own costs, so check your model provider's pricing; the README reports a $0.59 total model cost for one benchmark run.

What do I need to install Midscene?

You need Node.js, an AI model configured per the Model Configuration guide, and either a Playwright or Puppeteer setup for web tests or ADB, WebDriverAgent or HDC for mobile. The README does not state a Node.js version.

Can I use Midscene with Playwright?

Yes. The README shows a PlaywrightAgent that wraps an existing Playwright page, letting you call aiAct, aiWaitFor and aiAssert inside your current tests.

Does Midscene work without sending screenshots to a hosted model?

The README says Midscene supports open-source models you can self-host, but it does not detail the setup. Check the Model Selection and Model Configuration guides if you cannot send screenshots to a hosted provider.

Are Midscene test results deterministic?

Not necessarily. Results depend on the AI model's judgment and its behavior, so the same test may not always produce the same result. Teams that need fully repeatable runs should weigh this before adopting it.

How does Midscene compare to Playwright, Puppeteer, Appium or Stagehand?

The README lists these as comparable projects but does not compare them in detail. Midscene's main difference, per the README, is locating elements from screenshots rather than fixed selectors, and it can drive mobile and desktop apps as well as web.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

Midscene suits teams who want less selector maintenance, want to add visual checks to Playwright or Puppeteer, or need to test mobile or canvas-based UIs. It is a poor fit if you need fully deterministic runs, cannot send screenshots to a model provider, or only need simple unit-level tests.

Install options & downloads