Midscene
Describe UI tests in plain English and an AI model runs them by looking at the screen.
- Stars
- 15.5k
- Forks
- 1.2k
- Licence
- MIT
- Language
- TypeScript
- Last update
- 4 days ago
- Type
- Dev tool
What it is
Midscene is an open-source framework for end-to-end testing of web, mobile, and desktop apps. You write steps and expected results in natural language, and an AI vision model uses screenshots to click, type, scroll, and check what is visible, instead of relying on fixed selectors. It can run as YAML test cases through Midscene Test or inside existing Playwright or Puppeteer tests.
- Write a YAML test case that searches a shop site, filters results, and asserts prices
- Add aiAssert checks for colors, selection states, or error messages to a Playwright test
- Use aiQuery to extract structured data from a page during a test
- Automate a mobile app on an Android device through ADB
- Open the generated HTML report to review screenshots and assertion results after a run
Great for
- Teams writing end-to-end tests for web apps who want less selector maintenance
- Developers adding visual checks to existing Playwright or Puppeteer suites
- QA engineers testing mobile apps on Android or iOS devices via ADB or WebDriverAgent
- Testers who need to check canvas content or visual details like highlights and layout
Not ideal for
- Teams that need fully deterministic, repeatable test runs, since results depend on an AI model's judgment and the model's cost and behavior
- Projects that cannot send screenshots to a hosted model provider, unless they set up a self-hosted model (the README mentions open-source options without detailing setup)
- Simple unit-level tests, which do not need a visual agent
How to get it
- 1
Install with a package manager
You'll need: Node.js (version not stated in README excerpt) · An AI model configured per the Model Configuration guide · A Playwright or Puppeteer setup for web integration, or ADB/WebDriverAgent/HDC for mobile
Clone the repository
git clone https://github.com/web-infra-dev/midscene - 2
Run it
- Create a Midscene Test project following the guide at midscenejs.com/midscene-test/use
- Configure a model and the Web runtime
- Write a YAML case with aiAct, aiWaitFor, and aiAssert steps
- Run the case and open the generated HTML report
- 3
Get the most out of it
- Use the Playground to try and refine instructions against your interface before writing tests
- Wrap reusable setup like API calls or data cleanup in TypeScript Nodes and share them across cases
- Export the Node Spec as Markdown so developers and AI agents can see project capabilities
- Install Midscene Skills to let AI coding agents operate and verify the UI
- Pick a model using the Model Selection Guide to balance accuracy and cost
Downloads
Strengths
- Uses screenshots rather than DOM selectors, so it can target icon-only buttons, canvas, and cross-origin iframes
- Works with the same API across web, Android, iOS, HarmonyOS, and desktop
- Integrates with existing Playwright and Puppeteer tests and produces interactive HTML reports
Trade-offs
- Requires configuring an AI model, and each run may incur model costs
- Results depend on the chosen model; the README's benchmark numbers come from specific runs and configurations
How it compares
| Project | Stars | Type | Level | Updated | Best for |
|---|---|---|---|---|---|
| midscene (this) | 15.5k | Dev tool | intermediate | 4 days ago | Teams writing end-to-end tests for web apps who want less selector maintenance |
| stagehand | 25.9k | Library | intermediate | 6 days ago | Developers building browser agents that must work on sites they do not control |
| browser-use | 120k | Library | intermediate | 4 days ago | Python developers who want to prototype an AI agent that operates websites |
| storybook | 91.4k | Dev tool | intermediate | 4 days ago | Frontend teams building a shared component library or design system |
| jest | 45.5k | Library | beginner | 4 days ago | JavaScript or TypeScript developers who want a testing setup that works with little configuration |
| cua | 31.6k | Dev tool | intermediate | 4 days ago | Developers building agents that operate desktop apps or browsers |
Stats synced from GitHub. The assessment above was drafted with AI from the project's README on 2026-10-29 and may miss recent changes — check the README before relying on a command. MIT licence: free to use, modify and distribute, including commercially, as long as you keep the copyright notice.