Skip to content
Get packs

Self-Operating Computer

Lets an AI model look at your screen and control the mouse and keyboard to complete a task.

GitHub Download v1.5.8
Stars
10.3k
Forks
1.4k
Licence
MIT
Language
Python
Last update
17 days ago
Type
Framework

What it is

This is a Python framework that lets a multimodal AI model view your computer screen and decide on a sequence of mouse and keyboard actions to reach a goal you give it. The README says it uses the same inputs and outputs a human operator would. It is commonly used to experiment with AI 'computer use' and to compare how different vision models perform at operating a computer.

  • Give it a goal in plain language and watch it attempt to complete it on screen
  • Switch between supported models such as gpt-4o, gemini-pro-vision, claude-3, or a local LLaVA via Ollama using the -m flag
  • Give the agent a spoken objective using voice mode after installing the audio requirements
  • Test the optical character recognition mode (gpt-4-with-ocr) or Set-of-Mark prompting mode (gpt-4-with-som)

Great for

  • Developers exploring AI computer-use agents
  • Researchers comparing vision-capable models on desktop tasks
  • Students learning how screen-reading agents work
  • People who already have an OpenAI or other supported model API key

Not ideal for

  • Anyone who needs reliable unattended automation, since the README warns that error rates are high for some models such as LLaVA and that it may not work as expected during model outages
  • Users who do not want a model reading their screen and sending content to a third-party API
  • Windows or Linux users who cannot grant screen and input access, since the README requires those permissions
Step-by-step guide: how to use Self-Operating ComputerInstall, first run, examples, tips and FAQs
Self-Operating Computer vs UFO

How to get it

  1. 1

    Install with a package manager

    You'll need: Python · An API key for at least one supported model (e.g. OpenAI) · Terminal access with screen recording and accessibility permissions on macOS

    Install the project

    pip install self-operating-computer

    Run the project

    operate
  2. 2

    Run it

    • Install with pip install self-operating-computer
    • Run operate in your terminal
    • When prompted, enter your OpenAI API key (stored in .env)
    • Grant screen recording and accessibility permissions if on macOS, then give the agent an objective
  3. 3

    Get the most out of it

    • Use operate -m with a model name to choose a different backend, such as gemini-pro-vision or claude-3
    • Try operate --voice for spoken objectives after installing requirements-audio.txt and portaudio
    • Use the gpt-4-with-ocr mode, which the README says performed better than SoM and vanilla GPT-4 in recent tests
    • Join the Discord #self-operating-computer channel for community support

Downloads

Source code (.zip)main branch · straight from GitHubRelease v1.5.8Release notes and sourceOfficial website & docswww.hyperwriteai.com/self-operating-computer

Strengths

  • Supports several multimodal models, including local options via Ollama according to the README
  • Installs with a single pip command and runs with a single command
  • MIT licensed and open source with an active community Discord

Trade-offs

  • The README states error rates are very high with LLaVA and the project is described as a base to build on
  • Depends on paid third-party model APIs for most options, and gpt-4o access requires a minimum spend noted in the README
  • Requires granting screen recording and accessibility permissions on macOS

How it compares

ProjectStarsTypeLevelUpdatedBest for
self-operating-computer (this)10.3kFrameworkintermediate17 days agoDevelopers exploring AI computer-use agents
langchain149kFrameworkintermediate4 days agoPython developers building LLM-powered applications
huginn50.3kAppintermediate4 days agoPeople who want IFTTT-style automations without sending their data to a third-party service
CowAgent47.8kAppintermediate4 days agoPeople who want a self-hosted personal assistant reachable from their everyday chat apps
LibreChat46.6kAppintermediate4 days agoTeams that want one shared chat interface across several AI providers
ai-goofish-monitor15.9kAppintermediate27 days agoPeople hunting for specific second-hand items on Xianyu over time

Stats synced from GitHub. The assessment above was drafted with AI from the project's README on 2026-10-29 and may miss recent changes — check the README before relying on a command. MIT licence: free to use, modify and distribute, including commercially, as long as you keep the copyright notice.