On Computer Use

Over the past few months, I have minimized the time I spend inside a coding agent interface unless it is for very specific reasons. Instead, I have been talking to my chief-of-staff agent, "Jeff," to work. This ranges from requests related to $DAYJOB (working on projects, brainstorming ideas, running benchmarks) to benign requests while I am driving, like checking whether there are any spots available for a tour on recreation.gov, without even needing to take my eyes off the road.

If one of the initial tenets was to stop looking at code, the next step in this process is to build systems for yourself that let you not look at computer screens unless you need help visualizing things (or it is faster for you to see and read something than to listen).

Before we dig deeper into why I love it now and how it works, we need to define our terminology. What I am going to be calling computer use encompasses both browser use (by way of an extension that lets you interact with the DOM, network traffic, etc.) and computer use by way of looking at accessibility trees and taking screenshots. So, basically, interacting with a computer in any way or form a human would, and then some, by relying on agents. For this post, computer use means giving an agent access to a computing environment and all of my credentials (more on this later).

This post is itself kind of meta, since the first draft I made was done entirely while using ChatGPT's new real-time voice mode. Jeff has superseded the app now, but back then I was packing my apartment: I had my AirPods on and the app connected to my remote Windows box, and it was going through old conversations, recovering context, and helping me queue up work. Heck, I even use voice mode while driving sometimes to check on my work and queue up more things. I don't have to wait until I get to my laptop; I can simply remote into my boxes, give them a task, and wait until I get to them.

Computer use for me is getting work done without having to worry about how the work gets done. Is it calling an API? Is it taking screenshots and clicking around? Does it matter? It got the job done. I haven't reached the optimization stage yet, where I want everything to be done faster and faster.

I already had a Windows box that I usually use through RDP. It is stored in a server rack somewhere in the suburban Bay Area (which means it has a residential IP!). I could never get the bundled computer use and browser use with the Codex app working on my setup, so I instead installed open-browser-use and open-computer-use. Now I have a remote Linux box too, with a full desktop.

The only thing that I am really missing out on is iMessage. That might be the only reason why you would want a Mac mini over a Windows or Linux box. There is nothing really special about macOS's accessibility approach that computer use exploits.

Computer use has made machines delegable. I was watching something on my media centre and I realised that the transcoding was terrible. But I also had my Windows box, with a GPU I could move the transcoding to. Instead of having to figure out how to set up remote transcoding, all I had to do was ask to offload transcoding from my poor old Xeon CPU to my Winbox. Ask and ye shall receive.

You are probably sold on the idea that these agents can do DevOps-y/infrastructure-y things pretty well; that is nothing new at all. What is new is the mode where you aren't even directly firing up Claude Code / Codex in these boxes, but instead using one central control plane / CoS agent that can figure out which box to connect to and delegate these tasks.

Computer use by itself can work with browsers, but having a browser-specific extension unlocks superpowers. It gives you a much more fine-grained surface area to do more. Since my browser is already logged in with my credentials, I don't have to do the boring work. I was setting up a Google Workspace CLI, and I was like, I don't want to go through the Google Cloud project, activate APIs, and do all the boring setup work. So I instead just told the agent to go ahead and set it up, and to control the browser to do whatever it needed to do. Or, when I was working on a Slack app, why would I click through and reinstall the app? Just go ahead and do it, man. "Why am I doing this?" is a question you have to ask yourself, and if there is no satisfactory answer, you gotta delegate it to agents. I have talked in my previous posts about building the software factory and building better validators, but this is exactly the same thing. You are giving agents tools so they can look and do even more.

I also understand the risk I am taking here by giving these models access to everything. I am okay with the tradeoff for now, but with open-weight models getting super good, this won't be a tradeoff I have to make for super long. Then the only tradeoff that remains is whether I am fine with the agents accidentally deleting everything.

The important shift with this is that I am getting lazier: I don't care how something gets done, as long as it gets done. The interface is the outcome. Whatever works best, works best.

The conversation/chief-of-staff agent is your control plane. Your "cloud" agent is the execution plane. I don't want agents to come back to ask how to do their job. They should come back when the job is done, or when I am the only person who can make the next decision. All these Mac minis that people bought for OpenClaw and that are now gathering dust were just a bit too soon. Computer use has gotten so much better since then that you can actually make these agents do whatever they need to. That said, a fresh VM with no accounts, applications, or persistent state is not much of a personal assistant. The useful environment needs to be provisioned first.

Give computer use a try again. The limitation for me personally is becoming that I am not creative enough; I am banging my head against a wall where I don't know what I cannot do now. The best interface might just be a conversation, and the most convincing demo might just be as ordinary as a movie that stops buffering without you needing to pull up your laptop.

We are all going to standardize onto something at some point. But if you want a taste of something similar to Jeff: get yourself a $20 ChatGPT subscription, set up Codex with Computer Use and Browser Use, and add your phone as a remote control. I am personally against vendor lock-in, but this is the easiest way for someone to try things without needing to go through the hoopla of setting everything up.

Since you also probably expect me to have some fun visualisations:

open-browser-use is a Chrome MV3 extension, a Go native host, and a local client route.

open-codex-computer-use uses the same tool shape on all three systems. get_app_state returns a screenshot and an accessibility tree. The platform runtime then uses the tree for semantic actions and the screenshot for visual coordinates.

The screenshot and the tree solve different problems. The screenshot gives visual coordinates, while the tree gives semantic metadata such as role, name, frame, value, and supported actions. The runtime can therefore click element_index=7 without guessing from pixels. It falls back to coordinates when the application does not expose a useful accessibility node.

The runtime prefers an element index when the tree has one, but it can also act at screenshot coordinates.

The final step is an event translation. A semantic click becomes an accessibility action when the target supports one, or a coordinate click becomes OS input. A drag is not a single event: the runtime sends button-down, interpolated motion points, and button-up. After this event, a new application state is collected because the old screenshot and tree can now be stale.