|
| When AI Sits at the Computer This month I used AI to build slides for three speaking engagements. I gave it my script and my slide template, and what came back wasn't instructions for making the deck. It was the deck, ready for me to download. It wasn't perfect, but it was very good, and it saved me about six hours per talk.
That work happened on a computer the AI runs for itself, where it writes code and builds the file. The next step is AI working on a computer the way we do: opening apps, reading the screen, typing, and clicking. It's called computer use. And depending on the AI company, some agents get their own computer. Others work on your computer, with your apps and your logins.
The demos get more impressive every few months, but the questions stay the same: what are you handing over, whose identity does it carry, does the work come back right, and what gatekeeper (harness) protects you from mistakes and hallucinations? | Meta's new Muse agent runs on its own cloud computer and can keep working after you close the app. Meta says all logins are stored and used without the agent seeing them, and every outbound request passes through a separate gatekeeper (harness) that checks it against rules you set. Two weeks after launch, TechRadar reported that Muse read Inc. writer Jason Aten's private text messages, which he says he chose not to share, and synced his Messages history to Meta's servers. Key Insight: A permission screen tells you what an agent is supposed to reach. Before you hand over your logins, test what it can actually reach, check what it syncs, and start with accounts where a mistake is cheap. |
Most AI benchmarks test what a model knows. This one, built by ServiceNow, tests what a model can actually do. It puts an AI agent into a simulated company with email, calendars, shared drives, customer service, HR ticketing, and IT ticketing. Then it assigns jobs: find the employee behind a support issue, update the ticket, schedule a follow-up, and message the right people. Right now, the best model completed 51% of the tasks correctly. My Take: Picture an HR task where the agent finds the case, updates it, adds the note, and emails the manager, then books the meeting in the wrong time zone. With current success rates, someone still has to check every result. See the Benchmark Results → Developer Bobby Hall argues the computer will become a tool for AI rather than for us. His progression: the chatbot says, "Here's how." The assistant says, "I'll help." The agent says, "I'll do the task." What's next is, "I'm responsible for that." Instead of asking for outbound messages to 50 prospects on a schedule, you might write "here's the CRM, you own prospecting." It's one developer's view, but it's a clear picture of where vendors are aiming. The Practical Angle: An agent can take on the task, but the result stays with you. If you're going to review outcomes instead of steps, write the definition of done in the job description, and check results before they go out. Read Hall's Argument → | Quick Hits.Foundations How AI Computer Use Works The model doesn't click anything. It reads a screenshot, plans the next move, and sends that to a separate layer called a harness, which holds permissions and decides whether the move happens. Understanding the harness, and whether it runs on your machine or its own, is critical. | .Video Watching Claude Use a Desktop Watch Claude use a computer the way you would, moving from the calendar to Reminders to a cart in Uber Eats. It tries connected apps first and asks permission before taking over an app. Midway it types "udder" instead of "butter," spots the mistake on screen, and fixes it. | .Deep Dive Some CAPTCHAs No Longer Stop AI Agents Researchers pitted AI browser agents against widely used bot checks. Agents beat puzzle-style challenges consistently. Google's reCAPTCHA v3 held them to 23% on average. The one agent that passed: a Chrome extension running in a real profile with cookies and history. |
|
| Industry DevelopmentsClaude Gets Its Own Browser Anthropic put a browser inside Claude and switched it on by default. It lives on your machine but has none of your logins unless you type them in. Still, Anthropic confirms malicious instructions hidden on a webpage (prompt injections) can hijack it. | Perplexity Runs the Agent on Your Machine Perplexity's Portable Computer for Windows, enables its agent to run locally on your own computer (provided you have a high-end graphics card), reaching Gmail, Outlook, Slack and GitHub without your data leaving the machine. It can also use cloud models but the harness asks permission before doing so. |
|
| |
|
